Skip to content

RTT-based vs. drop based congestion management

APNIC discusses different TCP congestion control algorithms, coming from Reno, going through CUBIC and Vegas, then introducing BBR (seems to be a variation on CoDel) and what they observed when running BBR in a network with other implementations.

TCP congestion control algorithms try to estimate the bandwidth limit of a multi-segment network path, where a stream crosses many routers. Each segment may have a different available capacity. Overloading the total path (that is, the thinnest subsegment of the path) will force packet drops by overloading the buffers of the router just in front of that thin segment. That in turn requires retransmits, which is inefficient and has nasty delays.

To make matters more complicated, the Internet is a dynamic environment and conditions can change during the lifetime of a connection.

The first half up to “Sharing” is nothing new if you followed the works of Dave Taht or later works of Van Jacobson.

BBR seems to work like CoDel: It measures the RTT, the time delay the sender sees between sending a piece of data over the TCP link and seeing the acknowledgement return. And like CoDel, BBR seems to try to keep buffers along the line just barely filled (that is, whenever a router finishes sending a thing, the next thing should be just ready to send in the buffer, but no more).

This again is not a new insight, it’s Theory of Constraints and especially DBR applied to networking.

The new insight in the paper is that RTT-based queue capacity management can in some circumstances starve drop-based algorithms. The authors show an example where BBR starves out a CUBIC connection on the same link.

The section on “Congestion Control and Active Network Profiling” is then somewhat politcal. BBR has been deployed by Youtube, and it not only relieved buffer-management between Youtube and the Youtube client, it also suddenly makes shapers on the way very visible.

Youtube is quoted with

“Token-bucket policers. BBR’s initial YouTube deployment revealed that most of the world’s ISPs mangle traffic with token-bucket policers. The bucket is typically full at connection startup so BBR learns the underlying network’s BtlBw [bottleneck bandwidth], but once the bucket empties, all packets sent faster than the (much lower than BtlBw) bucket fill rate are dropped. BBR eventually learns this new delivery rate, but the ProbeBW gain cycle results in continuous moderate losses. To minimize the upstream bandwidth waste and application latency increase from these losses, we added policer detection and an explicit policer model to BBR. We are also actively researching better ways to mitigate the policer damage.”

This is a very nice way to say “We are seeing your shitty traffic shapers and are working around them.”

Published inNetworking


  1. AndreasLobinger

    packet switching for near-isochronous transport? What can go wrong…

    • kris kris

      This is not about isochrony, it is about

      TCP/IP handles isochrony by requiring overprovisioning and buffers in the client, which is a nice way of saying that it doesn’t at all. On the other hand, in 2017 these are not completely unreasonable assumptions.

      On the other hand, no amount of overprovisioning and buffering is going to handle Bufferbloat, ever. In fact, larger buffers are making the problem worse. That is, because Bufferbloat is jitter introduced by the storage capacity of device buffers along the path interfering with the path bandwidth estimation algorithms of drop-based congestion management algorithms. Drop-based congestion management relies on packets getting lost for stepping down, which means it fills and then overfills buffers along the packets’ path. If the path contains large buffers, the drop-based algorithms are measuring the line speed wrongly as the buffer fill rate until the first drops occur, at which point you get timeouts, retransmits and then stepping up again until the buffers overflow again. The result are much higher than necessary latencies, but also an inane amount of jitter.

      RTT-based algorithms sense buffers along the path by observing wait times (queue times) instead and do not require packet drops to scale down, hence they also do not try to completely fill buffers along the path to function. In fact, their design goal is to keep a send buffer just barely non-empty.

  2. AndreasLobinger

    Kris, i’m somehow aware of this. I’m still no rounting/switching expert but i’m currently in the phase of understanding our systems architecture design (3gpp, 38.801 for interested parties, and yes that’s 5G) and more and more the question comes up: Why have we moved closer and closer mobile network systems to ‘the internet’ while actually ‘the internet’ has a list of inherent problems unsolved for decades. We have optimized fractions of ms out of the radio interface and adjacent layers but then shortly after the accessGW somehow the jungle starts…

Leave a Reply

Your email address will not be published. Required fields are marked *