Re: Shapers, CBWFQ, and tx_ring, oh my!

Keller Giacomarro @ 14/07/2012 07:07 -0300 dixit:
> I am trying to wrap my head around how QoS queueing actually functions. If
> you're able, please confirm or debunk my understanding!

Let's see if I can help. In any case, the guys that usually chime in
with details are Peter Lapukhov and, in other forums, Ivan Pepelnjak.
One issue with queueing is that usually a metaphor is used to
understand/explain it, but it does not always play nice down to
implementation details.

> Your 'normal' QoS setup involves a CBWFQ policy-map applied outbound on a
> WAN interface. As the tx_ring fills up, packets are queued into the CBWFQ
> policy-map queues (one per class-map), and are dequeued as normal. The
> tx_ring filling up is the trigger for filling queues. Okay, simple so far.

Sort of. The tx_ring filling up is the trigger to start "software
queues". But some of the CBWFQ configuration may be active even if the
tx_ring does not fill: police and shape commands (and remarking).

> The complication comes when you have an interface with a rate limit higher
> than your CIR (like a home Internet connection via 100Mbps ethernet with a
> CIR of 512Kbps). If CBWFQ is applied directly to the interface, even if
> the bandwidth is set, the tx_ring clears faster than the WAN circuit will
> take the data, and the software queues are bypassed entirely. In this
> situation, applying a CBWFQ policy-map directly to the interface, even
> setting the bandwidth command, does absolutely nothing.
>
> Here's where I get fuzzier. The solution to this is to put something else
> between the CBWFQ policy-map and the tx_ring: a shaper via nested policy
> maps. The shaper is configured to the correct CIR. As the shaper sees
> that the interface is transmitting too fast, it begins to fill up the CBWFQ
> policy-map queues instead of transmitting. In this way, the physical
> interface is faster than the CIR but we still create the necessary
> 'backpressure' to fill up the software queues.

I would not call this "backpressure". And the shaper is not "seeing" the
interface txing too fast, because it will never do so.

Let's recap, it's all about when something happens, and that something
is packet transmission.

When tx_ring is empty, a packet is transmitted when a packet is
received. So the trigger is packet reception. (And the code that does
the tx is in the rx interrupt service). (Well, not really transmitted,
but enqueued at a hardware level, so forget about it, its gone :)

Now, if the tx_ring is full, the rx interrupt service just lets the
packet in "a queue". It has to check first if the "queue" has space,
and this is quite involved too.
Then, when the tx_ring gets a place, it looks for which packet from the
queue has to be transmitted next. The tx is now controlled by the AR,
the rate of the tx interface. And CBWFQ is just a fancy way of defining
which should be the next packet to be xmitted.

When a shaper enters the mix, basically there's another trigger. If the
tx_ring gets space, it now has to check if actually getting a packet
from the queue would exceed the shaper rate, and if it would... it has
to do nothing. Well, nothing for a calculated amount of time. Call it
sleep, delay, whatever.

> Two things muck with my understanding:
> In https://supportforums.cisco.com/thread/2132501 , a Cisco employee says
> that the shaper uses WFQ (or HQF in the newest releases) to de-queue the
> CBWFQ queues. Why is the shaper implementing any dequeueing strategy at
> all? Shouldn't the CBWFQ policy-map be handling that (such as policy
> queues going first, etc)? And how can it possibly do that without full
> flow information?

I don't fully understand what you are saying here. What do you mean by
"without full flow information" ?
But the "shaper" has to choose the next packet to tx, and that usually
comes from a child policy. Or may be fair-queues, parallel flows
(conversations) with a neat selection algorithm.

> The other issue is that the show commands on my router support my
> understanding...almost. If I'm moving a lot of ssh data upstream (via
> scp), I can see the shaper queue fill and the CBWFQ queue fill, makes
> sense. Most of the time their values are the same. However, they do
> on occasion differ by a number or two. Show command artifact, or an
> indication that I have no idea what I'm talking about?
>
> gateway#show policy-map int f0/1
> FastEthernet0/1
>
> Service-policy output: pm-wan-out-shaper
>
> Class-map: class-default (match-any)
> 600317 packets, 143960388 bytes
> 30 second offered rate 631000 bps, drop rate 14000 bps
> Match: any
> Traffic Shaping
> Target/Average Byte Sustain Excess Interval Increment
> Rate Limit bits/int bits/int (ms) (bytes)
> 600000/600000 937 3750 3750 6 468
>
> Adapt *Queue* Packets Bytes Packets Bytes Shaping
> Active *Depth* Delayed Delayed Active
> - *3* 598394 141284492 90698 94225802 yes
>
> Class-map: cm-ssh (match-all)
> 70238 packets, 105192011 bytes
> 30 second offered rate 621000 bps, drop rate 14000 bps
> Match: protocol ssh
> Queueing
> Output Queue: Conversation 74
> Bandwidth 5 (%)
> Bandwidth 30 (kbps)
> (pkts matched/bytes matched) 62464/93671468
> *(depth/total drops/no-buffer drops) 2*/1816/3
> exponential weight: 9
> mean queue depth: 1

Please attach the policy config and what is that drives your attention
from this output!

>
> Appreciate your input -- hopefully this helps someone else too, as none of
> the standard study resources I've read have adequately explained how this
> works!
>
> Keller Giacomarro
> keller.g_at_gmail.com
>

HTH,
-Carlos

-- 
Carlos G Mendioroz  <tron_at_huapi.ba.ar>  LW7 EQI  Argentina
Blogs and organic groups at http://www.ccie.net