Microburst/Burst Traffic: Cisco Nexus 9508 with N9K-9564PX linecard - ALE/Northstar Buffer

Just as my normal working and study life, my blog posts are flipping back and forth between work-related and study-related content. Today is work-related.

My company recently purchased a pair of Nexus 9508 Chassis, and I chose line cards N9K-9564PX as the default line cards for these chassis.  In this blog post, I will talk about what we have run into with these line cards in the first 30 days in production, and what we have done (with TAC help) to resolve it.

Intro
Let's start with my selection. Cisco has 2 line cards for Nexus 9508 that provides 48-port 1 and 10 GE SFP+, N9K-9564PX and N9K-9464PX. The 9500 model support VXLAN. That's was the main reason we decided to go with 9564PX.

The Problem
Within the first 30 days of deployment, I have notice that both chassis have a lot of output discards, while the port utilization is fairly low. Below is a sample of our traffic between June 10th and June 16th for the Top 10 interfaces on one chassis, and top 10 interfaces with output discards

Top 10 Total Bandwidth 

Top 10 Interface with Output Discard - Display in packet/seconds

As you can see, none of our interface max over 10Gbps, and yet, we had a lot of discards. At a given time of the day, we have 7 packets drop per seconds, that is a lot of packets over a short period of 1 minutes. This means we have lots of micro-burst in our environment. I actually knew about this, and this was why we went from Nexus 5548 to these chassis to improve the performance and take care of micro-burst. Obviously, at default state, these new line cards still couldn't handle the burst in our environment. Noted we have Jumbo frame enabled for iSCSI.

The Buffer
Before we go into the solution, let's visit the buffer structure on these line cards. Cisco published a white paper on Cisco Nexus 9500 Series Switches Buffer and Queuing Architecture (hyperlink). Please feel free to read it if you're interested. In short, the 9500 line card has 2 buffers: Network Forwarding Engine (NFE) and Application Leaf Engine (ALE).

The NFE is a 12-MB buffer, shared among all active ports for both ingress and egress direction. Once the NFE is filled, the additional packets will overflow to the ALE's hairpin buffer.
The ALE is a 40-MB buffer, that got split into 3 regions:

• 10 MB for direct ingress traffic from the ALE to fabric modules (backplane).
• 20 MB for direct egress traffic from fabric modules to ALE.
• 10 MB for hairpin traffic - traffic between 2 ports on the same line cards. This is consider an additional 10 MB for NFE

Here is the tricky part, by default, the ALE is operating in "Mesh Optimized" buffer-profile, which means only 500KB is allocated for burst absorption. We can configured the line cards to operation in "Burst Optimized", which is 2MB max for burst absorption, and "Ultra-Burst Optimized", which technically is 10MB max for burst absorption. This information is actually part of the QoS Configuration Guide for 6.x firmware, under Monitoring Buffer Usage for ACI Capable Devices (hyperlink). I'm not sure why Cisco remove this particular section from 7.x guide, but we are running 7.x and this info is completely relevant.

The Solution
I'm sure by now you have figure out what we did to resolve our issue. We have changed our buffer from "Mesh Optimized" to "Burst Optimized" and that seem to take care of our burst traffic. We still have minor discards (1 or 2 over the course of 2 days), but to me, that is still acceptable. In addition, we still have "Ultra Burst" option available. Below are the very few commands to change buffer on these line cards

show hardware qos ns-buffer-profile

hardware qos ns-buffer-profile {burst|mesh|ultra-burst}