Saturday, July 2, 2016

Microburst/Burst Traffic: Cisco Nexus 9508 with N9K-9564PX linecard - ALE/Northstar Buffer

Just as my normal working and study life, my blog posts are flipping back and forth between work-related and study-related content. Today is work-related.

My company recently purchased a pair of Nexus 9508 Chassis, and I chose line cards N9K-9564PX as the default line cards for these chassis.  In this blog post, I will talk about what we have run into with these line cards in the first 30 days in production, and what we have done (with TAC help) to resolve it.

Let's start with my selection. Cisco has 2 line cards for Nexus 9508 that provides 48-port 1 and 10 GE SFP+, N9K-9564PX and N9K-9464PX. The 9500 model support VXLAN. That's was the main reason we decided to go with 9564PX.

The Problem
Within the first 30 days of deployment, I have notice that both chassis have a lot of output discards, while the port utilization is fairly low. Below is a sample of our traffic between June 10th and June 16th for the Top 10 interfaces on one chassis, and top 10 interfaces with output discards

Top 10 Total Bandwidth 
Top 10 Interface with Output Discard - Display in packet/seconds
As you can see, none of our interface max over 10Gbps, and yet, we had a lot of discards. At a given time of the day, we have 7 packets drop per seconds, that is a lot of packets over a short period of 1 minutes. This means we have lots of micro-burst in our environment. I actually knew about this, and this was why we went from Nexus 5548 to these chassis to improve the performance and take care of micro-burst. Obviously, at default state, these new line cards still couldn't handle the burst in our environment. Noted we have Jumbo frame enabled for iSCSI.

The Buffer
Before we go into the solution, let's visit the buffer structure on these line cards. Cisco published a white paper on Cisco Nexus 9500 Series Switches Buffer and Queuing Architecture (hyperlink). Please feel free to read it if you're interested. In short, the 9500 line card has 2 buffers: Network Forwarding Engine (NFE) and Application Leaf Engine (ALE).

The NFE is a 12-MB buffer, shared among all active ports for both ingress and egress direction. Once the NFE is filled, the additional packets will overflow to the ALE's hairpin buffer.
The ALE is a 40-MB buffer, that got split into 3 regions:

  • 10 MB for direct ingress traffic from the ALE to fabric modules (backplane).
  • 20 MB for direct egress traffic from fabric modules to ALE.
  • 10 MB for hairpin traffic - traffic between 2 ports on the same line cards. This is consider an additional 10 MB for NFE
Here is the tricky part, by default, the ALE is operating in "Mesh Optimized" buffer-profile, which means only 500KB is allocated for burst absorption. We can configured the line cards to operation in "Burst Optimized", which is 2MB max for burst absorption, and "Ultra-Burst Optimized", which technically is 10MB max for burst absorption. This information is actually part of the QoS Configuration Guide for 6.x firmware, under Monitoring Buffer Usage for ACI Capable Devices (hyperlink). I'm not sure why Cisco remove this particular section from 7.x guide, but we are running 7.x and this info is completely relevant.

The Solution
I'm sure by now you have figure out what we did to resolve our issue. We have changed our buffer from "Mesh Optimized" to "Burst Optimized" and that seem to take care of our burst traffic. We still have minor discards (1 or 2 over the course of 2 days), but to me, that is still acceptable. In addition, we still have "Ultra Burst" option available. Below are the very few commands to change buffer on these line cards
show hardware qos ns-buffer-profile
hardware qos ns-buffer-profile {burst|mesh|ultra-burst} 


  1. Nice post. What software are you using for traffic stats?

    1. Thank you. We use LogicMonitor as our monitoring tool

    2. IEEE Final Year Project centers make amazing deep learning final year projects ideas for final year students Final Year Projects for CSE to training and develop their deep learning experience and talents.

      IEEE Final Year projects Project Centers in India are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation.

      corporate training in chennai corporate training in chennai

      corporate training companies in india corporate training companies in india

      corporate training companies in chennai corporate training companies in chennai

      I have read your blog its very attractive and impressive. I like it your blog. Digital Marketing Company in Chennai

  2. Thanks Hieu! I'll share this with the rest of my team!

  3. This comment has been removed by the author.

  4. Hello does this command required a reboot or does it impact service


  5. All are saying the same thing repeatedly, but in your blog I had a chance to get some useful and unique information, I love your writing style very much, I would like to suggest your blog in my dude circle, so keep on updates.
    AWS Training in Chennai
    AWS Course in Chennai
    Big Data Training in Chennai
    Web Designing Course in chennai
    PHP Training in Chennai
    AWS Training in Porur
    AWS Training in Tambaram

  6. I should thank you for the efforts you've got put in penning this website online. I am hoping to see the identical high-grade weblog posts from you afterward as properly. In reality, info your creative writing capabilities has stimulated me to get my own, personal weblog now ;)