Solved

Bandwidth measure on Cumulus switch itself


Userlevel 1
Recently I thought about an idea to verify link bandwidth using tool, built into switch - like iperf or nuttcp. But when I ran actual test - I got only 1 Mbit/s result (two switches back-to-back via 40G interfaces). I checked acl, and added a new rule, which matches source-dest IP without any policing actions, and verified, that counters against this rule start increasing when I perform tests. But bandwidth is still at 1 Mbit/s.

It seems, that somewhere still there's a policy, which limits locally originated and locally-directed traffic to a switch.

Could anybody, please, assist me in finding the root cause of this behavior?
icon

Best answer by Scott Emery 6 June 2018, 19:46

Sergei,

Yes, you are correct that 1 Mbit is a little low. Most CPU/Switch ASIC interfaces can pump a couple hundred megabits. Some are lower, and you didn't mention which switch you are running this upon.

To open up the CPU traffic and make sure that there are absolutely no ACL rules are impeding your nuttcp or iperf traffic you can use this command:

code:
sudo cl-acltool -F all


That will remove all ACLs. If this gives you greater throughput, then you know that one of the ACLs in the /etc/cumulus/acl/policy.d/ directory is the culprit. If you are still at 1Mbit, then you might want to run tcpdump to see where packets are being dropped or if there are suspicious time delays.

Scott
View original

4 replies

Userlevel 3
Sergei,

You may want to re-think your approach. The problem is that the interface between the CPU and the switching ASIC is nowhere near 40Gbps. On most switches it is more like a couple hundred megabits. This is fine for management and control plane traffic, because that traffic doesn't need very much bandwidth. So, no matter how many restrictions (like ACLs) you remove, you'll never be able to use a traffic generator on the CPU to saturate the front panel ports.

Scott
Userlevel 1
Scott, thanks for reply.
I totally agree with you, that I might not get 40G, and I expected that. But I didn't expect to find 1 Mbit as well - that's why I asked this question. Are you aware of any policing mechanism other than in firewall rules?

Sergei.
Userlevel 3
Sergei,

Yes, you are correct that 1 Mbit is a little low. Most CPU/Switch ASIC interfaces can pump a couple hundred megabits. Some are lower, and you didn't mention which switch you are running this upon.

To open up the CPU traffic and make sure that there are absolutely no ACL rules are impeding your nuttcp or iperf traffic you can use this command:

code:
sudo cl-acltool -F all


That will remove all ACLs. If this gives you greater throughput, then you know that one of the ACLs in the /etc/cumulus/acl/policy.d/ directory is the culprit. If you are still at 1Mbit, then you might want to run tcpdump to see where packets are being dropped or if there are suspicious time delays.

Scott
Userlevel 1
Thanks, Scott.
I tried removing all rules - and bandwidth jumped to ~300 Mbit/s - now it really seems, that I hit the limit of CPU.

Sergei.

Reply