It seems, that somewhere still there's a policy, which limits locally originated and locally-directed traffic to a switch.
Could anybody, please, assist me in finding the root cause of this behavior?
Best answer by Scott Emery
Yes, you are correct that 1 Mbit is a little low. Most CPU/Switch ASIC interfaces can pump a couple hundred megabits. Some are lower, and you didn't mention which switch you are running this upon.
To open up the CPU traffic and make sure that there are absolutely no ACL rules are impeding your nuttcp or iperf traffic you can use this command:
sudo cl-acltool -F all
That will remove all ACLs. If this gives you greater throughput, then you know that one of the ACLs in the /etc/cumulus/acl/policy.d/ directory is the culprit. If you are still at 1Mbit, then you might want to run tcpdump to see where packets are being dropped or if there are suspicious time delays.