Question

100g switch with breakout cable won't change to 10G


Howdy,

I have a 100G edge-core 7712-32X switch running Cumulus 6.3.1.

I want to attach some servers with 10G interfaces and have configured swp50-3 for that. Example:

auto swp5s0
iface swp5s0
bridge-access 603
link-speed 10000
mtu 9214

With a breakout DAC cable (100G QSFP to 4 x SFP+) I thought I could connect to a 10G port. However, looking at the interface with ethtool:

$ ethtool swp5s0
Settings for swp5s0:
Supported ports: [ FIBRE ]
Supported link modes: 25000baseCR/Full
25000baseSR/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: BaseR
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 25000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 68
Transceiver: internal
Auto-negotiation: off
Link detected: no

Do you have any ideas how I might fix this so I can get 10G ports on that?

$ ifup swp5s0
cmd '/sbin/ethtool -s swp5s0 speed 10000' failed: returned 255 (Cannot set new settings: Operation not suppd
not setting speed


Thanks ahead of time for any ideas that might come to mind!
Kurt

5 replies

Userlevel 3
When you break out the port, say swp5, you need to specify that the broken-out ports are to run at 10G. The Tomahawk ASIC requires that all of the ports that are broken out from a single QSFP28 port must run at the same speed. To specify that the broken-out ports are to run at 10G instead of 25G, use "4x10G" in /etc/cumulus/ports.conf. For example:
code:
5=4x10G


Scott
Thanks Scott,

I had previously done a:
code:
net add interface swp5 breakout 4x


which put a 4x in the ports.conf
"net add" won't allow a `4x10G`
I did go into the ports.conf and edit that to 4x10G, got cumulus to commit the changes (just set another port to 100G and did a commit), but still the same results in ethtool and ifup.
I have some production database systems on that switch and am loathe to restart switchd, or I'd try that, too.

Nice idea, though! I really appreciate the response!

-kb
Userlevel 3
Kurt,

In order to activate the changes in ports.conf you must restart switchd:
code:
sudo systemctl restart switchd

That will happen "behind the scenes" when you use NCLU and do the "net commit". I suspect that your attempt to make the manual change to ports.conf take affect, actually did not do anything. NCLU is smart and will realize that if there are no real changes, it will not modify anything. Try "net pend" before the commit to see if any modifications are pending.

The easiest way to make sure that your manual modifications take affect is to edit ports.conf with the 5=4x10G line and issue the above systemctl command. This will cause traffic disruption on your database systems (and anything else connected through the switch), so its best to do that when such disruption can be tolerated.

Scott
Hi all,

Looks like this is a hardware fix. On direction from Edge-Core, I'll need a 40G QSFP to 4 x 10G breakout cable to down-link to 10G connections. I have a cable on order and will let you know the result once I fire it up.

-kb
Userlevel 3
Thanks for letting us know @Kurt Bendl. I'm glad Edgecore helped you through this.

Reply