1Gb Breakout on Mellanox SN2100

Running Cumulus Linux 3.3 on Mellanox 2100 switch with a 40GB QSFP optical module and MPO to 8xLC breakout cables.


net add interface swp1 breakout 4X

I can only breakout into a maximum 4x. Is it possible to breakout into 8x ? The upstream connections in this case will be 1Gb.

8 replies

Userlevel 3
Ok, so first off, this is physically not possible using a standard MPO fiber, since it only has 6 pairs. So at most, this is 6 bidirectional channels. Secondly, the ASIC only supports 4x breakout. I am not aware of a standard for 40G -> 8x20G. Do you have a 40G->8xLC cable?
Hi Karl. 40GB is made up of four 10G physical 'lanes' or channels, each lane can be one interface. So it is not physically possible to do 8 interfaces. Once you break the 40G down to four 10G interfaces, then you can reduce the speed of each interface to 1G, but you cannot gain more interfaces.
Thanks for the replies.

I guess the 8xLC in the product name then refers to 8 fibres resulting in 4 pairs. Assuming that's correct then setting the breakout to 4x should work for our requirements.

Does that sound more realistic?

Yes, 4 pairs of LC sounds totally normal for a data-center patch panel set of connections. However, I am concerned about the 1G speed.

When I spoke about reducing the speed of a 10G to 1G, I was thinking of copper connections, which can drop the speed easily. However, you are speaking about fiber - presumably SR (short-range), but the same applies to LR. Fiber optical modules use different technologies and encodings to do 10G vs 1G. There is such a thing as 10G/1G dual rate SFPs which have the ability to do both, but you are looking at using a 40G QSFP. It is not likely that the 40G (4x10G) optical modules you have will also have dual rate capabilities - I am not sure they exist at all. So for fiber connections you will need to run at 40G or 4x10G breakout mode. The 2100 also support 100G and 4x25G breakouts modules and cables.

To run 1G, you would need a copper breakout.
Looks like I wasn't clear in my initial email. The QSFP module I have is actually their adapter (http://www.mellanox.com/related-docs/prod_cables/PB_QuadtoSerial_Small_From_Factor_Pluggable_Adapter...)

Probably a vital bit of information. Sorry for missing that initially.

From the doc above: "The QSA is qualified for 10GbE SFP+ and 1GbE SFP transceivers meeting the Small Form Factor Pluggable (SFP) Transceiver Multi-source Agreement (MSA)."

So this adapter will slot into, say swp1. I then split this to 4x and ensure that swp1s0 terminates into a 1G transceiver.

Sounds easy, right? 😉

I have previously been advised by Mellanox that the above configuration will work, but that's likely to have been in a MLNX-OS context. Assuming the hardware does what I think it will do, do you see any issue with Cumulus supporting this configuration?

Or am I still missing something here?
Userlevel 3
Well that helps tremendously! This is the MLNX QSFP+ to SFP+ adaptor we use in our lab (we call it a "cage), and it works great and we do support using it.

So exactly as you said, you will break the 40G port out into 4x, and the swp1s0 will map to the SFP+ in the "cage". Then to use a 1G SFP, you need to set the port with the following:
iface swp1s0
link-speed 1000
link-duplex full link-autoneg off
For completeness I can confirm this works with minor changes. In our case the the adaptor doesn't support auto-negotiation or duplex setting.

$ sudo ethtool swp1s0
Settings for swp1s0:
Supported ports: [ TP MII FIBRE ]
Supported link modes: 1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Transceiver: internal
Auto-negotiation: off

Hi Karl.

Glad you hear you got this working.

The Mellanox port firmware automatically 'does the right thing' with duplex and autonegotiation when it detects the cable type, and the info isn't always correctly represented in the ethtool output. As long as the link comes up reliably, you don't need to be concerned with duplex and autoneg outputs in ethtool.