Link from Cumulus to Switch without cumulus dont run more than 3 hours on QSFP+ port, why? 10GbE SFP+ runs. any idea?!

Cumulus Switch connected with QSFP+ / 40GbE Link goes offline after 2-3 hours, why? Link works with 10GbE but we will use the 40GbE Ports. Any idea? From Cumulus to Cumulus, it works, but not to the Switch without cumulus. We tried it with few FW versions but nothing helps... Also Reflashed the Gbics with diffrent Brand Firmware...

3 replies

Userlevel 4
Hey Andreas,

If you have real hardware feel free to open a support ticket with Cumulus global support services:

That being said.

goes offline after 2-3 hours

That is too ambiguous, what does "offline" mean here? When you do a ip link show is LOWER_UP on the 40Gb port? Do you have a diagram so we can reference the right ports and ask questions? Do you have config, model numbers, etc?
Hi Forum, our admin is in contact with cumulus specialists... But since now we havent got a solution. So i tought i ask that forum, if someone has the same problem. i am little smarter now... Link goes down, from up do down. the Physical link. We use a MLNX SN2700 100GbE Switch with cumulus 3.2.1. limited one port to QSFP+ 40GbE, and connected that one to a quanta T3048-LY2 Switch.
I tested that szenarion with direct attached cables, link is up. So i think that must be a Gbic Problem. We have a distance about 30 meters at clients racks. MPO costs to much, so we deside to do a LC-LC connection, unfortunately we have no gbics for that problem, no mellanox FW for Gbics. Has someone an idea to use QSFP+ with LC-LC cables? i try it with cisco firmware, that works but only for 2-3 hours, than the link is down. now i thougt that the gbics are the problem, we use gbics which allow us to change the firmware on it... tomorrow i start few tests with diffrent firmwares, if someone has a smart idea, many thanks from my side...
Hi Andreas,

to use LC-LC (two fibers) with QSFP+ you need the correct kind of QSFP+ module that uses multiple wavelengths to transmit 4 data streams over one fiber. The branding on the modules is just used to white-list vendor supplied modules, it does not change the functionality.

Link-down after a couple of hours might be related to temperature (overheating QSFP+ modules) or too many errors on the link resulting in an error disabled port on one side. It might be something else as well, of course.

You might be able to use active optical cables (AOC) for a distance of about 30m, which should be significantly less expensive than using above mentioned QSFP+ modules.

Edit: See the PacketPushers blog post Why Compatibility And Support Don’t Justify $1000 Optics for an interesting perspective on the matter.