I'm wondering what being a mgmt VRF implied exactly.
Mostly what's different between :
* a mgmt VRF + using the default one for "public internet" routing.
* Using default one for mgmt + using a data plane VRF for "public internet".
Currently I was more leaning toward the second solution since it seems to have less limitation (i.e. not limited to eth0 for instance), also if I ever want OSPF for my internal private network, that part has to be in the default one (not sure if VRF OSPF is planned for the future ?).
Here's a bit of explanation about my setup if needed for context.
- 2 * 10G switches running cumulus with mlags to the hosts.
- The switches 'eth0' are connected to an 'emergency' gateway that will only be used as a last resort. Most likely we won't administer them through there using something goes very wrong.
- What we actually call "management network" that has the machine BMCs and such is just a separate VLAN but connected to the same switches (there is way too many ports so we can afford to 'waste' a few 10G port with 1G copper for that). The switches would have a L3 interface on that VLAN and that's where they would be managed from most of the time and also get their 'management' internet access (fetch packages ...)
- Most machines (physical & virtual) only have private RFC1918 IPs (in the management VLANs and most other VLANs for inter VM communication). And I'm trying to keep that as separate of the "public internet" side of things as possible. Machines on that private network get their internet access from routers "on a stick" on those switches (doing the NAT, firewall, ...)
- Routes in those private IP vlans are redistributed using OSPF (to other sites) and there are dedicated routers that handle that. So technically the switch don't have to participate to OSPF, but maybe in the future ...
- Those switches act as our border routers to the public internet (but they only receive default routes + a few more specific ones). I would put everything BGP and public IP stuff in a separate VRF.
Maybe actually doing both options at once is worth considering:
- Use the mgmt VRF for the 'emergency access' so it has its own routes.
- Use the default VRF for our RFC1918 internal network
- Use a data plane VRF for all the BGP public internet stuff.