07-07-2015 11:37 AM - edited 08-03-2015 01:01 PM
I have some FCX switches that are in production as layer 2, with Jumbo set globally. Our Citrix Xen servers are having network errors and it is suspected that this is due to them not supporting Jumbo frames (others that do will attempt to send Jumbo frmaes to the Xen servers). So I need to convert these switches to run the layer 3 version of the same release. This will allow me to set Jumbo MTU per port. I have TFTP'ed the layer 3 code to flash secondary. I have tested this on a spare FCX. The management IP was gone, so I had to create a ve in the management VLAN with an IP, create a default route, and add DNS servers.
Can anyone think of anything else I would need to worry about?
Solved! Go to Solution.
07-07-2015 03:38 PM - edited 08-03-2015 01:03 PM
OK, so maybe I have run into some things that I now have questions about.
I rebooted my test bench FCX switch, with a copy of a production config, in layer 3 code. When booting up in layer 3 code, it gives two warnings. The setting from the layer 2 config that assigned a managment IP to the switch (example "ip address 192.168.1.1 255.255.255.0") gave an error. So did a message about DHCP client renew or something. So I switched to my regular data vlan and created a router-interface, then assigned it an IP address that the switch used to use.
1.) Jumbo is enabled on the whole switch. So how do I set all the ports to the default MTU of 1500?
My whole goal is to only enable Jumbo on the ports going to devices that support it. I couldn't find a command to configure the MTU for switch ports. So how do I control what ports are jumbo and what ports are not?
2.) I don't want to create a router-interface in the managment vlan for this switch. I read in another thread that with layer 3 code (I'm running version FCXR07202d) routing happens between vlans even without router-interfaces. On my test bench, I can not communicate between two devices connected to this switch, each on their own vlan, in different subnets. Normally my network core would do this, but I don't have one for the test bench. So how can I have an IP address assigned to this switch, in layer 3 mode, that I can reach it for remote management? Is the only way to use a router-interface in a vlan?
07-21-2015 06:21 AM - edited 07-21-2015 06:23 AM
1) If you have jumbo enabled, to get 1500 MTU on the ports you need to assign it individually by interface in your scenario. You can do this by using the interface range:
int e1/1/1 to 1/1/48
ip mtu 1500
Your port range will vary based on your equipment. Then, for ports that need jumbo, just back the command out at the interface config.
2) If you are running layer 3 code, you will need a routing interface on your management VLAN to connect as far as I know. You could also set up OOB management on the management interface.
07-21-2015 08:53 AM
Thanks for your reply GB_KIR!
My understanding is that you can not set the MTU on individual ports if they are members of of a l2 vlan. All my ports are members at least one l2 vlan. If get rid of all l2 vlans, set the MTU on all ports to what I need them to be (some 1499, some 9000), then, can I create my l2 vlans, and the individual ports will retain their MTU?
If that is not the case, then I may need to buy some more nics for my servers. I have to large data storage servers that backup each other, but they also allow workstation client access. I had Jumbo on the servers, but not the clients. If I have to put the servers in their own l2 vlan with jumbo, then i guess I will need to create a ve for client traffic to be routed to them, right?
08-03-2015 12:59 PM
OK, so after testing this, I have figured it out on my own.
If you want to set the MTU on idividual ports, they can not be a current member of any layer 2 vlan when you change the MTU. So you can remove the port from all layer 2 vlans, change it's MTU to what you want, then add it back to the layer 2 vlans. This way it will retain the MTU setting even when added back to the layter 2 vlans. This seems like extra steps to me, but I'm sure there is a reason this limitation exists, I just dont see why.
So, if you can not have EVERY DEVICE on your network use JUMBO frames, then you should run the full layer 3 router code (which is free) on your FCX and set the MTU per port. My transition from layer 2 code to layer 3 code was this.
1.) Save a config backup of your switch.
2.) Document everything that is plugged into each port to determine if each device can be set to use JUMBO frames or not.
3.) Document every port's membership in each layer 2 vlan you have.
4.) Get the layer 3 code onto your switch's unused flash location.
5.) Set your switch to boot to the layer 3 code.
6.) Layer 3 code doen't allow an IP address to be assigned to the switch for management. So, either use the out-of-bound management port for managing your switch, or you will have to create a virtual ethernet interface in the vlan that you want to manage it from and assign the ve an IP address.
7.) For each port you want to change the MTU on, remove dual-mode if it's enabled on the port, and remove it from all layer 2 vlans.
8.) Configure your desired port's MTU with the commands:
int eth 1/1/33
ip mtu 1492
9.) Add you port back to all layer 2 vlans, and re-enable dual-mode (if it was enabled from the begining).
I'm so glad I figured this out! Now we can enable High Availability on our Citrix Xen Servers, as they don't support JUMBO on their management interface.
Hope this helps someone else as I could not find this documented anywhere.