Ethernet Switches & Routers

Reply
Contributor
Posts: 44
Registered: ‎07-08-2011

FCX Jumbo Frames per Port and iSCSI traffic

I was reading the discussion and linked article here and saw some great topics, but alas more questions are left: http://community.brocade.com/thread/4664?tstart=30#12919

We have the following setup:    Firewall ---- L3Switch1 ----------------------- L3Switch2 ---- iSCSISW2

                                                                      |                                        |

                                                                      ------ iSCSISW1                  --------  LANSW2

See attached picture in case ASCII doesn't show up well.

So basically our firewall comes down into an L3 switch. Customers come through this connection. Off that L3 switch is another switch with just storage connected to it. That storage is the replication from other storage elsewhere. That first L3 switch serves as  the core for switches and devices directly attached to it. however connected via fiber across the way is another L3 switch. Off that switch are 2 switches, 1 for mroe iSCSI storage and 1 for regular LAN traffic on servers.

The 2nd iSCSI switch (i2) has jumbo frames enabled. It is enabled across the board since it's L2. the value is actually 10222 mtu according to the statistics. L3Switch has VE's for the connections to the i2 switch and I want to enable Jumbo frames for just those ports. Not for the ports that connect to the LAN Switch with servers on it. This seems doable.

However, since storage off of i2 replicates to storage on i1, then jumbo frames must be enabled on the fiber port connections. Since that path is also the path the LAN switch data takes to get back to the customers outside the firewall, isn't there huge overhead for 1500 MTU data to be rolled up into 10222 data? Additionally when that data continues through the first L3 switch and up to the firewall, the firewall (and the internet paths back to the customer) do not support jumbo frames and thus they will now have to break up that MTU 10222 frame into 1500. Causing further overhead. Is there any real way around this without telling me to just connect LanSW2 right off of L3Switch1? That is not possible in our situation.

Lastly, it turns out i2 is an older switch where the MTU value when jumbo frames is turned on globally, is 10222. however the newer L3 switch is telling me the max value I can set the MTU to is 10218. Both Brocade. So now it seems that the i2 switch is going to package it bigger than the L3 switches can handle and thus it will break it up or drop it. Am I missing something here or does this just the way it is. If I could set the MTU value on the L2 switches even globally I would set them slightly smaller than 10218 so that there's enough left for those switches when they package it up.


See attached

Thanks!

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX Jumbo Frames per Port and iSCSI traffic

Hi JackOfTrades,

     Who should have a L3 VLAN assigned to carry the iSCSI traffic  betwwen the i1 and i2 of your switches.

The data traffic (non iSCSI) should be in another VLAN at standard MTU.

You server should have two nic, 1 in the data VLAN and one in the iSCSI VLAN.

In this way clients would talk to the servers on standard MTU.

The servers and storage should be configured on the iSCSI VLAN for a MTU of 9k - in this way all switches can handle that size MTU.

Hope this answers your questions and meets your requirements.

Thanks

Michael.

Contributor
Posts: 44
Registered: ‎07-08-2011

Re: FCX Jumbo Frames per Port and iSCSI traffic

We do have an L3 VLAN assigned between the two switches, but they are not physically connected directly. Both travel through the L3 switches.

the data triffic is in another VLAN.

the servers have 4 NICs, 2 for data and 2 for iSCSI, both for redundancy protection.

Are you suggesting that data on one vlan can travel over the two ports using one MTU setting while iscsi data can travel over the same 2 ports, but in a different VLAN and both have different MTU values?

I'm still a bit confused on how to set different MTU values. It looks like you have to put the "jumbo" line in the config to be able to set different MTU values per port. I definitely don't want it on for the data ports. Or is really just a setting per vlan and not per port?    

Thanks!

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX Jumbo Frames per Port and iSCSI traffic

hi JackOfTrades,

We do have an L3 VLAN assigned between the two switches, but they are not physically connected directly. Both travel through the L3 switches.

     If their are other L3 switches in the path and you do not have control over them, then it will depend on IF they support jumbo frame.

Are you suggesting that data on one vlan can travel over the two ports using one MTU setting while iscsi data can travel over the same 2 ports, but in a different VLAN and both have different MTU values?

     Yep that is how it should be.

I'm still a bit confused on how to set different MTU values. It looks like you have to put the "jumbo" line in the config to be able to set different MTU values per port. I definitely don't want it on for the data ports. Or is really just a setting per vlan and not per port?

     Yep correct.  The setting of the MTU should be on the VE for the VLAN not the port.

Thanks Michael.

Contributor
Posts: 44
Registered: ‎07-08-2011

Re: FCX Jumbo Frames per Port and iSCSI traffic

Forgive me for being dense, but can you give me an example of what I need to do from the CLI?

When I do a Show Run I have vlan entries like the following:

vlan 231 name VLAN-231 by port
tagged ethe 1/1/41 to 1/1/48 ethe 2/1/41 to 2/1/48
router-interface ve 231
!

later I have corresponding interface ethernet lines like so:

interface ethernet 1/1/41
port-name Trunk1
!

This is a 4 port trunk to a switch dedicated for iSCSI SAN connections.

Lastly, I have lines like the following:

interface ve 231
port-name VE-231
ip address 10.0.31.1 255.255.255.0
!

It appears as if there's no MTU value. However through testing I saw that if I set the MTU value to something other than 10218 (the max) then it shows the entry in the config ve entry (ex: IP MTU 1500). But if you set it to the max, it disappears. Maybe when the max is set it disappears so that it takes the default global value?

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX Jumbo Frames per Port and iSCSI traffic

Hi,

It appears as if there's no MTU value. However through testing I saw that if I set the MTU value to something other than 10218 (the max) then it shows the entry in the config ve entry (ex: IP MTU 1500). But if you set it to the max, it disappears. Maybe when the max is set it disappears so that it takes the default global value?

Not sure here, but a simple test will find us the answer.

do a 'show interface ve 231'

Should see something like

ve231 is up, line protocol is up
  Hardware is Virtual Ethernet, address is 0012.f2cf.1200 (bia 0012.f2cf.1200)
  Port name is VE-port-name
  Internet address is a.b.c.d/y, IP MTU value bytes, encapsulation ethernet

If the shown IP MTU is the MAX then yes it is enable (and not shown in the config - please let me know)

Ok what you should do is set jumbo on the global level on all switches in the path. (Think you have done that now)

Set IP MTU 1500 on the VLAN's you do not want jumbo frame on.

Set IP MTU 9216 on the VE for your iSCSI VLAN, on all switches that carriy this VLAN.

Set jumbo frame support to 9216 on all storage and server's connect to the iSCSI VLAN. (Note choose 9216 if ALL devices support this value, if you find one server or storage that only supports 9000 then change all to 9000)

When that is done, use "ping -f -l 9216 10.0.31.1" from a server connected on the iSCSI VLAN, this should work.

Then "ping -f -l 9216 a.b.c.d" from a server connected on the DATA VLAN (where a.b.c.d is the ip address of the VE for the DATA VLAN), this should  fail - and that is what you want.


Config wise

jumbo

vlan 231 name VLAN-231 by port
tagged ethe 1/1/41 to 1/1/48 ethe 2/1/41 to 2/1/48
router-interface ve 231
!

interface ethernet 1/1/41
port-name Trunk1
!

interface ve 231
port-name VE-231
ip address 10.0.31.1 255.255.255.0

ip mtu 9216
!

interface ve DATA-VLAN-ID
port-name something
ip address 10.0.31.1 255.255.255.0

ip mtu 1500
!

Thanks

Michael.

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX Jumbo Frames per Port and iSCSI traffic

Hi JackOfTrades,

     Does the above answer you question?

Thanks

Michael.

Contributor
Posts: 44
Registered: ‎07-08-2011

Re: FCX Jumbo Frames per Port and iSCSI traffic

Hi, Sorry for the delayed response. Been at the datacenter all week doing installs. I just got a similar switch so I can test, so I'll use that and hceck it out. too risky on the production gear.

Thanks!         

Super Contributor
Posts: 1,087
Registered: ‎12-13-2009

Re: FCX Jumbo Frames per Port and iSCSI traffic

no ptoblems, please let us know the results after testing.

Thanks

Michael.

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook