Fibre Channel (SAN)

Reply
Occasional Contributor
Posts: 5
Registered: ‎02-17-2015

latency with vmware virtual machines.

We are experiencing intermittent connectivity on our vms both on the network side and on the storage side. When a user complains about a specific virtual machine I ping the machine and it will have latency up to 3983ms but when I ping the blade that the machine resides on its connectivity seems unaffected in the ping status.  I am not for sure if this is a host issue or if something else is going on. Not only does remote desktop sessions fail when it does work the machine has extreme lag to the point of unavailability.

.

We currently have 28 hosts in our main cluster. I think VMWARE's recommended limit  is 33 hosts which we are in that range. I don't know where to begin troubleshooting this issue considering the physical infrastructure seems to be working properly. 

 

Does anyone have any suggestions that would be a sound process in diagnosing this problem? 

 

Are there any network diagnostic tools for virtual environments to assist in troubleshooting? 

 

 

 

 

Regular Contributor
Posts: 164
Registered: ‎05-11-2011

Re: latency with vmware virtual machines.

Hi,

 

you are here in the SAN-Community. Are you talking about IP-Latency?

 

 

Kind Regards
Ralf
Occasional Contributor
Posts: 5
Registered: ‎02-17-2015

Re: latency with vmware virtual machines.

[ Edited ]

Both we are expericing problems with both. We learned there was a storage issue when we were preparing for our exchange servers. We are over the 10ms that microsoft reccomends for write times somewhere around 40-50ms response time.

 

 Also network connectvity to our virtual servers can be slow and laggish at different times. We think it might be all releted to the storage connectivity.If the server does not have good access times to its disk then that would affect its ability to serve data over the network correct?

 

Vmware is saying that it looks like the hosts has storage issues. 

Contributor
Posts: 54
Registered: ‎07-08-2011

Re: latency with vmware virtual machines.

 

Hi,

 

     Virtual Machines not directly connected to SAN. So proper investigation needs to be done on VM level which includes Virtual machine,ESX server , VM Tools and etc.

 

" Vmware is saying that it looks like the hosts has storage issues " == > Please be specific. ( Proper output from VMware support/investigation.

Occasional Contributor
Posts: 5
Registered: ‎02-17-2015

Re: latency with vmware virtual machines.

The vmkernel log is showing these errors:

 2015-02-17T23:05:16.033Z cpu32:1082860)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c34824c0, 34126) to dev "naa.60002ac000000000000000e8000062a5" on path "vmhba2:C0:T1:L31" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

 

2015-02-17T23:05:16.033Z cpu32:1082860)ScsiDeviceIO: 2338: Cmd(0x4136c34824c0) 0x89, CmdSN 0x100b9 from world 34126 to dev "naa.60002ac000000000000000e8000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

2015-02-17T23:05:55.462Z cpu34:32811)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c263d480, 34126) to dev "naa.60002ac000000000000000eb000062a5" on path "vmhba2:C0:T1:L34" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

2015-02-17T23:05:55.462Z cpu34:32811)ScsiDeviceIO: 2338: Cmd(0x4136c263d480) 0x89, CmdSN 0x626af from world 34126 to dev "naa.60002ac000000000000000eb000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

2015-02-17T23:18:40.406Z cpu34:2974725)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c2921d80, 34126) to dev "naa.60002ac000000000000000e6000062a5" on path "vmhba1:C0:T1:L29" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

2015-02-17T23:18:40.406Z cpu34:2974725)ScsiDeviceIO: 2338: Cmd(0x4136c2921d80) 0x89, CmdSN 0x10386 from world 34126 to dev "naa.60002ac000000000000000e6000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

2015-02-17T23:25:05.437Z cpu8:4812392)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x412ec8bd3680, 34263) to dev "naa.60002ac00000000000000064000062a5" on path "vmhba1:C0:T1:L14" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

2015-02-17T23:25:05.437Z cpu8:4812392)ScsiDeviceIO: 2338: Cmd(0x412ec8bd3680) 0x89, CmdSN 0x17f from world 34263 to dev "naa.60002ac00000000000000064000062a5" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

Valued Contributor
Posts: 761
Registered: ‎06-11-2010

Re: latency with vmware virtual machines.

Hi there,

 

the event you posted:

 

Spoiler
 2015-02-17T23:05:16.033Z cpu32:1082860)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x89 (0x4136c34824c0, 34126) to dev "naa.60002ac000000000000000e8000062a5" on path "vmhba2:C0:T1:L31" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0. Act:NONE

 

Cmd 0x89 stands for the COMPARE AND WRITE command, used with VAAI ATS.

SCSI condition H:0x0 D:0x2 P:0x0 stands for CHECK CONDITION which points at the Target device.

SENSE CODE 0xe 0x1d 0x0 stands for MISCOMPARE DURING VERIFY OPERATION

 

The VAAI ATS pimitive compares a on-disk sector to a given buffer and if the information in both is not the same, it reports the above event. 

 

Are there any other events before/after this one?

Does your storage array support VAAI ATS operatives?

 

If this behavior is impacting your ESX performance and if the SAN is not reporting any error I would have the array checked by the storage vendor.

 

 

Rgds,

Felipon

 

Occasional Contributor
Posts: 5
Registered: ‎02-17-2015

Re: latency with vmware virtual machines.

We have had our storage vendor check out the device. We are using HP 3PAR storage and it seems that each device on each end of the circuit reports fine. We have also ran the Brocade Analyzer SAN Health utility and are awaiting to hear from Brocade on this.

New Member
Posts: 1
Registered: ‎03-10-2015

Re: latency with vmware virtual machines.

@konfiguratonking let me know if you have received the analysis from brocade for "H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0" what were the steps taken to resolve the issue?

 

New Member
Posts: 1
Registered: ‎03-31-2015

Re: latency with vmware virtual machines.

any update on this? having the same message with 3PAR 7400 3.2.1

New Member
Posts: 1
Registered: ‎11-16-2011

Re: latency with vmware virtual machines.

We have experienced same problem. Did you ever get solution to this?

 

 

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.