Ethernet Switches & Routers

Reply
New Contributor
Posts: 3
Registered: ‎12-10-2010

Traffic Manager (TM) Internal Error

Yesterday, i saw a strange problem with my BigIron RX-16, one of its copper module (slot-4) stopped functioning without any alarm, warning messges ...

All ports on module were up, but could not communicate with each other ... on detailed troubleshooting i saw following alams ...

INFO:Oct 17 11:01:44: TM Internal Error: LP2/TM1  (Reg: 0x2838, Value: 0x8c080)
INFO:Oct 17 11:01:43: TM Internal Error: LP4/TM2  (Reg: 0x2838, Value: 0x8e48f)
INFO:Oct 17 11:01:41: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x8dd06)
INFO:Oct 17 11:01:38: TM Internal Error: LP2/TM2  (Reg: 0x2838, Value: 0x8fcab)
INFO:Oct 17 11:01:37: TM Internal Error: LP1/TM2  (Reg: 0x2838, Value: 0x8f6a0)
INFO:Oct 17 11:01:35: TM Internal Error: LP2/TM1  (Reg: 0x2838, Value: 0x883e3)
INFO:Oct 17 11:01:35: TM Internal Error: LP4/TM2  (Reg: 0x2838, Value: 0x888cc)
INFO:Oct 17 11:01:32: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x88c17)
INFO:Oct 17 11:01:29: TM Internal Error: LP2/TM2  (Reg: 0x2838, Value: 0x8fbdc)
INFO:Oct 17 11:01:28: TM Internal Error: LP1/TM2  (Reg: 0x2838, Value: 0x8d8ff)
INFO:Oct 17 11:01:27: LP2/NP1 packet path diagnostic failure (tx 3883980, rx 3883974, tx_err 0, rx_err 5, rx_data_err 0)
INFO:Oct 17 11:01:27: TM Internal Error: LP2/TM1  (Reg: 0x2838, Value: 0x88f58)
INFO:Oct 17 11:01:26: LP4/NP2 packet path diagnostic failure (tx 3883965, rx 3883959, tx_err 0, rx_err 5, rx_data_err 0)
INFO:Oct 17 11:01:26: TM Internal Error: LP4/TM2  (Reg: 0x2838, Value: 0x86a3c)
INFO:Oct 17 11:01:23: LP4/NP1 packet path diagnostic failure (tx 3883965, rx 3883959, tx_err 0, rx_err 5, rx_data_err 0)
INFO:Oct 17 11:01:23: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x8152f)
INFO:Oct 17 11:01:20: LP2/NP2 packet path diagnostic failure (tx 3883979, rx 3883015, tx_err 0, rx_err 958, rx_data_err 2)
INFO:Oct 17 11:01:20: TM Internal Error: LP2/TM2  (Reg: 0x2838, Value: 0x80c31)
INFO:Oct 17 11:01:19: LP1/NP2 packet path diagnostic failure (tx 3883883, rx 3883877, tx_err 0, rx_err 5, rx_data_err 0)
INFO:Oct 17 11:01:19: TM Internal Error: LP1/TM2  (Reg: 0x2838, Value: 0x8042d)
INFO:Oct 17 11:01:18: TM Internal Error: LP2/TM1  (Reg: 0x2838, Value: 0x8022f)
INFO:Oct 17 11:01:17: TM Internal Error: LP4/TM2  (Reg: 0x2838, Value: 0x801a8)
INFO:Oct 17 11:01:15: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x80454)
INFO:Oct 17 11:01:06: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x80724)
INFO:Oct 17 11:00:59: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x808da)
INFO:Oct 17 11:00:48: TM Internal Error: LP4/TM1  (Reg: 0x2838, Value: 0x80a2d)
ALARM:Oct 17 09:39:38: TM Internal Error: LP2/TM2  (Reg: 0x444, Value: 0x4000030)
ALARM:Oct 17 09:36:42: TM Internal Error: LP2/TM2  (Reg: 0x444, Value: 0x4000038)
ALARM:Oct 17 07:47:53: TM Internal Error: LP2/TM2  (Reg: 0x444, Value: 0x4000030)
ALARM:Oct 17 00:49:52: TM Internal Error: LP2/TM2  (Reg: 0x444, Value: 0x4000038)
ALARM:Oct 16 23:55:27: TM Internal Error: LP2/TM2  (Reg: 0x444, Value: 0x4000030)
ALARM:Oct 16 21:43:36: TM Internal Error: LP2/TM2  (Reg: 0x444, Value: 0x4000038)

Notebaly, all SFM-to-Slot links were fine, see ouput of "show snm-links by-lp"

Slot 1/FAP1/Link 1 (N)-- SNM2/FE3/Link30 : up
Slot 1/FAP1/Link 3 (N)-- SNM1/FE1/Link30 : up
Slot 1/FAP1/Link 4 (N)-- SNM3/FE1/Link30 : up
Slot 1/FAP1/Link 5 (A)-- SNM2/FE1/Link30 : up
Slot 1/FAP1/Link 6 (A)-- SNM2/FE2/Link32 : up
Slot 1/FAP1/Link 7 (A)-- SNM1/FE3/Link30 : up
Slot 1/FAP1/Link 8 (N)-- SNM1/FE2/Link32 : up
Slot 1/FAP1/Link 9 (N)-- SNM3/FE2/Link32 : up

Slot 1/FAP2/Link 2 (N)-- SNM2/FE3/Link54 : up
Slot 1/FAP2/Link 3 (A)-- SNM2/FE1/Link54 : up
Slot 1/FAP2/Link 4 (A)-- SNM2/FE2/Link33 : up
Slot 1/FAP2/Link 5 (A)-- SNM1/FE3/Link54 : up
Slot 1/FAP2/Link 6 (N)-- SNM1/FE1/Link54 : up
Slot 1/FAP2/Link 7 (N)-- SNM3/FE1/Link54 : up
Slot 1/FAP2/Link 8 (N)-- SNM1/FE2/Link33 : up
Slot 1/FAP2/Link 9 (N)-- SNM3/FE2/Link33 : up

Slot 2/FAP1/Link 1 (N)-- SNM2/FE3/Link22 : up
Slot 2/FAP1/Link 3 (N)-- SNM1/FE1/Link22 : up
Slot 2/FAP1/Link 4 (N)-- SNM3/FE1/Link22 : up
Slot 2/FAP1/Link 5 (A)-- SNM2/FE1/Link22 : up
Slot 2/FAP1/Link 6 (A)-- SNM2/FE2/Link 3 : up
Slot 2/FAP1/Link 7 (A)-- SNM1/FE3/Link22 : up
Slot 2/FAP1/Link 8 (N)-- SNM1/FE2/Link 3 : up
Slot 2/FAP1/Link 9 (N)-- SNM3/FE2/Link 3 : up

Slot 2/FAP2/Link 2 (N)-- SNM2/FE3/Link62 : up
Slot 2/FAP2/Link 3 (A)-- SNM2/FE1/Link62 : up
Slot 2/FAP2/Link 4 (A)-- SNM2/FE2/Link62 : up
Slot 2/FAP2/Link 5 (A)-- SNM1/FE3/Link62 : up
Slot 2/FAP2/Link 6 (N)-- SNM1/FE1/Link62 : up
Slot 2/FAP2/Link 7 (N)-- SNM3/FE1/Link62 : up
Slot 2/FAP2/Link 8 (N)-- SNM1/FE2/Link62 : up
Slot 2/FAP2/Link 9 (N)-- SNM3/FE2/Link62 : up

Slot 3/FAP1/Link 1 (N)-- SNM2/FE3/Link13 : up
Slot 3/FAP1/Link 3 (N)-- SNM1/FE1/Link13 : up
Slot 3/FAP1/Link 4 (N)-- SNM3/FE1/Link13 : up
Slot 3/FAP1/Link 5 (A)-- SNM2/FE1/Link13 : up
Slot 3/FAP1/Link 6 (A)-- SNM2/FE2/Link26 : up
Slot 3/FAP1/Link 7 (A)-- SNM1/FE3/Link13 : up
Slot 3/FAP1/Link 8 (N)-- SNM1/FE2/Link26 : up
Slot 3/FAP1/Link 9 (N)-- SNM3/FE2/Link26 : up

Slot 3/FAP2/Link 2 (N)-- SNM2/FE3/Link37 : up
Slot 3/FAP2/Link 3 (A)-- SNM2/FE1/Link37 : up
Slot 3/FAP2/Link 4 (A)-- SNM2/FE2/Link39 : up
Slot 3/FAP2/Link 5 (A)-- SNM1/FE3/Link37 : up
Slot 3/FAP2/Link 6 (N)-- SNM1/FE1/Link37 : up
Slot 3/FAP2/Link 7 (N)-- SNM3/FE1/Link37 : up
Slot 3/FAP2/Link 8 (N)-- SNM1/FE2/Link39 : up
Slot 3/FAP2/Link 9 (N)-- SNM3/FE2/Link39 : up

Slot 4/FAP1/Link 1 (N)-- SNM2/FE3/Link 7 : up
Slot 4/FAP1/Link 3 (N)-- SNM1/FE1/Link 5 : up
Slot 4/FAP1/Link 4 (N)-- SNM3/FE1/Link 5 : up
Slot 4/FAP1/Link 5 (A)-- SNM2/FE1/Link 5 : up
Slot 4/FAP1/Link 6 (A)-- SNM2/FE2/Link 7 : up
Slot 4/FAP1/Link 7 (A)-- SNM1/FE3/Link 7 : up
Slot 4/FAP1/Link 8 (N)-- SNM1/FE2/Link 7 : up
Slot 4/FAP1/Link 9 (N)-- SNM3/FE2/Link 7 : up

Slot 4/FAP2/Link 2 (N)-- SNM2/FE3/Link47 : up
Slot 4/FAP2/Link 3 (A)-- SNM2/FE1/Link45 : up
Slot 4/FAP2/Link 4 (A)-- SNM2/FE2/Link58 : up
Slot 4/FAP2/Link 5 (A)-- SNM1/FE3/Link47 : up
Slot 4/FAP2/Link 6 (N)-- SNM1/FE1/Link45 : up
Slot 4/FAP2/Link 7 (N)-- SNM3/FE1/Link45 : up
Slot 4/FAP2/Link 8 (N)-- SNM1/FE2/Link58 : up
Slot 4/FAP2/Link 9 (N)-- SNM3/FE2/Link58 : up


Moreover output of "show chassis"     

Module                             Status      Ports  Starting MAC
M1 (upper): RX-BI-MR Management Module     Active
M2 (lower): RX-BI-MR Management Module     Standby (Ready State)
F1: RX-BI-SFM3 Switch Fabric Module         OK
F2: RX-BI-SFM3 Switch Fabric Module         OK
F3: RX-BI-SFM3 Switch Fabric Module         OK
F4: RX-BI-SFM3 Switch Fabric Module         not present
S1: RX-BI-24C 24-port 1 GbE Copper Module  CARD_STATE_UP    24   001b.ed27.dc00
S2: RX-BI-24C 24-port 1 GbE Copper Module  CARD_STATE_UP    24   001b.ed27.dc00
S3: RX-BI-24C 24-port 1 GbE Copper Module  CARD_STATE_UP    24   001b.ed27.dc00
S4: RX-BI-24C 24-port 1 GbE Copper Module  CARD_STATE_UP    24   001b.ed27.dc00
S5:
S6:
S7:
S8:
S9:
S10:
S11:
S12:
S13:
S14:
S15:
S16:

regards

thanks in advance ...

Frequent Contributor
Posts: 117
Registered: ‎07-26-2010

Re: Traffic Manager (TM) Internal Error

Hi.

I saw and see those things on NI MLX-16.

This could also be caused by high load of multicast traffic.

In some of my cases the NI MLX software tried to reset the LP after three times TM errors on one LP, but most times failed to do.

I have done two things:

1. software update because Brocade has several fixes on TM errors

2. rma'ed the LP

Hope this helps a little bit.

regards

Karl

New Contributor
Posts: 3
Registered: ‎12-10-2010

Re: Traffic Manager (TM) Internal Error

Thanks Karl,

Ok, i will upgrade it to ironware 2.7.2f release from existing 2.7.1b .... and what will let you know ...

i have two questions

1. Does TM internal error indicates some hardware failure on the copper module (BI-RX-24C) or it is mere a software issue (as you said that it is multicast related)

2. Some post suggested that "cycling the module or SFM would resolve the issue"

ref: http://www.gossamer-threads.com/lists/nsp/foundry/24874

do you agree to this ...

regards

Frequent Contributor
Posts: 117
Registered: ‎07-26-2010

Re: Traffic Manager (TM) Internal Error

Well, it could be "everything":

-- LP hardware issue

-- software issue

-- SFM hardware issue

-- chassis backplane issue

I had everything of this at my customer already on NI MLX.

I would suggest:

1. reseat the LP in case

2. clear tm log

3. check "show tm log" on a regular basis for new errors on this LP

4. maybe the command "dm rw-snm all all get-link-status b" is available on a BigIrin RX also. You should get all links and its status; check for Errors.

repeat the command.

Be aware that this are inofficial debug commands; no one is responsible if things go wrong on executing it.

Link22: Sig Lock Yes, Leaky Bucket 63
Link23: Sig Lock Yes, Leaky Bucket 63
Link24: Sig Lock Yes, Leaky Bucket 63 (ERR: CRC Misalig CodeGroup)
Link25: Sig Lock Yes, Leaky Bucket 63 (ERR: CRC Misalig CodeGroup)
Link27: Sig Lock Yes, Leaky Bucket 63
Link28: Sig Lock Yes, Leaky Bucket 63

5. You could disable all except one SFM

clear tm log

check show tm log

check with all SFMs one after the other.

If all fails..... case at Brocade.

regards

Karl

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.

vADC is now Pulse Secure
Download FREE NVMe eBook