Monitoring and Alerting System notification - Rule defCHASSISFLASH_USAGE_90 violated. Obj:CHASSIS 0 [ Edited ]

[chapyn]

Hi buddies,

I keep receiving this message with my Brocade FC5022  16GB switch on my Chassis Flex. How Can I fix this message?

Thanks

[Martin.Sjölin]

Hi,

looks like you are running out of storage on the flash memory used for FOS.  What is your FOS version?

Quick and dirty, you can try the following command (based on 7.4 FOS) (as admin role) to see if it recover some space:

supportsave -R

which will remove any core files. Notice that if you have / had issues and core files are needed to resolve issue, you better run a supportsave to same the core files.

[chapyn]

7.4.1c

[chapyn]

How can I check the realtime usage on my switch?

[Thierry.Zimmermann]

Hello,

mapsSam command will provide you the usages.

Sample:

mapsSam --show flash

[Martin.Sjölin · Accepted solution]

Hi,

can you please login as root and run the following command:

df -h

ls -la /var/log

There are few instances for FC5022  with flash growing over 90%

[chapyn]

Oups wrong accept solution click...

Switch1:root> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 376M 347M 9.4M 97% /
/dev/hda3 194M 17M 167M 9% /core_files
/dev/hda2 376M 193M 163M 54% /mnt
Switch1:root> ls -la /var/log
total 156940
drwxr-xr-x 3 root sys 4096 Sep 28 10:51 ./
drwxr-xr-x 12 root sys 4096 Sep 28 10:52 ../
prw------- 1 root sys 0 Mar 15 2017 authpriv|
prw-rw-rw- 1 root sys 0 Mar 15 2017 authpriv0|
prw-rw-rw- 1 root sys 0 Mar 15 2017 authpriv1|
-rw-r--r-- 1 root root 482 Jun 30 09:59 bc.history
-rw-r--r-- 1 root root 505 Jun 30 09:58 config_library.log
-rw-r--r-- 1 root root 505 Jun 30 09:46 config_library.log.save
-rw-r--r-- 1 root admin 5644 Sep 28 10:51 config_updown.log
-rw-r--r-- 1 root root 5620 Jun 30 09:58 dmesg
-rw-r--r-- 1 root root 19428 Sep 28 10:53 esmd_lib.log
-rw-r--r-- 1 root root 128 Jun 30 09:46 esslog.old
-rw-r--r-- 1 root root 128 Jun 30 09:59 esslog.txt
-rw-r--r-- 1 root root 100 Jun 30 09:59 fabos.log
-rw-r--r-- 1 root root 134 Jun 30 09:46 fdmilog.old
-rw-r--r-- 1 root root 134 Jun 30 09:59 fdmilog.txt
-rw-r--r-- 1 root root 133 Jun 30 09:46 ficulog.old
-rw-r--r-- 1 root root 133 Jun 30 09:59 ficulog.txt
-rw-r--r-- 1 root admin 75480 Sep 28 10:51 fwdl.log
-rw-r--r-- 1 root admin 151 Jun 18 11:25 fwdl.log.save
-rw-r----- 1 root root 1912 Jun 18 11:48 glog_3.dmp
-rw-r----- 1 root admin 78040 Sep 28 13:17 glog_4.dmp
-rw-r--r-- 1 root root 50033 Jun 30 10:00 hasm.log
-rw-r--r-- 1 root root 53010 Jun 30 09:58 hasm.log.save
prw-r----- 1 root root 0 Sep 28 10:52 kmsg|
----r-s--- 1 root root 684 Jun 26 2014 log_1.dmp*
-rw-r----- 1 root root 52530 Sep 28 10:52 messages
-rw-r--r-- 1 root root 124 Jun 30 09:46 nslog.old
-rw-r--r-- 1 root root 124 Jun 30 09:59 nslog.txt
-rw-r--r-- 1 root root 1 Jun 26 2014 ntp_addr_sync_mode
-rw-r--r-- 1 root root 129 Jun 30 09:46 rcslog.old
-rw-r--r-- 1 root root 129 Jun 30 09:59 rcslog.txt
-rw-r--r-- 1 root root 0 Jun 30 09:58 rebootreason
-rw-r--r-- 1 root root 0 Jun 30 09:59 rm.log
-rw-r--r-- 1 root root 0 Jun 30 10:00 ruleslog.0
-rw-r--r-- 1 root root 146 Jun 30 09:46 seclog.old
-rw-r--r-- 1 root root 146 Jun 30 09:59 seclog.txt
-rw------- 1 root root 32076 Jun 30 10:01 slpd.log
-rw-r--r-- 1 root root 160006144 Sep 28 13:15 slpd_xmalloc.log
-rw-r--r-- 1 root root 276 Jun 30 09:46 snmplog.old
-rw-r--r-- 1 root root 276 Jun 30 09:59 snmplog.txt
drwxr-xr-x 2 root root 4096 Jun 18 11:34 sw0/
-rw-r----- 1 root root 1500 Aug 31 08:58 syslog.log
-rw-r--r-- 1 root root 0 Jun 30 09:59 tracecfg.log
-rw-r--r-- 1 root root 6051 Jun 30 10:00 user.log
-rw-rw-r-- 1 root utmp 768 Sep 28 13:17 wtmp
-rw-r--r-- 1 root root 134 Jun 30 09:46 zonelog.old
-rw-r--r-- 1 root root 134 Jun 30 09:59 zonelog.txt
Switch1:root>

I think is releated with this log file : slpd_xmalloc.log what's that?

here anotehr switch which i have problems too

Switch2:root> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 376M 357M 0 100% /
/dev/hda3 194M 21M 163M 11% /core_files
/dev/hda1 376M 194M 162M 54% /mnt

[Martin.Sjölin]

Hi,

notice a few mail, but no posts, and since this is end of my working day I post the below:

if you a very large /var/log/slpd_xmalloc.log file, you have hit DEFECT000595324 fixed in 7.4.2, 8.0.2, and 8.1.0:

Symptom:    CF space utilization may go to 100%, mostly from large file /var/log/slpd_xmalloc.log.
Condition:  This may be seen on Brocade BR6547 switch.

Try to delete the fslpd_xmalloc.log file, notice that it involved with the management software FSM(?) or CMM(?) for the FC5022, to it will start growing again.  For a real fix, contact your OEM or possible upgrade to a release with a fix.

[chapyn]

I deleted the file

switch2:root> rm /var/log/slpd_xmalloc.log

and I still have the problem 100% use...

switch2:root> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 376M 357M 0 100% /
/dev/hda3 194M 21M 163M 11% /core_files
/dev/hda1 376M 194M 162M 54% /mnt

[Martin.Sjölin · Accepted solution]

Hi,

since the slp* (needed for the mgmt of the switch via the chassis) probably have the file open (in spite of it being open, it will continie to grow.  We should have done a haReboot directly after the rm. Options are

1. haReboot - which might not work since the flash is at 100% (and it needs to write the current state some where)

2. restart the slp* daemon which will close the file and open a new. I have no FC5022 to test so I am not sure what might happen at restart time. As root:

sh /etc/rc.d/init.d/slpd restart  #(or stop and then start).  

Then check with df -h and ls -l /var/log /mnt/var/log if files are gone. You might have to do rm /mnt/var/log/slpd_xmalloc.log to clean out the secondary partition

Option 2 is feel safer for me.