ufs_readdir: bad dir warning messages
We observed below warning messages on server:
Jan 12 16:15:15 TEMM
ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 273728,
fs = /var/tmp/tempdir1
Jan 12 16:15:15 TEMM ufs: [ID
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 273729, fs =
/var/tmp/tempdir1
Jan 12 16:15:15 TEMM ufs: [ID
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 71432, fs =
/var/tmp/tempdir1
Jan 12 16:15:23 TEMM ufs: [ID
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 93184, fs =
/var/tmp/tempdir1
Jan 12 16:15:23 TEMM ufs: [ID
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 1124043, fs =
/var/tmp/tempdir1
Jan 12 16:16:05 TEMM ufs: [ID
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 681448, fs =
/var/tmp/tempdir1
We have checked the disk usage, however it was fine. TEMM:/var/opt/mediation/dumpfiles# df -kkh Filesystem size used avail capacity Mounted on /dev/dsk/c0t1d0s0 229G 91G 135G 41% / /devices 0K 0K 0K 0% /devices ctfs 0K 0K 0K 0% /system/contract proc 0K 0K 0K 0% /proc mnttab 0K 0K 0K 0% /etc/mnttab swap 63G 1008K 63G 1% /etc/svc/volatile objfs 0K 0K 0K 0% /system/object sharefs 0K 0K 0K 0% /etc/dfs/sharetab /usr/lib/libc/libc_hwcap1.so.1 229G 91G 135G 41% /lib/libc.so.1 fd 0K 0K 0K 0% /dev/fd swap 63G 63M 63G 1% /tmp swap 63G 36K 63G 1% /var/run /dev/dsk/c0t2d0s0 274G 233G 39G 86% /var/tmp/tempdir /dev/dsk/c0t3d0s0 274G 126G 146G 47% /var/tmp/tempdir1 TEMM:/var/opt/mediation/dumpfiles#date Thu Jan 12 16:20:52 GST 2017 We have checked the server and found that we are getting warning messages for below partition. /dev/dsk/c0t2d0s0 274G 233G 39G 86% /var/tmp/tempdir The current usage of the partition is 72% and now we are no more getting these error messages. TEMM:/var/adm# df -h /var/tmp/tempdir Filesystem size used avail capacity Mounted on /dev/dsk/c0t2d0s0 274G 193G 78G 72% /var/tmp/tempdir We had faced last warning message last on 15th Jan. Jan 15 11:52:03 TEMM ufs: [ID The device was last mounted on 7th Dec. /var/tmp/tempdir on /dev/dsk/c0t2d0s0 The issue was caused by high filesystem usage which lead to either inode number shortage or FS corruption. TEMM:/var/adm# df -g /var/tmp/tempdir 31805914 free If the issue is with inode shortage, it is automatically addressed by reducing the FS usage this However to make sure there is no other issue with Filesystem(like corruption) with this hard disk(with only one partition) we recommend you to run FSCK on this partition/disk. 2. c0t2d0 <LSI cyl 36348 alt 2 hd 255 sec 63> Part 0 unassigned wm 1 – 36347 278.43GB (36347/0/0) 583914555
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 5043585, fs =
/var/tmp/tempdir
read/write/setuid/devices/rstchown/intr/largefiles/logging/xattr/onerror=panic/dev=840100 on Wed Dec 7 06:36:49 2016
/var/tmp/tempdir
(/dev/dsk/c0t2d0s0 ): 8192
block size 1024 frag size
575075616 total blocks 170182316 free blocks 164431560
available 34594560 total files
files 8651008 filesys
id ufs fstype
0x00000004
flag
255 filename length
time.
/pci@0,0/pci8086,3c08@3/pci1000,9262@0/sd@2,0
Tag Flag Cylinders Size Blocks
Steps to run FSCK (make sure this partition is not used by any process i.e. during Maintenance window) 1. Umount the partition. # umount /var/tmp/tempdir 2. Run Filesystem check. #fsck –y /dev/rdsk/c0t2d0s0 3. Mount the partition again. # mount /var/tmp/tempdir
These issues appeared only
due to high disk utilization during this time there was a CDR processing for
live CCN nodes done on the test EMM node.
There are no soft or hard
errors reported on the disk. If this issue occurs while the disk is not fully
used then we have the procedure so will run.
For now since the issue is
not occurring so we will close the case.