ufs_readdir: bad dir warning messages

We observed below warning messages on server:

Jan 12 16:15:15 TEMM ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 273728, fs = /var/tmp/tempdir1

Jan 12 16:15:15 TEMM ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 273729, fs = /var/tmp/tempdir1

Jan 12 16:15:15 TEMM ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 71432, fs = /var/tmp/tempdir1

Jan 12 16:15:23 TEMM ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 93184, fs = /var/tmp/tempdir1

Jan 12 16:15:23 TEMM ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 1124043, fs = /var/tmp/tempdir1

Jan 12 16:16:05 TEMM ufs: [ID 703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 681448, fs = /var/tmp/tempdir1

Analysis

We have checked the disk usage, however it was fine.

TEMM:/var/opt/mediation/dumpfiles# df -kkh

Filesystem size used avail capacity Mounted on

/dev/dsk/c0t1d0s0 229G 91G 135G 41% /

/devices 0K 0K 0K 0% /devices

ctfs 0K 0K 0K 0% /system/contract

proc 0K 0K 0K 0% /proc

mnttab 0K 0K 0K 0% /etc/mnttab

swap 63G 1008K 63G 1% /etc/svc/volatile

objfs 0K 0K 0K 0% /system/object

sharefs 0K 0K 0K 0% /etc/dfs/sharetab

/usr/lib/libc/libc_hwcap1.so.1 229G 91G 135G 41% /lib/libc.so.1

fd 0K 0K 0K 0% /dev/fd

swap 63G 63M 63G 1% /tmp

swap 63G 36K 63G 1% /var/run

/dev/dsk/c0t2d0s0 274G 233G 39G 86% /var/tmp/tempdir

/dev/dsk/c0t3d0s0 274G 126G 146G 47% /var/tmp/tempdir1

TEMM:/var/opt/mediation/dumpfiles#date

Thu Jan 12 16:20:52 GST 2017

We have checked the server and found that we are getting warning messages for below partition.

/dev/dsk/c0t2d0s0 274G 233G 39G 86% /var/tmp/tempdir

The current usage of the partition is 72% and now we are no more getting these error messages.

TEMM:/var/adm# df -h /var/tmp/tempdir

Filesystem size used avail capacity Mounted on

/dev/dsk/c0t2d0s0 274G 193G 78G 72% /var/tmp/tempdir

We had faced last warning message last on 15th Jan.

Jan 15 11:52:03 TEMM ufs: [ID
703900 kern.warning] WARNING: ufs_readdir: bad dir, inumber = 5043585, fs =
/var/tmp/tempdir

The device was last mounted on 7th Dec.

/var/tmp/tempdir on /dev/dsk/c0t2d0s0
read/write/setuid/devices/rstchown/intr/largefiles/logging/xattr/onerror=panic/dev=840100 on Wed Dec 7 06:36:49 2016

The issue was caused by high filesystem usage which lead to either inode number shortage or FS corruption.

TEMM:/var/adm# df -g
/var/tmp/tempdir

/var/tmp/tempdir
(/dev/dsk/c0t2d0s0 ): 8192
block size 1024 frag size
575075616 total blocks 170182316 free blocks 164431560
available 34594560 total files

31805914 free
files 8651008 filesys
id ufs fstype
0x00000004
flag
255 filename length

If the issue is with inode shortage, it is automatically addressed by reducing the FS usage this
time.

However to make sure there is no other issue with Filesystem(like corruption) with this hard disk(with only one partition) we recommend you to run FSCK on this partition/disk.

2. c0t2d0 <LSI cyl 36348 alt 2 hd 255 sec 63>


/pci@0,0/pci8086,3c08@3/pci1000,9262@0/sd@2,0

Part
Tag Flag Cylinders Size Blocks

0 unassigned wm 1 – 36347 278.43GB (36347/0/0) 583914555



Steps to run FSCK (make sure this partition is not used by any process i.e. during Maintenance window)

1. Umount the partition.

# umount /var/tmp/tempdir

2. Run Filesystem check.

#fsck –y /dev/rdsk/c0t2d0s0

3. Mount the partition again.

# mount /var/tmp/tempdir



These issues appeared only due to high disk utilization during this time there was a CDR processing for live CCN nodes done on the test EMM node.

There are no soft or hard errors reported on the disk. If this issue occurs while the disk is not fully used then we have the procedure so will run.

For now since the issue is not occurring so we will close the case.

Leave a Reply

Your email address will not be published.