Solved: kernel: INFO: task df:147233 blocked for more than 120 seconds.
Today in this post we will see how to solve the error message “df: task blocked for more than — seconds”.
Lets view the error message and how to solve this.
I. Error Message
kernel: INFO: task df:147233 blocked for more than 120 seconds.
II. Analysis: Where to check the error messages.
Check out the respective messages file.
[root@ngelinux001 log]# cat messages-20220727 | grep -i "blocked for more than" | more Jul 25 23:20:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:22:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:24:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:26:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:28:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:30:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:32:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:34:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:36:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:38:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. [root@ngelinux001 log]# In some cases, like my case, there could be kernel panic due to “df” waiting to mount the partition i.e. failing to mount the hard NFS mount on server. Jul 25 23:20:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds. Jul 25 23:20:27 ngelinux001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 25 23:20:27 ngelinux001 kernel: df D ffff9758aafbc100 0 147233 144770 0x00000004 Jul 25 23:20:27 ngelinux001 kernel: Call Trace: Jul 25 23:20:27 ngelinux001 kernel: [] ? __schedule+0x3ff/0x890 Jul 25 23:20:27 ngelinux001 kernel: [] schedule+0x29/0x70 Jul 25 23:20:27 ngelinux001 kernel: [] schedule_timeout+0x221/0x2d0 Jul 25 23:20:27 ngelinux001 kernel: [] ? finish_wait+0x56/0x70 Jul 25 23:20:27 ngelinux001 kernel: [] ? mutex_lock+0x12/0x2f Jul 25 23:20:27 ngelinux001 kernel: [] ? autofs4_wait+0x420/0x910 Jul 25 23:20:27 ngelinux001 kernel: [] wait_for_completion+0xfd/0x140 Jul 25 23:20:27 ngelinux001 kernel: [] ? wake_up_state+0x20/0x20 Jul 25 23:20:27 ngelinux001 kernel: [] autofs4_expire_wait+0xab/0x160 Jul 25 23:20:27 ngelinux001 kernel: [] do_expire_wait+0x1e0/0x210 Jul 25 23:20:27 ngelinux001 kernel: [] autofs4_d_manage+0x7e/0x1d0 Jul 25 23:20:27 ngelinux001 kernel: [] follow_managed+0xba/0x310 Jul 25 23:20:27 ngelinux001 kernel: [] lookup_fast+0x12d/0x230 Jul 25 23:20:27 ngelinux001 kernel: [] path_lookupat+0x16d/0x8b0 Jul 25 23:20:27 ngelinux001 kernel: [] ? __mem_cgroup_commit_charge+0xe2/0x2f0 Jul 25 23:20:27 ngelinux001 kernel: [] ? kmem_cache_alloc+0x35/0x1f0 Jul 25 23:20:27 ngelinux001 kernel: [] ? getname_flags+0x4f/0x1a0
III. Root Cause Analysis and Solution
a. Run “strace df -h” command
To identify d:which partition is hanged, run below comman
# strace df -h
statfs("/home/user1", {f_type="NFS_SUPER_MAGIC", f_bsize=65536, f_blocks=16777216, f_bfree=11979570, f_bavail=11979570, f_files=21251126, f_ffree=8615236, f_fsid={0, 0}, f_namelen=255, f_frsize=65536}) = 0
statfs("/home/oracledf",
b. Identify where the above command is hanged, i.e. the partition which is getting hanged.
In above output the culprit is /home/oracledf. Now we can check from where it is mounted and try to troubleshoot that.
The above error message means “df” is waiting for the hung task to complete, and regularly try to mount that partition on the server which in the end turned into kernel panic.