Solved: kernel: INFO: task df:147233 blocked for more than 120 seconds.

Today in this post we will see how to solve the error message “df: task blocked for more than — seconds”.

Lets view the error message and how to solve this.

I. Error Message

 kernel: INFO: task df:147233 blocked for more than 120 seconds.

 

II. Analysis: Where to check the error messages.
Check out the respective messages file.

[root@ngelinux001 log]# cat messages-20220727 | grep -i "blocked for more than" | more
Jul 25 23:20:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:22:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:24:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:26:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:28:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:30:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:32:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:34:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:36:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:38:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
[root@ngelinux001 log]#


In some cases, like my case, there could be kernel panic due to “df” waiting to mount the partition i.e. 
failing to mount the hard NFS mount on server.

Jul 25 23:20:27 ngelinux001 kernel: INFO: task df:147233 blocked for more than 120 seconds.
Jul 25 23:20:27 ngelinux001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 25 23:20:27 ngelinux001 kernel: df              D ffff9758aafbc100     0 147233 144770 0x00000004
Jul 25 23:20:27 ngelinux001 kernel: Call Trace:
Jul 25 23:20:27 ngelinux001 kernel: [] ? __schedule+0x3ff/0x890
Jul 25 23:20:27 ngelinux001 kernel: [] schedule+0x29/0x70
Jul 25 23:20:27 ngelinux001 kernel: [] schedule_timeout+0x221/0x2d0
Jul 25 23:20:27 ngelinux001 kernel: [] ? finish_wait+0x56/0x70
Jul 25 23:20:27 ngelinux001 kernel: [] ? mutex_lock+0x12/0x2f
Jul 25 23:20:27 ngelinux001 kernel: [] ? autofs4_wait+0x420/0x910
Jul 25 23:20:27 ngelinux001 kernel: [] wait_for_completion+0xfd/0x140
Jul 25 23:20:27 ngelinux001 kernel: [] ? wake_up_state+0x20/0x20
Jul 25 23:20:27 ngelinux001 kernel: [] autofs4_expire_wait+0xab/0x160
Jul 25 23:20:27 ngelinux001 kernel: [] do_expire_wait+0x1e0/0x210
Jul 25 23:20:27 ngelinux001 kernel: [] autofs4_d_manage+0x7e/0x1d0
Jul 25 23:20:27 ngelinux001 kernel: [] follow_managed+0xba/0x310
Jul 25 23:20:27 ngelinux001 kernel: [] lookup_fast+0x12d/0x230
Jul 25 23:20:27 ngelinux001 kernel: [] path_lookupat+0x16d/0x8b0
Jul 25 23:20:27 ngelinux001 kernel: [] ? __mem_cgroup_commit_charge+0xe2/0x2f0
Jul 25 23:20:27 ngelinux001 kernel: [] ? kmem_cache_alloc+0x35/0x1f0
Jul 25 23:20:27 ngelinux001 kernel: [] ? getname_flags+0x4f/0x1a0

 

III. Root Cause Analysis and Solution

a. Run “strace df -h” command
To identify d:which partition is hanged, run below comman

# strace df -h
statfs("/home/user1", {f_type="NFS_SUPER_MAGIC", f_bsize=65536, f_blocks=16777216, f_bfree=11979570, f_bavail=11979570, f_files=21251126, f_ffree=8615236, f_fsid={0, 0}, f_namelen=255, f_frsize=65536}) = 0
statfs("/home/oracledf",

 

b. Identify where the above command is hanged, i.e. the partition which is getting hanged.

In above output the culprit is /home/oracledf. 
Now we can check from where it is mounted and try to troubleshoot that. 

 
 

The above error message means “df” is waiting for the hung task to complete, and regularly try to mount that partition on the server which in the end turned into kernel panic.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments