RHEL or Centos Linux: Kdump generated manually however not able to find it on reboot ?

By Saket Jain Published November 17, 2021 Linux/Unix

Today in this article we will see an interesting issue where you crash a server manually however the crash dump is not available on server post reboot.

In my scenario i have configured the kdump on a separate path “/var/crash”. However i am not able to find it even after crash.

Lets see the issue details and its solution.

I. Issue Details
Kdump configured and operational.
However when i crashed server, i can’t see dump files in /var/crash and /var became 100% full.

[root@ngelinux01 ~]# service kdump status
Kdump is operational

[root@ngelinux01 ~]# ls -l /var/crash/
[root@ngelinux01 ~]#

II. Investigation
Now we will investigate the issue how come /var became 100% with no data on /var/crash.

a. Check what is consuming space on /var.
[root@ngelinux01 ~]# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-varlv
                      7.9G  7.9G     0 100% /var
[root@ngelinux01 ~]#

### As a precaution, just checking if we have enough space in VG.
[root@ngelinux01 ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  appvg    1   4   0 wz--n- 410.10g 390.10g
  rootvg   1  13   0 wz--n- 136.19g  54.19g

[root@ngelinux01 ~]# cd /var
[root@ngelinux01 var]# du -sh *
4.0K    account
0       adm
1.3G    cache
12K     cpq
16K     crash
48K     db
12K     empty
4.0K    games
4.0K    gopher
276M    lib
4.0K    local
16K     lock
253M    log
89M     logII
20K     lost+found
0       mail
4.0K    nis
230M    nmon
162M    opt
4.0K    preserve
116K    run
161M    spool
101M    tmp
4.0K    yp

[root@ngelinux01 var]# du -sh /var
2.6G    /var

Interesting the total used space is around 2.6GB however no free space.
Interesting, lets see any deleted/open files using the space.

### First check any deleted files
[root@ngelinux01 var]# lsof | grep -i deleted
JOBMAN     2953    root   17r      REG             253,11 14249984       8484 /usr/tws/TWS/SymphOld1636379233 (deleted)
JOBMAN     6550    root   17r      REG             253,11 13806592       8485 /usr/tws/TWS/SymphOld1636292817 (deleted)
nge_gathe  6607    root    1u      REG              253,7        0         46 /tmp/sf01.ib4 (deleted)
nge_gathe  6975    root    1u      REG              253,7        0         40 /tmp/sf3a.958 (deleted)
check_clo  7455    root    0r      REG              253,0        1      83165 /usr/local/cloudamize/bin/system_key (deleted)
check_clo  7472    root    0r      REG              253,0        1      83167 /usr/local/cloudamize/bin/system_key (deleted)
check_clo  7473    root    0r      REG              253,0        1      83166 /usr/local/cloudamize/bin/system_key (deleted)
qualys-cl  7532    root    8u      REG              253,0    12288     155844 /usr/local/qualys/cloud-agent/patchmanagement/PatchStatus.db (deleted)
qualys-cl  7532    root    9u      REG              253,0    32768     155845 /usr/local/qualys/cloud-agent/patchmanagement/PatchManagement.db (deleted)
sleep     23588    root    0r      REG              253,0        1      83166 /usr/local/cloudamize/bin/system_key (deleted)
JOBMAN    30545    root   17r      REG             253,11 14249984       8484 /usr/tws/TWS/SymphOld1636379233 (deleted)
nge_gathe 30599    root    1u      REG              253,7        0         34 /tmp/sf25.cuq (deleted)

### Any open files using space
[root@ngelinux01 var]# lsof /var
COMMAND     PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
rpm         676    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm         676    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm         676    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm         676    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm         676    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm         676    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm        1141    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm        1141    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm        1141    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13662    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13662    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13662    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13662    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13662    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       13833    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       13833    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13833    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13833    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13833    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13833    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       13880    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       13880    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13880    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13880    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13880    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13880    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       13960    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       13960    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13960    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13960    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13960    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13960    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15040    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       15040    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       15040    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       15040    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       15040    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       15040    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15246    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       15246    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       15246    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       15246    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       15246    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       15246    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15666    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       15666    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       15666    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       15666    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       15666    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       15666    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15704    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       16887    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       16887    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       16887    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       16887    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       19623    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       19623    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       19623    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       19652    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       19652    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       19652    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       19652    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       19652    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       19652    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       19991    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       19991    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003rpm       22038    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22038    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22038    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22038    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22038    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22038    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22063    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22063    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22063    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22063    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22063    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22063    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22441    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22441    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22441    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22441    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22441    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22441    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22680    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22680    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22680    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22680    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22680    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22680    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22737    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22737    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22737    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22737    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22737    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22737    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22972    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22972    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22972    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22972    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22972    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22972    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       23761    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       23761    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       23761    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       23761    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       23761    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       23761    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       24215    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       24215    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       24215    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       24215    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       24215    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       24215    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       24322    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       24322    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       24322    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       24322    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       24322    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       24322    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       25122    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       27399    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       27399    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       27399    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       27399    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
bash      27840    root  cwd    DIR  253,6     4096     2 /var
rpm       28485    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       28485    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       28933    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       28933    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       28933    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       29026    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       29026    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       29026    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       29026    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       29026    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       29026    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       29035    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       29035    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       29035    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       29035    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       29035    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       29035    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
lsof      29954    root  cwd    DIR  253,6     4096     2 /var
lsof      29955    root  cwd    DIR  253,6     4096     2 /var
rpm       29960    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       29960    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       29960    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       29960    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       29960    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       29960    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
yum       29976    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
yum       29976    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
yum       29976    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
yum       29976    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
yum       29976    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
yum       29976    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       30208    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       30208    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       30208    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       30208    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       30208    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       30208    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       30648    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       30848    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       30848    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       30848    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       30848    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       30848    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       30848    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       31019    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       31019    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       31019    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       31019    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       31019    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       31019    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       31036    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       31036    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       32675    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       32675    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       32675    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       32675    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name

[root@ngelinux01 var]# du -sh /var/lib/rpm/Packages
57M     /var/lib/rpm/Packages

[root@ngelinux01 var]# ps -ef | grep -i var
root      6619     1  0 Nov05 ?        00:00:13 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
ntp       7156     1  0 Nov05 ?        00:00:02 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root     31389 27840  0 11:34 pts/2    00:00:00 grep -i var

b. No files found using space, try to restart Kdump service.

[root@ngelinux01 var]# service kdump restart
Stopping kdump:                                            [  OK  ]
Starting kdump:                                            [  OK  ]
[root@ngelinux01 var]#


### No improvement after service restart.
[root@ngelinux01 var]# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-varlv
                      7.9G  7.9G     0 100% /var
[root@ngelinux01 var]#

### /var is 100% used, and /var/crash is empty.
[root@ngelinux01 var]# df -h /var/crash
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-crashlv
                       35G  177M   33G   1% /var/crash
[root@ngelinux01 var]#

III. Solution

If you have observed, /var became full after kdump collection and /var/crash is empty as it is a separate partition.
So it seems interesting, when we crashed the system using echo c > /proc/sysrq-trigger, it just umounted other partitions on system except the main hierarchy and hence kdump was collected, however not visible.

Lets see the core files.

### First umount /var/crash
[root@ngelinux01 var]# umount /var/crash

### Now list the files
[root@ngelinux01 var]# ls -ltr /var/crash
total 12
drwsrwsrwt. 2 root root 4096 May 24 20:21 127.0.0.1-2021-05-24-20:14:31
drwsrwsrwt. 2 root root 4096 Aug 30 14:10 127.0.0.1-2021-08-30-14:10:01
drwsrwsrwt. 2 root root 4096 Nov 5 15:37 127.0.0.1-2021-11-05-15:37:46
[root@ngelinux01 var]#

So its interesting that we can see the files now.

Hence we came to conclusion that system umounts all non-dependent filesystems when we crash the system and collects the dump on main partition path.

0 0 votes

Article Rating

Tags: /var /var/crash centos collect collection core crash dump files kdump linux mount not partition redhat rhel umount unix visible

Vishu on How to create full size one partition using parted command in Linux ?: “Thanks a lot. This was exactly what I was looking for. Other blogs are very confusing but this worked for…” Jul 30, 23:26
cccc on Print only usernames from /etc/passwd file using grep, awk or cut commands.: “love it” Oct 18, 16:13
Saket Jain on How to configure and install Nagios Server on Linux ?: “Please check your system resolv.conf/DNS settings, it looks its not able to resolve the hostname. The URL is correct.” Jul 18, 13:37
deepanshu on How to configure and install Nagios Server on Linux ?: “[root@localhost nagios]# wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.5.tar.gz –2023-07-02 19:15:08– https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.5.tar.gz Resolving assets.nagios.com (assets.nagios.com)… failed: Name or service not known. wget: unable to resolve host…” Jul 3, 08:13
aasdasdKEKEK on Solved: subscription-manager – Not supported by a valid subscription.: “You Genius. How do we “verify if we have enough subscription available on redhat support to add this new server.”” May 27, 18:26

RHEL or Centos Linux: Kdump generated manually however not able to find it on reboot ?

Like this:

Related

TAGS

Share this NG Linux post:

Like this:

Related