In the daily operation and maintenance process, we often need to deal with disk space issues, when we receive alarms, the first time we will go to find those large files, generally such as centos, the large file may be /var/log/messages.

But sometimes, there is a situation where you can’t find the big files, and when you look for them by du, the size of the statistics doesn’t correspond to the space occupied by df.

If the inode is not full by df -i, then most likely, there is a large file is directly rm, but there are still processes open the file.

In this case, because the process did not exit, so the space occupied by the file will not be released; until the process exits, the disk space will not really be released.

Question 1: How do I find out which process opened the file?

On linux, since the process is still alive, it can be viewed by looking at all the fd’s opened by the process and if the file has been deleted, when viewed, it will show (deleted).

The example is as follows.

1
2
3
4
5
6

$ sudo find /proc/*/fd -ls | grep  '(deleted)'
   388609      0 lrwx------   1 zerotier-one zerotier-one       64 Aug 21 00:19 /proc/29400/fd/4 -> /tmp/ibpX85Vd\ (deleted)
   388610      0 lrwx------   1 zerotier-one zerotier-one       64 Aug 21 00:19 /proc/29400/fd/5 -> /tmp/ibCwAgAj\ (deleted)
   388611      0 lrwx------   1 zerotier-one zerotier-one       64 Aug 21 00:19 /proc/29400/fd/6 -> /tmp/ibRZ5rep\ (deleted)
   388612      0 lrwx------   1 zerotier-one zerotier-one       64 Aug 21 00:19 /proc/29400/fd/7 -> /tmp/ibBuNEzA\ (deleted)
   388616      0 lrwx------   1 zerotier-one zerotier-one       64 Aug 21 00:19 /proc/29400/fd/11 -> /tmp/ibG68kpG\ (deleted)

Question 2: How can this be avoided?

Do not delete the file directly, but free up disk space by truncating the file.

One way to do this is to

1

cat /dev/null > ${filename}

or

1

: > ${filename}

In this way, the space can be released quickly.