0

I have a strange discrepancy. With "df" I see that my root disk is using almost 229G.

# df -h Filesystem Size Used Avail Use% Mounted on /dev/dm-2 241G 229G 13G 96% /

but "du" tells me that I have only about 12GB (rounded up) in use.

# du -sh /* | grep G 3.2G /run 2.1G /usr 6.0G /var

Both commands were run as root. What I could imagine is that there is "hidden" data in a directory where now an NFS share is mounted to. If this is the case. How could I find out what files are there "hidden". Or do you have other ideas why those numbers are so totally different?

Improve this question
1

2 Answers 2

Reset to default
2

You probably have an app/process that has that file open. When you delete a file that is held open by an app the OS still sees the file size in df as the file is still open in memory.

Here is some documentation I wrote for a co worker that should get you what you need.

Truncate large open files

You have deleted files to free space but space not free afterward. Now df -lah and du -lah show different sizes

Use LSOF to get deleted but held files

lsof |grep deleted

This will show all files deleted but held open by an app.

java 2943 gateway 410w REG 253,3 50482102 139274 /opt/span/app/node/default/var/attachments/att180368_0.part (deleted) java 2943 gateway 411w REG 253,3 46217973 139284 /opt/span/app/node/default/var/attachments/att182230_0.part (deleted) java 2943 gateway 412w REG 253,3 50483894 139280 /opt/span/app/node/default/var/attachments/att181920_0.part (deleted)

You can either restart the app to unlock files or truncate the files.

To truncate the files you will have to look at the output above to get the PID and fd (file descriptor number).

truncate the file with 

echo > /proc/PID/fd/fd_number

Example: to zero the file size of three listed above you would issue the following

echo > /proc/2943/fd/410 echo > /proc/2943/fd/411 echo > /proc/2943/fd/412

if you have many to truncate bash to the rescue. 

for n in {410..412}; do 'echo > /proc/2943/fd/$n'; done;

df -lah should show free space now BUT files will show under lsof |grep deleted but will be 1 in size 

java 2943 gateway 410w REG 253,3 1 139274 /opt/span/app/node/default/var/attachments/att180368_0.part (deleted) java 2943 gateway 411w REG 253,3 1 139284 /opt/span/app/node/default/var/attachments/att182230_0.part (deleted) java 2943 gateway 412w REG 253,3 1 139280 /opt/span/app/node/default/var/attachments/att181920_0.part (deleted)

The files descriptors will be released on next reboot or restarting/reloading of the app that opened the files.

Of course you will have to adjust your commands to match the output of your locked files.

Improve this answer
2
  • Good idea. I didn't think of it. But sadly not the correct answer. I did find several files like this. But all of them belonged to elasticsearch. And after restarting elasticsearch the files didn't show up when doing a "lsof" but the size of "df" didn't change much. Commented Mar 24, 2020 at 12:15
  • I have seen poorly built log rotate setups do this. They needed to truncate the logs but just rotated them. The result was the logs rotated but the old file was never flushed. I don't think it tried to delete them so nothing shows up as deleted in lsof. Logrotate just renamed them and tried logging to new filename. The rename never worked and did leave ghost files that misreported in df. Didn't find it was happening until the volume alert went off as the vol was filling up from not properly rotating. It has been a long time so I don't remember the details. Commented Mar 24, 2020 at 20:44
1

In my case (and you helped me in a PM by telling me the same) it was a over-mount problem. I had some data in a directory on the partition, which du couldn't see, because a mount has hidden them.

So try to umount all mounts and look for data in the remaining directories.

Have a good time and thank you!

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.