OS is Debian. I have a few systems that seem to not be freeing disk space correctly.
On a normal system:
du --max-depth 1 / --exclude=/proc -h -x Seems to almost exactly agree with:
df -h / ie, ~800 gigs are reported used in both cases for my normal system test case.
On an affected system, df shows 95 gigs used, but du only shows ~55 gigs used!
This is a huge discrepancy, especially compounded by the fact that the affected system also reports 0 bytes available for /.
lsof | grep deleted shows nothing. Filesystem is ext4.
What is going on here?
Edit: The issue ended up being a problem with lsof. To briefly summarize: running lsof without any parameters ended up taking so long (hours) that it appeared that nothing was getting returned, but I think lsof was just getting killed. Running lsof like this: lsof / | grep deleted was much, much faster, would exit correctly, and when doing so would show one very large deleted file. Running lsof / also excludes any other mount points, even if they are "inside" of /.