Why is the disk on the server overflowing and how to find the "hidden" files?
Greetings, friends!
Sudden disk space depletion on a Linux server is one of the most common causes of emergency failures. This issue can stop databases, break web server operations, and freeze system processes.
Administrators often encounter a paradoxical situation: standard utilities report 100% disk usage, yet browsing folders visually reveals no large files. In this guide, we will break down the systemic causes of disk space leaks and learn how to find "hidden" data. By following this guide, you will be able to free up disk space and avoid potential service outages.
Key Takeaways: Core Conclusions
Deleted Open Files: If you delete an active log file that a process (like Nginx or MySQL) is writing to, the disk space will not be freed until the process is restarted or terminated.
Exhausted Inodes: A disk can lock up even with gigabytes of free space left if the system runs out of inode limits (e.g., due to millions of small PHP sessions or cache files).
Hidden Mount Points: If you mount a new disk or partition over an existing directory containing files, the old data becomes "hidden" beneath the new mount point and continues to consume space.
5 Main Causes of "Invisible" Disk Space Leaks
1. Unlinked Open Files
When you delete a file using the rm command, only its entry in the file system is removed. If a process (e.g., systemd-journald, nginx, or mysqld) keeps this file open, the operating system will not release the disk blocks.
How to find and fix: Locate "hung" files using the lsof utility:
sudo lsof +L1
Or with an alternative command:
sudo lsof | grep deleted
The output will display the process PID and the size of the held file. To release space without restarting the entire server, simply restart the specific service:
sudo systemctl restart <service_name>
2. Inode Exhaustion
A Linux file system stores metadata for each file in a structure called an Inode. The total number of inodes is fixed when formatting the disk. If millions of small files are generated (caches, session files, tiny logs), inodes run out, causing system write errors despite available gigabytes.
How to find and fix: Check inode status using:
df -i
If IUse% reaches 100%, locate the directory containing the highest number of files:
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -nr | head -n 10
Clean up the identified folders containing small cache or temporary files.
3. Full Journald and System Service Logs
Log rotation (logrotate) may fail, or the standard systemd journal might fill gigabytes of disk space with verbose debug logs.
How to find and fix: Check the disk volume consumed by systemd logs:
journalctl --disk-usage
Safely reduce log size, leaving only the last 500 MB, for example:
sudo journalctl --vacuum-size=500M
4. Forgotten Docker Containers, Images, and Volumes
The Docker environment actively clutters disks behind the scenes. Unused images (dangling images), stopped containers, and anonymous volumes can accumulate dozens of gigabytes.
How to find and fix: Assess Docker disk usage:
docker system df
Perform a safe, thorough cleanup of unused Docker resources:
docker system prune -a --volumes
5. Files "Hidden" Beneath a Mount Point
If 50 GB of files were written to /mnt/data and then a separate new disk was mounted onto that same /mnt/data folder, the original 50 GB becomes invisible to standard tools while continuing to occupy space on the root / partition.
How to find: Check the size of the original partition by temporarily binding the root file system to an alternative mount point:
sudo mount --bind / /mnt/root_check
sudo du -sh /mnt/root_check/mnt/data
sudo umount /mnt/root_check
Checklist: Quick Search for Large Files on a Server
If your server ran out of space right now, execute these steps to isolate the issue:
Check overall partition and inode usage:
Bashdf -h df -iFind the TOP 10 largest files on the server:
Bashsudo find / -xdev -type f -size +100M -exec ls -lh {} \; | awk '{ print $5, $9 }' | sort -n -r | head -n 10Use the interactive
ncduscanner: Install the lightweightncduutility for visual folder analysis:Bashsudo apt install ncdu -y # Ubuntu/Debian sudo dnf install ncdu -y # AlmaLinux/Rocky sudo ncdu -x /(The
-xflag instructs the utility to stay within the same file system and ignore mounted network drives).
FAQ: Frequently Asked Questions
Why do
duanddfshow different amounts of used space? Thedf(Disk Free) command queries the kernel and accounts for all locked blocks (including deleted files held by active processes). Thedu(Disk Usage) command recursively scans the directory tree and counts only physically existing files. The difference between their readings represents the volume of "deleted but open" files.Can I clear log files using the
rmcommand? It is not recommended to delete active.logfiles viarm, as the service will continue writing to the deleted file descriptor. Instead, truncate the file contents without deleting the file itself:Bashsudo truncate -s 0 /var/log/nginx/access.logWhat should I do if disk space fills up rapidly within hours? Set up process monitoring using
fatraceor the built-inauditdutility to track which specific process is generating an abnormal volume of disk writes (Disk I/O) in real-time.
Conclusion
Disk space exhaustion is always a consequence of insufficient control over automated processes: log rotation, temporary file cleanup, and database growth. Configuring logrotate properly, performing periodic Docker cleanup (prune), and quickly identifying open descriptors will prevent 99% of emergency server halts.
However, for critical services and growing projects, having a resilient disk subsystem and the ability to scale storage on the fly without downtime is vital.
If your project requires a flexible infrastructure with guaranteed resources, check out NVMe VPS options from MivoCloud. We offer fast disk space expansion in just a few clicks, pure KVM virtualization, and enterprise-grade ultra-fast NVMe storage built to handle heavy I/O loads.
Article Author: Anatolie Cohaniuc