df shows free space but still get 'no space left on device'

Quick answer

Run df -i to check inodes. If that shows 100%, delete small files. If inodes are fine, run lsof | grep '(deleted)' — those deleted files still use space. Kill or restart the process holding them.

Why this happens

I've seen this at least 20 times. You check df -h and there's 20GB free. But touch test.txt gives "No space left on device". The two main reasons: inode exhaustion or file handles to deleted files. Inodes are metadata entries for files. If you have millions of tiny files (cached images, temp files, Docker containers), you can run out of inodes even with disk space free. The other case: a process opens a file, you delete it, but the process keeps the handle. That file's space stays allocated until the process closes or dies. Both are dead simple to check.

Fix steps

1. Check inodes first — Run df -i. Look at the IUse% column. If it's 100% (or close), you've got inode exhaustion. Find the directory with the most files: for i in /var /tmp /home /opt; do echo \"$i: $(find $i -xdev -type f | wc -l)\"; done. Usually it's /var/spool (mail or cron) or /tmp (session files). Delete old files. For mail spool, use find /var/spool/mail -type f -mtime +30 -delete.
2. If inodes are fine, check deleted files — Run sudo lsof | grep '(deleted)'. You'll see processes holding file handles to deleted files. Look at the SIZE column. If you see something like "259584 bytes" it's not huge. But if you see "1073741824" (1GB) or more, that process is leaking space. Note the PID and file path.
3. Fix deleted files — You have two options: kill the process (kill -9 ) or restart the service (e.g., systemctl restart nginx). Killing a critical process might crash the app, so try restarting first. For a quick check: ls -la /proc//fd/ — you'll see symlinks to the deleted files with no path. Truncate them with : > /proc//fd/ — but careful, only if you know what you're doing.
4. Verify space is freed — Run df -h and df -i again. If the error persists, check for hidden mounts (like Docker overlay filesystems). Run findmnt — sometimes a mount point hides the real filesystem.

Alternative fixes if main steps fail

• Docker cleanup — If you use Docker, old containers and images eat inodes. Run docker system prune -a — but back up anything important first.
• Log rotation — Check /var/log. Some apps (looking at you, Apache and syslog) generate millions of small logs. Use logrotate with copytruncate to avoid the deleted file problem.
• Run du on root — du -sh --exclude=/proc --exclude=/sys /* 2>/dev/null — find big directories. If you see a huge /var, dig deeper: du -sh /var/*.

Prevention tip

Set up monitoring on inode usage. Most monitoring tools (Nagios, Zabbix, Prometheus) have a check for df -i. Alert at 80% inode usage. Also, set user limits on file counts in /etc/security/limits.conf — user soft nproc 4096, user hard nproc 8192. That stops a runaway process from filling your inodes overnight. And for deleted files, restart long-running processes (like web servers) during maintenance windows. They accumulate file handles over time.

Real-world triggers

I've seen this three times this month alone: a PHP session handler that created files in /tmp without cleanup (inode exhaustion), a buggy NFS mount that left stale file handles, and a logrotate config that wasn't reloading rsyslog (deleted files). Check those first.