ENOSPC: Fix No Space Left on Device When df Shows Free

df says you've got gigabytes free but writes still fail with ENOSPC. The culprit is almost always deleted-but-open files or inode exhaustion. Here's how to find and kill it.

You deleted the log, df -h says 40 gigs free, and something still can't write a byte. Annoying, I know. Let's fix it.

The Fix

Run this first. It's the answer about 80% of the time:

lsof +L1 | head -50

The +L1 flag lists files with a link count under 1 — meaning the directory entry is gone but a process still holds the file descriptor open. The kernel can't reclaim those blocks until the last process closes the file.

You'll see something like:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
nginx   18422 root    4w   REG  253,0  18.4G    0  1234567 /var/log/nginx/access.log (deleted)

That's an 18-gig log file that rm unlinked but nginx is still writing to. Space won't come back until nginx lets go.

Three ways to reclaim it, pick based on your situation:

Restart or reload the process. Cleanest option. systemctl restart nginx releases the fd and the kernel frees the blocks.
Truncate via the fd. If you can't restart, and you know the PID and fd number, do : > /proc/18422/fd/4. That zeroes the file in place without killing the daemon.
Leave it alone. If it's a short-lived process and the file's already deleted, waiting works. Don't cargo-cult this one though — big logs sit there for hours.

If lsof +L1 comes back clean, check inodes next:

df -i

A filesystem can have plenty of blocks but zero inodes. Look at the IUse% column. If it's at 100%, you're out of inode slots, not disk space. Find what's hoarding them:

find /var -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head -20

Nine times out of ten it's a mail spool, a session directory, or a Docker overlay. Millions of 4 KB files eat inodes fast.

Why This Happens

Unix filesystems separate three things: the directory entry (the name you see), the inode (metadata + block pointers), and the data blocks. When you rm a file, you remove the directory entry. The blocks only free when the inode's reference count hits zero.

An open file descriptor counts as a reference. So a process holding the fd keeps the inode alive, and the blocks stay allocated. df reads the filesystem's free-block counter, which is accurate — the space really is in use. It's just in use by a file with no name.

The classic trigger: logrotate runs mv access.log access.log.1 instead of copytruncate, then something rms the rotated file while nginx still has the original fd open. Or someone did rm /var/log/whatever.log to "clear space" without restarting the writer.

Inode exhaustion is a different beast. ext4 allocates inodes at mkfs time. You can't add more without remaking the filesystem (or using tune2fs on ext4 with -I in rare cases, but honestly, rebuild). This is why 20 million tiny files on a 100 GB volume will fill up with 80 GB still free.

Less Common Variations

ext4 reserved blocks

ext4 reserves 5% of the volume for root by default. On a 1 TB disk that's 50 GB that df shows as used even though your app can't touch it. If a non-root process is failing with ENOSPC and df shows 4–5% free, this is why:

tune2fs -m 1 /dev/sda1

Drops the reserve to 1%. Don't go to 0 on the root volume — you want that buffer for fsck and emergency writes.

Journal space on XFS

XFS reserves space for its log. df doesn't always count this cleanly. xfs_info /mountpoint shows the real numbers. If statfs free blocks are near zero but df looks fine, that's the culprit.

Docker/containerd overlay2

Containers write to /var/lib/docker/overlay2. Deleted containers sometimes leave orphaned layers. Check with docker system df -v and prune carefully. Also, a running container can hold deleted files open just like any other process — lsof +L1 shows them under the host PID namespace.

NFS and stale handles

On NFS, an unlinked-but-open file counts against the server's space, not the client's view. If df on the client shows free space but writes fail, check the server. lsof +L1 on the NFS server, not the client.

Btrfs and snapshots

Btrfs snapshots pin old blocks. df reports unallocated space, but a snapshot holding old data blocks prevents reuse. btrfs filesystem usage /mnt shows the truth. Delete old snapshots with btrfs subvolume delete.

Prevention

A few things that stop this from coming back:

Fix your logrotate config. For daemons that don't reopen logs on HUP, use copytruncate. For nginx, use postrotate systemctl reload nginx. Test with logrotate -d /etc/logrotate.conf before it bites you at 3 AM.
Alert on inode usage, not just blocks. Nagios, Zabbix, Prometheus — whichever you use, add df -i to the checks. Nobody watches inodes until it's too late.
Never rm a log you're not sure about. Use truncate -s 0 /var/log/foo.log instead. Truncating keeps the inode and frees the blocks immediately, even if a process has it open.
Cap your container logs. Add "log-opts": {"max-size": "50m", "max-file": "5"} to /etc/docker/daemon.json. Default is unlimited, and I've watched a single noisy container eat 400 GB on prod.
Install a simple watcher. apt install ncdu or just a cron job that emails when df -h or df -i crosses 85%. Cheap insurance.

Nine out of ten ENOSPC-with-free-space tickets end with lsof +L1 and a restart. The other one is inodes. Everything else on this page is the long tail.