Linux: my disk is full and the server stopped working

Question

My app crashed with "No space left on device". `df -h` shows `/` at 100%. How do I find what's eating the space and fix it quickly?

Accepted Answer

**1. Find the culprit:**
```bash
sudo du -xh / --max-depth=1 2>/dev/null | sort -h | tail
# then drill down, e.g.
sudo du -xh /var --max-depth=1 | sort -h | tail
```
Or use `ncdu /` for an interactive view.

**2. The usual suspects:**
- **Logs**: `/var/log`, especially app logs without rotation. Clear systemd logs with `sudo journalctl --vacuum-size=200M`.
- **Docker**: images, stopped containers, build cache and container logs:
```bash
docker system df
docker system prune -a     # removes unused images/containers (careful)
docker builder prune
```
- **Old backups or dumps** left on the same disk.
- **Deleted files still held open**: `sudo lsof +L1`. Restart the process holding them to free the space.

**3. Prevent it:** configure `logrotate`, Docker's log limits (`"log-opts": {"max-size": "10m", "max-file": "3"}` in `daemon.json`), and an alert at 80% disk usage.

1 Answer