Deleted logs but df still shows 100%—if the process isn't restarted, will the space never come back?

Bottom line first: in most cases, the space hasn't come back because a process still holds the file

After a disk alert, if you delete logs and df does not show a drop in used space, the common cause is not deleting the wrong directory. It is that a process still holds a handle to that file, and lsof will show it as deleted. The directory entry is gone, but the inode and data blocks remain. They are released only after the process holding it closes the file or restarts. Based on common delivery practice in 2026, first use lsof +L1 or the fd directory of the corresponding process under /proc to find the holder, then use the difference between df and du to decide whether it is 'deleted but not freed' or whether the usage is not in this directory at all. df counts blocks allocated by the filesystem; du counts files whose names are still visible. A mismatch does not necessarily mean the system is broken.

• Deleted but not freed: a process is still writing to this log, and the log framework has not reopened the file.
• Freed immediately after deletion: no process holds it; the usage is elsewhere, so keep looking.
• df -h shows free space but writing a file reports No space left on device: check df -i for inodes first; this is unrelated to block space.

Deleted but still held: commonly three types of references remain open

In Linux and most Unix-like systems, deleting only removes the file name from the directory. As long as a process has the file open, the kernel keeps the inode and data blocks until the last handle closes. This is especially common with logs: the application opens app.log at startup and keeps appending, and rm only removes its name.

• Process handles: the application itself, a log collection agent, or a long-running tail -f terminal may hold it; restarting the collector can also release it.
• Mount overlay: after data is written into a mount point, the original directory files are covered; du cannot see them, but the space is indeed occupied; umount first and then verify.
• Container writable layer: logs written into the container writable layer or overlay upper layer; the host sees the image-layer perspective, and deleting files inside the container may not free host space.
• Other large consumers: database binlog, backup packages, core dump, package manager cache; typical range from hundreds of MB to several GB.

Troubleshooting order: measure blocks first, then check inodes, then find handles

This order is used because the information cost increases: df returns a result in a second, lsof requires permissions and is slow on large machines, and restarting a process has business cost. Do the low-cost checks first, and you can avoid one midnight restart.

1. Measure blocks: use df -h to see the usage of each mount point and identify which partition is full.
2. Check inodes: use df -i to see inode usage; when there are many small files, session files, thumbnails and mail queues often fill up inodes first.
3. Find handles: use lsof +L1, or look for items marked deleted in each process's fd directory under /proc, and note the process name and file size.
4. Confirm release: for applications that support a log reopen signal, send the signal first (check the application's official documentation for specifics); for those that do not, schedule a restart at low peak, then recheck with df after reopening.

The third step is often skipped. On delivery sites, we have seen ops delete files, see no drop in space, and then also delete an archive directory. The real space was an 8GB handle still open. That was a wasted deletion round and lost historical logs. Running lsof before deletion and rechecking df once after deletion is sufficient; there is no need to script every time.

A delivery-site experience: a backend project wrote logs directly to a local directory, and a collection agent held handles open for a long time. After the disk reached 90% and alarmed, ops first rm'd the day's logs; df did not drop, so they deleted an archive directory too. The constraint was that the application could not be restarted directly because scheduled tasks were running. The approach was to use lsof +L1 to identify the process holding the handle, send a reopen signal to the application, restart the collection agent through the platform, and then use truncate to zero out the file still being written. Result: used space fell within minutes, at the cost of missing a segment of that day's logs and accidentally deleting archives. Afterwards, the acceptance checklist added the log directory, retention days (typical range 7–30 days), and a 70%–80% warning threshold.

Delete, truncate, or rotation: which one fits?

All three make logs smaller, but they differ in release timing, scope of log loss, and operation cost. The basis for choosing is not which is more advanced, but whether you can accept losing the current logs, whether you can restart or send a signal, and whether someone maintains the rotation configuration long term.

• rm directly and have the process reopen the log: release depends on whether the reopen succeeds; the operation window is typically a few minutes to half an hour (including verification); the cost is that content written before the reopen is lost, and if you forget to reopen, the space never comes back.
• Truncate the file (: > app.log or truncate -s 0): if the handle is unchanged, space is released immediately and the process does not need to restart; the cost is that historical content is cleared, and if the current write offset is handled improperly, null characters may appear; suitable for emergency space recovery.
• logrotate daily rotation + retention count + compression: a long-term solution, retaining 7–30 days is a typical range; the cost is that copytruncate or a postrotate reopen signal must be configured correctly; misconfiguration may rotate without effect or still point to the old handle after rotation.

Experience-based practice is to use truncation for emergency stopgap, add rotation the same day, and use direct rm only when the process is going to restart anyway. Put the log directory, retention days and alert threshold in the deployment acceptance checklist; that is easier than firefighting months after launch.

Applicability boundaries: when not to rely on deleting logs

This troubleshooting fits single-machine or small-instance scenarios where disk usage is mainly text logs and some historical logs can be discarded, especially projects that have just launched and whose log strategy is not yet finalized.

Conversely, if the bulk of disk usage is database data files, object storage cache or business backups, deleting logs only relieves the problem for a day or two. If industry compliance has hard requirements for log retention, first check retention days against industry and platform norms before deciding whether logs can be deleted. For containerized setups where logs are written directly to the writable layer, the correct approach is to change the collection method, not to periodically enter the container and delete files.

• Suitable: logs dominate usage, brief downtime is acceptable or a log reopen signal is supported, and basic monitoring exists.
• Not suitable: data files fill the disk, compliance retention requirements are explicit, or the critical path can neither be restarted nor sent a signal.

Common questions

If I rm logs and don't restart the process, will space come back by itself?

In most cases no. Space is released only after the process holding the file closes the handle; if the application supports a log reopen signal, send the signal first and recheck with df.

df shows full but du totals differ a lot; does that mean the system was compromised?

Don't jump to conclusions. The common causes are deleted-but-not-freed files or directories covered by mounts; cross-check with lsof +L1 and fd under /proc, and most possibilities are ruled out.

Is emptying a log and deleting it the same for disk?

No. Emptying releases space immediately and the original handle remains valid; after rm, the handle points to a deleted inode, and space is released only after the process closes or restarts.

Logs in a container keep growing; I delete them inside the container but the host is still full—why?

Logs may be written to the container writable layer or a host mount directory; the two views differ. Per platform norms, changing to stdout collection or mounting a persistent volume is more stable.

Next step: on the alarmed machine, first run df -h, df -i and lsof +L1 and keep the results before deciding whether to truncate for stopgap or switch to logrotate. If disk usage mainly comes from data files, or compliance retention requirements are explicit, this approach does not apply; move to capacity planning and archiving strategy.