Container service restarts at night with only "Killed" in the log: does raising the memory limit fix it?

When a service running in a container restarts by itself in the middle of the night and the log shows only "Killed," in most cases the container memory limit was exceeded and the kernel OOM killer terminated it—not an exception thrown by the code. But don't rush to raise the memory limit: based on 2026 project delivery experience, the share of cases that are truly overwhelmed by business growth is low; more often, four numbers are out of alignment—the container limit, the application heap limit, runtime extra overhead, and system overhead. Verify these four layers first, then decide whether to tune parameters or hunt for a leak; the rework cost differs greatly.

Being killed vs. crashing with an error: the scenes look different

When the application itself runs out of memory, the runtime usually leaves evidence first: stacks, out-of-memory exceptions, heap dump files, and even an error log entry. When the kernel OOM killer picks off a container, the kernel sends SIGKILL directly; the process has no chance to run any cleanup logic, so the scene is often clean—logs stop on a few normal requests, the process disappears, and the orchestrator then restarts it according to the restart policy.

This is also why troubleshooting easily goes off course: someone who only reads application logs concludes that "nothing reported an error," while the evidence is actually at the container and kernel layers. The recommended order is from the outside in: first confirm the container termination reason, then look at the process memory composition, and only then go back into business code to find growth points.

• Symptom: Restart times cluster together, and the memory curve jumps sharply before each restart—this looks more like a memory problem than a random crash.
• Evidence: A heap dump file usually means the in-app heap was exhausted; only a termination record and no dump at all usually points to the container layer.
• Timing: Restarts during off-peak hours mean it is not a concurrency issue but resident memory itself growing.

Four-layer memory check: first see whether the numbers add up, then whether there is a leak

Container memory is a total pool defined by cgroups; the application can use only part of it, with several layers of overhead in between that are not written in business code. The reason for splitting it into four layers is that when each layer has a problem, the fix is entirely different: the first two layers are configuration issues that take effect after one change; the last two are either capacity issues or true leaks, where changes cost far more.

1. Container or instance memory limit: Determined by deployment configuration; this is the total pool. When checking, note that sidecar containers in the same Pod and log collection processes also eat from this pool.
2. Application heap or main memory limit: For example, JVM parameters such as -Xmx. The experience range is to set it to 50%–75% of the container limit, leaving headroom for off-heap memory.
3. Runtime extra overhead: Metaspace, thread stacks, direct memory, GC auxiliary structures, native library caches. This does not occupy the heap but does occupy container memory.
4. System and adjacent process overhead: Page cache, temporary files, other processes on the same machine. Scenarios such as batch exports and large file decompression can easily push this layer up.

Each layer has different points to watch: Layer 2 is not safer the smaller it is—squeezing the heap too small causes frequent collection and longer API response times; Layer 3 is the easiest to overlook, especially for services using direct memory or with many threads; Layer 4 must be controlled through implementation choices, such as not reading the entire result into memory at once in an export API. What counts as passing: the four layers' totals add up to the container limit, rather than assuming everything is fine just because "the heap limit is smaller than the container limit."

How to confirm from on-site evidence that it was exceeding memory, not another restart cause

There are at least four categories of service restart causes: terminated for exceeding memory, recreated after consecutive health check failures, evicted due to insufficient node resources, and node failure itself. Their fixes are completely different, so the first step is not optimization but assembling the full chain of "who was killed, at what time, and by whom."

• Container termination reason: The container layer records the previous termination reason and exit code; the exit code for SIGKILL termination is commonly 137, and memory overruns usually come with a corresponding reason tag.
• Kernel logs: When a memory overrun occurs, the kernel side leaves records showing which process was picked and why it was terminated.
• Eviction markers: If eviction happened because node resources were tight, it shows up in events and is not the same as a simple memory overrun.
• Health checks: Restarts caused by consecutive probe failures often show probe timeouts or unreachable ports in logs, while the memory curve stays flat instead.

A passing standard is: being able to point to the termination time and cause category, not merely seeing that "it restarted." If none of these are available, first fill in monitoring and event collection; otherwise every later optimization is guesswork. When evidence is needed, check item by item against official documentation and the delivery acceptance checklist, and do not draw conclusions from impressions.

Raise the limit, change batching, or hunt for leaks: trade-offs among three paths

After confirming it is a memory problem, the common disagreement is whether to "raise the limit first to hold things over" or "go straight to hunting the leak." Based on 2026 delivery experience, these two paths are not opposites; the key is the shape of the memory curve. The following comparison can be used to check:

• Step-shaped (rises with traffic, falls back after the peak): This is a capacity issue. Raise the limit first and leave 20%–30% headroom, while pairing it with rate limiting or scaling out for one to two weeks of observation.
• Gentle-slope (does not fall back during off-peak, stable slope): Suspected leak. Raising the limit only pushes the restart time later; a common approach is to take two memory snapshots several hours apart and compare them to locate the object that keeps growing. Locating usually takes one to three days, and the fix schedule depends on the iteration cadence.
• Pulse-shaped (concentrated after a certain type of operation): Change the implementation first. For operations such as large file parsing, batch exports, and full warm-up, switching to batched or streaming processing is usually cheaper than adding memory; the typical change scope is half a day to two days.
• Numbers-not-aligned (container limit and heap limit too close): Reconcile the numbers before discussing optimization; set the heap limit to 50%–75% of the container limit to leave space for off-heap memory. Parameter tuning itself usually takes under half a day, but an observation period of one to two weeks is recommended.

One easily overlooked boundary: raising the limit is not always a safe action. Physical memory on the machine is finite; once the limit exceeds what the node can allocate, the container may fail to start or be rejected by the scheduler, turning into a release incident instead.

A common constraint in projects is: the client provides only one fixed-spec machine, the delivery deadline is one week away, and architecture changes are not accepted. The usual approach then is to reconcile the four layers first, set the heap limit to the typical range of five to seven tenths of the container limit, change batch exports to streaming file writes, and lower the log level from debug back to normal. The result is that nighttime restarts no longer occur during the delivery observation period; the cost is that the export API time goes from a dozen or so seconds to dozens of seconds, requiring the front end to add a progress indicator, and this revision commonly takes half a day to a day. Xiyue Company has followed the same order in similar deliveries: align the memory numbers first, then touch the business implementation, and only then discuss scaling out.

Applicable scenarios and boundaries

This troubleshooting approach fits services running in containers or managed environments that have restarted with no error logs and already have basic memory monitoring. If the process crash left a full stack trace and heap dump, that is an application-level exception, and handling it as a code problem first is more direct; if it is just an internal single-machine tool that restarts once a week and does not affect the business, investing in a memory profile is not cost-effective.

• Suitable: Multi-instance external services, clustered nighttime restarts, memory curves with a clear shape, and container events that can be queried.
• Not necessary: Local development environments, one-off scripts, crashes with complete application logs, and stages with no memory monitoring at all.
• Do something else first: If restarts come with many API timeouts and slow queries, handle the slow queries first; exceeding memory may just be a symptom dragged along by them.

FAQ

Can I just set the container memory limit to the machine's physical memory?

Not recommended. A single container consuming all physical memory leaves no headroom for system processes and neighboring containers; the common practice is to reserve 20%–30% for the system and other processes on the same machine.

Does exit code 137 always mean the memory limit was exceeded?

No. 137 means the process was terminated by SIGKILL; exceeding memory is a common cause, but manual termination and node eviction can also produce it, so judge together with kernel records.

If a Java service already has a heap limit set, can it still be killed for memory?

Yes. Thread stacks, metaspace, and direct memory are not in the heap; the container counts total process memory, so if the heap limit and container limit are too close, termination becomes easy.

If memory rises slowly, must it be handled immediately?

Look at the slope. If it does not fall back during off-peak hours and rises steadily every day, the common approach is to schedule it into the iteration plan; if it will take weeks to reach the limit and alerts already exist, you can observe first and then set a date.

What is the cost of adding memory first to hold things over for a while?

It only buys time. If the growth slope has not changed, it usually just pushes the restart later while masking the real growth point, making later diagnosis harder instead.

For next steps, a three-step approach is recommended: first confirm the container termination reason and kernel records, then use the four-layer check to reconcile the container limit, heap limit, off-heap overhead, and system overhead, and finally choose parameter tuning, implementation changes, or leak hunting based on the shape of the memory curve. This method fits online services with monitoring and access to container events; if only internal tools are affected and observability is lacking, filling in monitoring first is more cost-effective than changing code first—that is the boundary here.