Scheduled Jobs Run Fine on One Machine but Duplicate on Multiple Servers — Where Should You Stop Them?

When a scheduled job duplicates across multiple servers, the root cause is usually not a coding mistake but the absence of a binding relationship between the "job" and the "instance": every instance assumes it is the only executor. The common sequence in 2026 is: first confirm whether the job really needs multi-instance parallelism, then choose one of three paths — scheduler sharding, a distributed lock, or a dedicated execution node — rather than rushing to add a lock.

Why it runs fine on one machine but starts duplicating after scaling to multiple instances

With a single-machine deployment, only one process holds the timer, so the job naturally exists as a single copy and the problem is hidden. Once the service is scaled to two or three instances for high availability, as is common in 2026, each instance registers its own timer on startup, so the same point in time gets multiple triggers. If the job involves side-effecting writes such as issuing coupons, deducting inventory, or sending push notifications, one duplicate execution can turn into financial loss or customer complaints.

Before changing anything, distinguish two kinds of duplication. Concurrent duplication happens when several machines execute at almost the same moment; catch-up duplication happens after node restarts or scheduler retries, with triggers staggered over time. The former is solved with a lock or sharding, while the latter requires a "skip if missed" compensation decision in the scheduling layer — the two are handled in different places. The diagnostic method is straightforward: print the instance identifier from the execution logs and look at how many times the same business key appears and the time gaps between those appearances.

• Concurrent duplication: multiple execution records appear within the same second, usually related to lock contention failures or each instance triggering on its own.
• Catch-up duplication: timestamps are scattered, often related to node restarts, scheduler retries, or job timeout decisions.
• Logic problems that look like duplication: the job itself has no idempotency key, and duplicate delivery from upstream produces the same symptom.

Three common paths for stopping duplication, and what each one can prevent

No single path can both completely avoid duplication and provide high availability; it is a trade-off. A distributed lock ensures only one instance executes at any moment, but if the lock TTL, renewal, or network partitions are not handled well, double execution can still occur; scheduler sharding assigns jobs to different instances by business key, which suits batch processing, at the cost of requiring the job itself to be splittable by key; a dedicated execution node separates the scheduling logic from the business service, making deployment simple, at the cost of adding one more component that needs operational attention.

• Distributed lock: typical rework effort is an experience range of 1–3 person-days, suitable for scenarios with few jobs and single executions under 1 minute; renewal and release must be handled, and if the execution time exceeds the TTL, double execution occurs.
• Scheduler sharding: typical rework effort is an experience range of 3–10 person-days, suitable for jobs that can be split by business key, such as reconciliation or batch push; jobs that cannot be split cannot use it.
• Dedicated execution node: typical effort is an experience range of 1–2 person-days, suitable for internal systems with few jobs and moderate high-availability requirements; if this machine goes down, the jobs stop, so extra monitoring is required.

Before choosing a path, ask one question: what is the cost of executing this job one extra time? If it only refreshes a cache or recalculates a statistic, the cost is low and there is no need to introduce a complex mechanism; if it involves money, inventory, or external notifications, one duplicate execution requires manual intervention, so both layers — "scheduling-layer leader election + data-layer unique constraint" — are needed; relying on only one of them is not stable.

Four-step decision: should this job run concurrently, and which layer holds uniqueness?

The following sequence is commonly used in project delivery. The core logic is to first decide "whether to run concurrently," then decide "how to guarantee uniqueness," and only then discuss which component to use. Doing the reverse — picking middleware first and patching in semantics later — is a major source of rework.

1. Define the job's business key. For example, for reconciliation by merchant, the business key is the merchant ID plus the billing period. For a job without a business key, any distributed approach is just a gamble.
2. Decide whether it can run in parallel. If it can be split by business key, prefer sharded parallelism for better throughput; for non-splittable jobs such as full-database aggregation or single-table migration, fall back to globally unique execution.
3. Choose the uniqueness safety net. A common practice is double insurance: scheduling-layer leader election plus a data-layer unique constraint. The scheduling layer ensures it most likely runs only once, and the unique index ensures that even if it does run twice, no dirty writes occur.
4. Add observability. Execution logs should always include job name, instance identifier, business key, duration, and result; missing any one of these adds hours to later troubleshooting.

The acceptance criteria can be very direct: you can write down the business key field, explain which keys can run in parallel, find the corresponding unique constraint in the database, and look up the latest execution by job name in the logging platform. Only when all four are done can you say duplicate execution has been brought under control.

Delivery field notes: where did the log go for the execution that failed to acquire the lock?

There was a typical project with a tight budget and a two-week timeline, running four or five scheduled jobs across three application instances. The initial approach was to add a lock with a 30-second TTL. After launch, one job took 90 seconds per execution, the lock expired early, and a second machine acquired the lock and ran it again, doubling the reconciliation results. The constraint was that the budget did not allow a separate scheduling component, so the approach was to split long jobs into small batches, each acquiring its own lock, and enable lock renewal. The result was that data no longer doubled, at the cost of an extra round of about two days of splitting rework and regression testing. In the retrospective, for long jobs like this, if a single run exceeds 1 minute, batch execution is a typical range; setting the TTL at 2–3 times the P95 also falls within the experience range.

What is easily overlooked here is "should a failed lock acquisition be logged?" If you only print one debug line, later you cannot tell whether the job "was not triggered" or "failed to acquire the lock." By delivery convention, a failed lock acquisition should also be logged at info level, with the job name and instance identifier, so that when confirming how many times it ran today, you can count it directly.

Applicable scenarios and boundaries

There are two preconditions for adding an extra coordination mechanism: the job's side effects cannot be repeated, and the service is indeed deployed in multiple copies. Conversely, if the job merely recalculates an idempotent aggregate result, or the service itself is deployed as a single instance, then adding a distributed lock or deploying a scheduler is over-engineering that only adds operational surface and troubleshooting cost. It is advisable to write this boundary into the design document so that later maintainers do not blindly add locks from a template.

• Suitable for adding a coordination mechanism: side-effecting jobs such as coupon issuance, deductions, reconciliation, external notifications, and batch data migration, where the service is deployed across multiple instances.
• Not necessary: pure computation, overwritable recalculation, and cache warm-up jobs; duplicate execution only wastes a little resources.
• Do not rush to middleware: with fewer than 5 job types running daily, a dedicated execution node is often more cost-effective than introducing a scheduling center.

A few pitfalls hit repeatedly

• Lock TTL shorter than job duration: a common source of double execution. The experience-based practice is to set the TTL at 2–3 times the job's P95 duration and enable automatic renewal.
• Locking only at the start of the job without verifying the holder before writing: after the lock expires, the original instance may still be writing data; you need to confirm you still hold the lock before writing.
• Using a "has it run today" flag table without a unique index: under concurrency, two queries can both determine it has not run, and it still executes twice.
• Nested calls to other jobs within a job: when the call chain gets long, lock granularity becomes unclear, making mutual waiting or missed locks more likely.

Frequently asked questions

Will a scheduled job definitely duplicate when running on multiple instances?

Not necessarily; it depends on the trigger method. Jobs triggered independently by in-application timers will duplicate; jobs triggered by an external scheduling center that assigns them per instance usually run only once, provided the sharding configuration is correct.

Can a database unique index alone, without a distributed lock, hold up?

It can prevent dirty writes, but it cannot prevent duplicate computation or duplicate calls to external APIs. For scenarios with external side effects, use a unique constraint together with scheduling-layer leader election.

How long should a distributed lock TTL be set for stability?

A typical range is 2–3 times the job's P95 duration, with renewal enabled, and also set a timeout cap on the job itself to avoid holding the lock for a long time during an anomaly.

Should scheduled jobs be deployed on the same instances as the business service?

If there are few jobs and they are short, co-deployment is fine; once long jobs start crowding out resources or lock contention becomes frequent, it is advisable to split them onto a dedicated execution node to reduce mutual impact.

How can you quickly confirm how many times it actually ran in production?

Always print the job name, instance identifier, and business key in the execution logs, then count distinct business keys; this is more direct than digging through database result tables afterward.

If you happen to have a job that duplicates across multiple instances, do not change the code first. Spend one day counting the execution logs by job name and instance identifier to confirm whether it is concurrent duplication or catch-up duplication, then follow the four steps above. The applicable premise is that the job has a clear business key and its side effects cannot be repeated; if the job itself can be recalculated idempotently, adding the unique constraint is usually enough, and there is no need to introduce a scheduling component. By the common delivery pace in 2026, for a medium-sized system, this kind of governance can typically be implemented in one to two weeks.