Two Machines Are Running the Same Scheduled Job and Month-End Produces Two Reconciliation Statements—What Part Can a Distributed Lock Actually Block?

When the same scheduled job is triggered on two or more machines in a cluster and finance receives two reconciliation statements, the usual cause is not a broken scheduler but unconverged ownership of “who triggers it” and a task that is not idempotent. Based on common 2026 delivery experience, a distributed lock mainly blocks concurrent triggering; it does not block timeout redelivery, manual reruns, historical data recalculation, or external side effects such as push notifications and deductions. For entry points like reconciliation, settlement, and coupon issuance, scheduling-layer convergence, execution-layer idempotency keys, and daily reconciliation alerts usually need to be treated separately.

First Distinguish Duplicate Triggering from Duplicate Execution

The two paths look similar but require different handling. Duplicate triggering is a scheduling-layer problem: the same trigger time is claimed by multiple instances. Duplicate execution is an execution-layer problem: two rows appear for the same business date in the business table, while the scheduling log may show only one instance. The judgment order is fixed: check the scheduling log first, then the business table; when the two disagree, the business table is authoritative.

A common misdiagnosis: a job runs for 12 minutes while the platform timeout threshold is set to 10 minutes, so it is marked failed and automatically redelivered; the second execution starts before the first has finished. Adding a lock helps, but if the timeout threshold is not adjusted, duplication is merely postponed until after the lock expires. Reconciliation, settlement, and coupon issuance write externally visible results on every execution, so duplication costs manual reversal; read-only aggregation can usually tolerate an extra run. First determine whether side effects are internal or external, then decide which layer the duplicate-prevention should cover.

• Multiple instance IDs at the same trigger time in the scheduling log: check leader election or sharding first.
• Only one instance ID, but two rows for the same business date in the business table: check execution retries or manual reruns first.
• Job duration approaches or exceeds the platform timeout threshold: adjust the threshold first, then evaluate locking.
• A job calls downstream systems in batches and reruns the whole job after one batch fails: successful batches will be repeated, so make idempotency batch-specific.

What Each of the Three Layers Blocks: Scheduling, Execution, and Reconciliation

The three layers answer three questions: who is running at the same moment, whether duplicate runs will cause errors, and how long it takes to discover an error. The probability of occurrence decreases from the first to the last, while the cost per fix increases, so the layer to implement depends on the size of the side effects. Doing only the scheduling layer means betting risk on the reliability of the lock.

1. Scheduling-layer convergence: ensure that only one instance claims the same job at the same moment. This can be done with scheduler-platform sharding or leader election, single-instance deployment, or preemption through a database task table. The lock must support renewal; if a job runs for 20 minutes while the lock expires in 5 minutes, a second machine will enter.
2. Execution-layer idempotency: give the job a business idempotency key, commonly written as “task code + business date + document number”. Before execution, try to insert an execution record; continue only if the insert succeeds. The idempotency key must include the business date; otherwise, the next day’s normal execution will be blocked by the previous day’s record.
3. Reconciliation-layer fallback: compare row counts and total amounts between the source table and the result table every day, and alert when the difference exceeds a threshold. The alert must identify the specific task code and business date; reporting only “reconciliation mismatch” is as good as no alert at all.

The three layers do not have to be implemented all at once. For a small system deployed as a single instance and completed within an internal loop, the scheduling layer is already converged, so the focus should be on execution-layer idempotency. If a scheduling platform is already in use, start with the platform’s built-in sharding and failure policies, then add reconciliation alerts. In terms of order, make results discoverable first, then optimize how triggering works.

Comparing Cost and Failure Points of Common Approaches

Which approach to choose depends on the existing infrastructure and the level of task side effects. The time periods below are experience ranges estimated from common 2026 project delivery cadence; they vary with team familiarity and should not be treated directly as scheduling commitments.

• Single-instance deployment: split the job into a separate deployment unit so only one copy is running. Low cost, with a typical onboarding period of about 1 day; the failure point is that the job stops if that instance goes down, making it suitable for batch jobs that can tolerate several hours of delay.
• Database task-table preemption: use one status row to decide who runs. Low cost, with a typical period of 1–3 days; the failure point is that the status stays stuck in “running” after an instance crashes, so timeout recovery is required.
• Redis distributed lock: fast to lock and easy to integrate. Typical period of 2–5 days, including renewal, release validation, and exception handling; failure points include primary-replica failover, lock lease expiration, and long instance pauses.
• Scheduler-platform sharding: the platform decides which instance gets the job. Typical period of 1–3 days; the failure point is that a wrong sharding rule can assign the same batch of data to two instances, so execution-layer fallback is still required.
• Business unique index: closer to the result. The period depends on table data volume, typically 1–3 days; the failure point is that it only blocks writes that can have a unique key and does not block external calls such as push notifications or deductions.

From the Delivery Floor: The Lock Exists, but the Job Still Runs Twice

A common constraint in projects: the client already has two application servers, no scheduling platform, only two weeks left in the schedule, and a budget that does not support another middleware purchase. The simpler approach is to split the job into a separate single-instance deployment. If the client insists on not adding instances, use a one-row task lock in the database for preemption, with 15-minute timeout recovery. The cost usually appears in lock details: the first version had no renewal, the job ran past 10 minutes and was taken over by the second machine, and rework before acceptance added renewal and release validation, costing 1–2 extra days. This type of rework falls within the typical range of 1–2 days; renewal and release validation should be checked before go-live.

Applicable and Non-Applicable Boundaries

Duplicate prevention usually requires two conditions to be true at the same time: multi-instance deployment and a task with externally visible side effects. If only one condition is met, execution-layer idempotency and reconciliation alerts are usually more cost-effective than adding a distributed lock. If neither is met, generally nothing needs to change.

• Suitable: multi-instance deployment, and the task performs push notifications, deductions, coupon issuance, or balance changes.
• Suitable: job duration approaches the scheduling interval or platform timeout threshold, making redelivery likely.
• Suitable: the back office provides manual rerun or replay entry points, and manual operations collide with automatic scheduling.
• Not necessary: single-instance deployment, and the task only performs read aggregation, so duplicate execution produces no visible results.
• Not necessary: the task is naturally idempotent, for example a full overwrite recalculation by business date, where running once and running twice produce the same result.
• Not necessary: the operational cost introduced by locking (renewal, primary-replica failover, release validation) is clearly higher than the business cost of one duplicate run.

One boundary that can be quoted independently: When a task is already idempotent, the operational complexity of a duplicate-prevention lock often exceeds the cost of one duplicate execution; such tasks are better served by focusing effort on reconciliation and alerting.

Frequently Asked Questions

Can a single Redis lock completely prevent duplicate execution?

No. The lock mainly covers concurrent triggering; timeout redelivery, manual reruns, and downstream failure retries can bypass it, so a business idempotency key and a reconciliation layer are still needed.

What is a suitable lock expiration time?

The experience range is 1.5–2 times the job’s average duration, with renewal enabled. Prefer renewal over a very large fixed value; otherwise, after an instance crashes, no one will take over the job for a long time.

If a job times out and is redelivered by the platform, does that count as duplicate triggering?

Yes. This type of duplication comes from the scheduler platform’s retry policy. You can first increase the timeout threshold or disable automatic redelivery, switch to alerting with manual confirmation, and then evaluate locking.

If the business table already has a unique index, is a distributed lock still needed?

A unique index blocks duplicate writes but not duplicate push notifications or deductions. It is suitable as an execution-layer fallback and should not be relied on alone.

Do scheduled jobs deployed on a single machine also need all three layers?

Usually a distributed lock is not needed. With a single instance, the scheduling layer is already converged; just implement execution-layer idempotency and reconciliation alerts. Whether to add a business key depends on the side effects.

If there is room for only one round of changes, first classify scheduled jobs by “whether they have external side effects”: leave jobs without side effects as they are; for jobs with side effects, add business idempotency keys and daily reconciliation alerts first, and move scheduling-layer leader election and sharding to the next iteration. Xiyue Company usually follows this order in delivery as well: secure the results first, then discuss trigger optimization. Doing the reverse and adding a lock first easily exposes the same problem again when the lock expires or primary-replica failover occurs.