Why do morning peak timeouts get worse after raising the connection pool limit from 20 to 50, and who is holding the connections?

Conclusion first: Morning peak batch API timeouts and Connection is not available logs are commonly caused not by too small a pool limit, but by connections held by slow queries, long transactions, external API calls inside transactions, or leaked connections. The pool is only where the alarm shows first. Based on 2026 project delivery experience, first look at the active connection curve and acquisition wait percentiles, then decide whether to tune the pool or fix SQL; raising only the pool limit tends to push queuing pressure unchanged onto the database max_connections, creating the phenomenon where the larger the setting, the worse it gets.

What is the connection pool error actually saying

A connection pool is a reuse mechanism that creates a batch of database connections at application startup and returns them after use, to save the overhead of frequent handshakes and authentication. When the pool is full, requests do not fail immediately; they enter a wait queue and throw an acquisition timeout after the configured time. Logs commonly show Connection is not available, request timed out after N milliseconds. This only means no connection was obtained at that moment; it does not directly mean a particular SQL statement is slow.

The two root causes require opposite responses: Insufficient capacity means concurrent requests exceed the pool limit for a long time, active connections hug the upper limit, and wait time rises with concurrency. Occupancy means connections are held for a long time by certain operations, active connection count may look low, but turnover is slow. If the direction is wrong, tuning the pool magnifies application-layer queuing into a database-layer problem.

• Look at active connections, idle connections, and threads waiting to acquire connections together; only then can you tell whether the pool is full or connections are held while idle.
• For acquisition latency, look at P95/P99; averages alone easily miss queuing spikes.
• Check timeout distribution: whether timeouts cluster around hourly tasks, report exports, or bulk imports.
• Check database-side Threads_connected, slow query count, and the long transaction list.

First-round diagnosis: three checks and one calculation

This sequence usually produces a preliminary conclusion in two or three hours to half a day, without waiting for a full monitoring dashboard. Its purpose is to use low-cost evidence to rule out directional errors first, then decide whether to change configuration or code.

1. Check the curve: Take the connection curve from 30-60 minutes before peak to the end of peak, put it on the same time axis as request volume, and see whether requests rise and connections follow, or requests are unchanged but connections fill first.
2. Check waits: Pull acquisition latency percentiles. If P95 rises from tens of milliseconds to seconds, requests are queuing; if acquisition is fast but execution is slow, the problem is on the SQL side.
3. Check occupancy: Check database slow queries and long transactions, focusing on whether transactions contain HTTP calls, file processing, or loop queries; on the application side, look at several chains with long connection hold times.
4. Calculate the limit: Instance count multiplied by per-instance pool limit should fall within 70%-80% of the database max_connections, leaving headroom for operations, batch processing, and failover.

When calculating the limit, do not use only the current instance count; use the post-scaling count. Align the curve with release times and scheduled tasks; otherwise hourly reports can easily be mistaken for code regression.

Cost comparison of tuning the pool, fixing SQL, and read-write splitting

The three routes are not mutually exclusive, but order matters: first use low-cost means to shorten connection hold time, then consider capacity expansion. The following are experience ranges under common 2026 delivery cadence; specifics vary with project scale.

• Raise the connection pool: A configuration change, usually completed in half a day to one day, with validation covering 3-5 workday peaks. Suitable when there are few instances, database headroom is sufficient, and concurrency capacity shortage is confirmed; the cost is higher database connection count and memory. Once total pool size exceeds the max_connections headroom, waits are transferred to the database side.
• Fix slow SQL and shorten transactions: Changes are concentrated in a few chains, commonly 1-3 weeks including regression. Suitable when active connections are not high but waits are long; the cost is code changes and regression testing. A common pitfall is adding indexes without checking execution plans.
• Read-write splitting or adding a cache layer: Commonly 3-6 weeks, and also requires handling replication lag and cache consistency. Suitable for read-heavy, write-light workloads where single-database connections are persistently tight; the cost is a longer chain and higher troubleshooting cost.

Rough decision rule: If peak active connections only reach about 60% of the pool limit, do not expand the pool first; if they stay around 90% for a long time and wait percentiles rise in tandem, then evaluate pool expansion and database scaling.

Delivery scenario: the time a larger pool made things worse

A multi-instance backend had 8 application instances, a per-instance pool limit of 20, and a database max_connections in the hundreds. The client wanted to raise the pool limit first to get through morning peak because database expansion required a procurement lead time. The approach was to first capture slow queries and long transactions 30 minutes before peak, which found a report statistic running for over ten seconds inside a transaction, and another chain calling an external API inside a transaction. After moving the statistics to an offline job, shortening transactions, and moving the external call out of the transaction, the pool limit was adjusted back to the experience range of 12-16 per instance (typical range, not a standard value). As a result, morning peak acquisition timeouts basically disappeared, at a cost of about two days of code changes and regression. If they had directly raised each instance to 50, 8 instances would mean 400, plus connections for operations, batch processing, and other systems; database headroom would be eaten up, and the failure would move from the application layer to the database layer.

Applicable and non-applicable boundaries

Suitable: multi-instance application deployment, a clear upper limit on database connections, timeouts concentrated at morning peak or hourly batch tasks, and logs that provide acquisition wait time. In these scenarios, use the three checks and one calculation; usually within one or two days you can distinguish capacity from occupancy.

Not suitable: database connections long-term use only 30%-40% of the limit, and timeouts come from a single SQL statement itself being slow, so the focus is not the pool; for request-billed short-connection forms, pool configuration logic differs from resident services and should be checked against platform specifications; for occasional one or two timeouts with no business impact, adding monitoring and observing first is more appropriate.

Boundary statement: Connection pool tuning manages connection supply and connection hold time; it cannot manage the execution cost of a slow query itself. If SQL latency cannot come down, a larger pool only swaps application queuing for database queuing.

How to judge success after changes

For acceptance, do not just check that errors stopped; observe continuously for 3-5 workdays, covering morning peak and hourly tasks.

• Peak active connections do not exceed 70%-80% of the pool limit, and no longer stay close to the limit for long periods.
• Acquisition wait P95 falls within the experience range of tens of milliseconds to two hundred milliseconds, and P99 has no second-level spikes.
• Acquisition timeouts drop to single digits per day or zero, and are not concentrated at fixed times.
• Database-side connection count leaves 20%-30% headroom, and slow query count declines in tandem.
• Several chains with long hold times have clear upper limits, and transactions no longer contain external calls.

FAQ

Does a larger connection pool make API timeouts less likely?

No. If total pool size exceeds the number of connections the database can tolerate, waits are transferred to the database side. A common practice is to back-calculate from instance count and max_connections, rather than blindly increasing the per-instance setting.

Are Connection is not available and SQL execution timeout the same thing?

No. The former occurs at the connection acquisition stage; the latter occurs after a connection has been obtained and the SQL runs too long. Checking which layer the error stack falls in can save considerable troubleshooting time.

Load testing is fine, but production still times out at morning peak. What is different?

Load tests mostly use fixed concurrency and small data; production has burst traffic plus real data distribution. After slow queries, lock waits, and hourly tasks stack up, connection hold time becomes noticeably longer.

With a cloud database or database proxy, does the application still need a connection pool?

Yes. A proxy converges backend connections, but the application-side pool limit, wait timeout, and leak checks still need to be set. Both sides' limits must be calculated together so they do not fight each other after scaling.

If you are stuck on morning peak timeouts, it is advisable to first take a peak connection curve and acquisition wait percentiles, then compare them with slow queries and long transactions; usually one or two days are enough to distinguish capacity from occupancy. The above judgment applies to resident services with a clear database connection limit and multi-instance deployment; single-machine low-traffic or short-connection forms need not be copied directly, and checking against platform specifications is safer.