The threshold has fired — but what actually happened?
Ticket sales begin in one hour. Traffic is rising quickly, and a static threshold triggers an alert. The on-call engineer can see the spike, but the chart does not answer the important question: is this the expected arrival of the audience or the start of a failure?
Simply disabling the alert is risky. If every peak is treated as a normal event effect, the team might miss an overloaded database or a failing dependency. Waking the whole team for predictable traffic is not a good answer either. Repeated false alarms exhaust people and make future alerts easier to ignore.
A practical solution is to compare the current value with a dynamic baseline rather than one fixed threshold. The baseline shows what traffic normally looks like at this time, in this region, and for this component.
Several weeks before the event: build the baseline
For every minute of the event, take the matching minutes from the previous four weeks. For example, the baseline for Tuesday at 19:30 can use the values from 19:30 on the four preceding Tuesdays. Use their median as the expected value.
This method reflects the normal weekly rhythm and is less affected by one isolated spike. The history still needs review. A major launch, an earlier outage, or missing measurements can distort otherwise useful data.
Do not combine every country into one chart. An evening peak in one region may coincide with a nightly low in another. Build separate baselines for important regions, traffic classes, and critical components. Excessive splitting is also a problem: in a segment with almost no traffic, one unexpected request can look like enormous growth.
For normalization, divide the current value by the baseline:
ratio = current traffic / baseline
Suppose the normal value is 10,000 requests per minute and the current value is 20,000. The ratio is 2. To display increases and decreases symmetrically, apply log₂: a ratio of 2 becomes +1, a ratio of 1 becomes 0, and a ratio of 0.5 becomes −1. A doubling and a halving are therefore equally visible on the chart.
The day before: turn the calendar into context
Mark the start and end of the event in the observability system, along with expected audience waves and scheduled mass notifications. These markers do not prove that a change is safe. They tell the on-call engineer why behavior may be changing at that particular moment.
Run a short rehearsal. If the team expects traffic to double but a load test has confirmed stable operation only at 1.6 times the baseline, changing an alert threshold will not solve the problem. Add capacity, reduce expensive request work, or revise the event plan. Check scaling, connection limits, and queues. Safely warm the caches that must absorb the first wave of requests.
Every temporary alert rule needs an owner, activation time, and end time. Assign one person to watch the event and agree in advance which signals require escalation to the wider team.
During the event: examine more than traffic
When the alert fires an hour before the start, the on-call engineer first checks the calendar marker, affected region, and normalized deviation. The next step is to inspect three technical signals: latency, error rate, and resource saturation.
If traffic has doubled while latency and errors remain stable and resources stay below a tested limit, the change resembles an expected audience wave. Monitoring can continue without escalating to the whole team.
The conclusion changes when traffic matches the forecast but latency rises, errors accumulate, and the connection pool is exhausted. That is a service problem even though the traffic chart looks normal. A traffic drop should not be dismissed as audience behavior either if it affects only one region and arrives with an increase in errors.
Comparing regions and components helps narrow the search. A simultaneous change across many regions is more likely to match a shared event. A deviation isolated to one API, data center, or access channel needs its own investigation.
Four traps that weaken the method
- One global baseline hides local failures and time-zone differences.
- A traffic-only alert cannot detect slow responses, errors, or exhausted resources.
- Dividing by a near-zero baseline produces enormous ratios. Low-volume segments also need an absolute minimum.
- A temporary rule without an expiry time becomes a permanent monitoring gap.
After the event: restore normal rules
Restore standard thresholds after the event and verify that temporary exceptions are actually disabled. Record the maximum baseline deviation, highest latency, error rate, resource saturation, and outcome of each alert. Do not turn an event-specific peak into the new normal without a deliberate decision.
Cloudflare’s World Cup data case shows that major events can change Internet traffic from minute to minute and that the effect depends on country and local time. The four-week median and log₂ method comes from that analysis. Capacity checks, escalation rules, and cache preparation are general recommendations that each team must adapt to its own service.
Sources
- Cloudflare — How the 2026 World Cup affected Internet traffic: https://blog.cloudflare.com/2026-world-cup-internet-traffic/
Quick checklist
- collect per-minute data for the previous four weeks
- split the baseline by region and local time
- mark the start, phases, and end of the event on the calendar
- check traffic together with latency, errors, and resources
- test capacity headroom and warm the required caches
- set a time for automatically restoring normal alerts
Prepare a service monitoring plan for a major event
Help me prepare service monitoring for a scheduled event. First ask me for these inputs: - event date, time, duration, and affected regions; - main user journeys and service components; - per-minute traffic metrics for at least the previous four weeks; - current alert rules; - tested maximum capacity and required headroom; - available latency, error, and resource saturation metrics. Do not invent missing thresholds. Mark assumptions and ask follow-up questions. Return the result in this format: 1. Available inputs and gaps. 2. Baseline calculation method. 3. Regional and component breakdowns. 4. Alert rules before, during, and after the event. 5. Capacity and cache checks. 6. On-call engineer action sequence. 7. Conditions for restoring normal rules.