It is 6:40 a.m. on a Tuesday in January. A winter storm has settled over two hub airports, and operations control has canceled 140 flights in under two hours. The rebooking service wakes up to thousands of affected itineraries at once. It starts calling the passenger service system (PSS) as fast as it can.
Within minutes, PSS response times climb from milliseconds to tens of seconds. The rebooking service times out, retries, and times out again. By the time the on-call engineer opens the dashboard, roughly 3,000 passengers have received an SMS telling them they are confirmed on a new flight. Their bookings were modified, but their tickets were never reissued. At the airport, agents see reservations with no valid coupon, and the contact center queue passes four hours.
Nothing in that scene was a single bug. It was an architecture working as built, under conditions it was never designed for.
Why disruption breaks the rebooking path
Irregular operations (IROPs) are a worst case for most travel stacks, for three structural reasons.
The core system has a fixed ceiling. Whether you run a hosted PSS, a GDS connection or an in-house core, it usually has limited throughput and interfaces often designed around batch jobs and agent terminals. You cannot autoscale it during a storm. Scaling out your microservices just adds more callers to the same bottleneck.
Rebooking is not one call. Recovering a single passenger involves a chain: find alternatives, hold inventory, modify the PNR, reissue or revalidate the ticket, issue hotel and meal vouchers if rules require it, calculate compensation eligibility, send notifications, and sometimes process a refund instead. When the chain is implemented as a sequence of synchronous calls inside one request handler, a failure at step four leaves steps one through three committed and nothing recorded about what still needs to happen.
Notifications are decoupled from truth. In many stacks, messaging fires as soon as one system says "success," not when the whole recovery is done. That is how a passenger ends up with a confirmation for a seat that has no ticket behind it, followed by a second message from a different channel that contradicts the first.
Add a traffic spike concentrated in a short window, and partial failures become the norm.
Why the usual workarounds fall short
Most teams have tried some of these, and each helps a little.
- Bigger timeouts and more retries. These make the overload worse. Aggressive retries against a struggling PSS multiply the load exactly when it can least absorb it.
- A queue in front of the PSS. Better, because it smooths the spike. But a queue alone does not know that a message represents step three of a seven-step process, or what to undo if step five fails.
- Nightly reconciliation jobs. These find the broken itineraries the next morning, after passengers have already reached the airport.
- Status flags in a database. Hand-rolled state machines with columns like rebook_status work until someone adds a voucher step or a new compensation rule. Then the flags drift, and recovery logic spreads across cron scripts.
The common gap is that none of these treat a passenger's recovery as a single, long-running, durable unit of work.
The general approach: events in, one durable workflow per passenger
A more reliable design separates three concerns.
1. Fan out disruption as events. When operations publishes a flight status change (delayed, canceled, diverted), that event goes onto a pub/sub topic. Consumers react independently: one finds affected passengers, another updates departure boards, another alerts crew scheduling. A slow consumer does not block the others.
2. Start one workflow per affected passenger (or per PNR). Each workflow owns the full recovery for that traveler: propose options, wait for a decision, modify, reissue, issue vouchers, notify. The workflow engine persists progress after every step. If a pod dies halfway through, the workflow resumes from the last completed step instead of starting over or silently stopping. If a step fails permanently, the workflow runs compensating actions, such as releasing held inventory, rather than leaving the booking in limbo.
3. Protect the PSS with explicit policies. Every call into the reservation system goes through a layer that enforces timeouts, bounded retries with exponential backoff, and circuit breakers. When the circuit opens, calls fail fast; pair the breaker with longer workflow-level retries or a durable timer before retrying, so workflows wait instead of failing. Their state is durable, so waiting costs nothing but time. The PSS gets room to recover instead of a retry storm.
Two further design choices matter in travel specifically.
Durable timers for passenger response windows. Many carriers give the passenger a window to accept or decline a rebooking offer. A durable timer survives restarts and deployments, so the offer expires correctly even if the service was redeployed in the meantime. On expiry, the workflow can auto-accept, release the hold or escalate to an agent, according to your policy.
Notify only from the workflow. The workflow is the single source of truth for a passenger's recovery status. Send the "you are confirmed" message as a step that runs only after the ticket is reissued.
Where AI agents fit, carefully
Rebooking is a reasonable place to use AI agents, but only as advisers. An agent can weigh connection times, fare rules, loyalty tier and partner availability, then propose a few ranked options, something rules engines often handle poorly in a badly disrupted network.
The agent should not commit anything on its own. Treat its output as a proposal, and keep the actual PSS modifications in deterministic workflow steps that run only after a passenger or agent confirms. Running the agent inside a durable workflow also helps: after a crash it resumes instead of re-running expensive steps.
Implementing it with Dapr
These principles are vendor-neutral. Dapr, the CNCF graduated open-source runtime, is one practical option because it packages the relevant pieces as building blocks behind a sidecar: pub/sub across brokers like Kafka or RabbitMQ (with CloudEvents and dead-letter topics), Dapr Workflow for durable, event-sourced orchestration, and resiliency policies for timeouts, retries and circuit breakers.
There is precedent in this industry: a low-cost US airline moved to a cloud-first, event-driven architecture on Dapr for scalability and cost efficiency.
A per-passenger rebooking workflow in Python might look like this. The workflow holds seats, sends the offer, and then waits for either the passenger's response or a two-hour timer, whichever comes first.
import dapr.ext.workflow as wf
from datetime import timedelta
wfr = wf.WorkflowRuntime()
pss_retry = wf.RetryPolicy(first_retry_interval=timedelta(seconds=2), max_number_of_attempts=5, backoff_coefficient=2, max_retry_interval=timedelta(minutes=1))
notify_retry = wf.RetryPolicy(first_retry_interval=timedelta(seconds=2), max_number_of_attempts=5, backoff_coefficient=2, max_retry_interval=timedelta(minutes=1))
@wfr.workflow(name="passenger_rebooking")
def passenger_rebooking(ctx: wf.DaprWorkflowContext, data: dict):
offer = yield ctx.call_activity(hold_seats, input=data, retry_policy=pss_retry)
yield ctx.call_activity(send_offer, input=offer, retry_policy=notify_retry)
response = ctx.wait_for_external_event("offer_response")
timer = ctx.create_timer(timedelta(hours=2))
winner = yield wf.when_any([response, timer])
if winner == timer or not response.get_result().get("accepted"):
yield ctx.call_activity(release_seats, input=offer, retry_policy=pss_retry)
return {"status": "released"}
yield ctx.call_activity(reissue_ticket, input=offer, retry_policy=pss_retry)
yield ctx.call_activity(notify_confirmed, input=offer, retry_policy=notify_retry)
return {"status": "confirmed"}
@wfr.activity(name="hold_seats")
def hold_seats(ctx: wf.WorkflowActivityContext, data: dict): …
The orchestrator must be deterministic, so PSS calls, SMS sends and voucher issuance live in activities. Completed activities are not re-executed after a crash, though an activity interrupted mid-run may run again, so make holds and SMS sends idempotent.
On top of the workflow-level retry policy, a Dapr Resiliency resource can apply timeouts and a circuit breaker to the PSS adapter as a whole. Operations can then tune it without a code deploy. Avoid stacking both retry layers on the same call; use the Resiliency resource for timeouts and the circuit breaker, and keep retries in the workflow. If you need hard throughput caps, a dedicated adapter service with a fixed number of workers consuming from a topic gives you natural rate limiting. A large retail bank in Asia throttles incoming traffic to a legacy core system using Dapr without modifying the core, and the same pattern applies to a reservation core.
Running this in production still means operating the runtime: control plane upgrades, certificates, state stores and observability. You can do that yourself on Kubernetes, use a managed integration such as the one in Azure Container Apps, or look at a hosted durable workflow platform built on Dapr. Whichever you choose, load-test against a PSS simulator at realistic storm volumes before trusting any vendor's claims.
A practical roadmap
- Map one real recovery end to end and list every system touched and how each step can fail.
- Publish flight status changes as events, even if only one consumer reads them at first.
- Wrap the PSS behind a single adapter with explicit timeouts, retries and a circuit breaker. Measure its real sustainable throughput.
- Move one flow, such as cancellation rebooking on a single route, into a per-passenger workflow.
- Add compensation for each committing step before adding more steps.
- Move all passenger notifications into the workflow.
- Introduce AI-suggested options only after the deterministic path is stable, and always behind confirmation.
Before your next winter season, check the following:
- ☐ Every PSS write is idempotent or guarded against duplicates.
- ☐ Each workflow step has a defined compensation or an explicit "no undo needed" note.
- ☐ Offer windows use durable timers, not in-memory schedulers.
- ☐ The circuit breaker thresholds have been tested against a slow PSS, not just a dead one.
- ☐ Tracing follows a single passenger from flight event to final SMS.
- ☐ Operations can see how many recoveries are waiting, running, failed or compensated.
- ☐ A game-day exercise has replayed a real past disruption at full volume.
Conclusion
The 140-cancellation morning is not avoidable, but the 3,000 broken tickets are. Treat each passenger's recovery as a durable, event-driven workflow, put explicit limits around the reservation core, and let only the workflow speak to the traveler. Open-source Dapr gives you the building blocks, and providers such as Diagrid offer managed options if you would rather not run the runtime yourself. Either way, the design matters more than the tooling.