Designing an LLM pipeline that survives provider outages

At Metaverse I built an AI travel planner where two LLM agents cooperate to produce a geolocated itinerary. Models fail, slow down and change behaviour, so the pipeline had to be built for that.

Hide the provider behind one interface

Every stage calls a small provider interface. Swapping a model, or moving between vendors, becomes a configuration change instead of a rewrite.

Retry, then fall back

Transient errors get a bounded retry with backoff. If a provider keeps failing, the stage falls back to the next model in the list, so users get an itinerary instead of an error.

Track latency per stage

One total response time hides where the time goes. Logging each stage separately showed which agent was the bottleneck and gave me real numbers to optimize.