LiteLLM Proxy Setup for Multi-Model Routing: From Zero to Production

Config.yaml walkthrough. Here's what that actually involved.

The Setup

Model routing rules

What I Built

Config.yaml walkthrough

Config.yaml walkthrough. This is where most implementations go wrong — not in the concept, but in the specifics that documentation tends to gloss over. The version that's actually running in production looks different from what the README describes.

Model routing rules

Model routing rules. This is where most implementations go wrong — not in the concept, but in the specifics that documentation tends to gloss over. The version that's actually running in production looks different from what the README describes.

Fallbacks

Fallbacks. This is where most implementations go wrong — not in the concept, but in the specifics that documentation tends to gloss over. The version that's actually running in production looks different from what the README describes.

What Broke First

Tool calling fix. This is the part that took the most time to figure out — not the implementation itself, but the failure mode you only encounter at the wrong moment. The fix is in the tooling, not the concept.

What's Running in Production

The stack: litellm, llm, proxy, ollama. Monitoring via Zabbix with a custom alert template. The rule I apply to everything in production: if it can't self-report a failure, it doesn't go in.

Key Takeaways

1. Config.yaml — config.yaml walkthrough
2. Model — model routing rules
3. Fallbacks — fallbacks

Questions or a different approach? Reply to The Operator's Edge newsletter — I read every response.

Live Life Automated covers the broader philosophy behind how I think about automation and building systems that don't require constant babysitting. Available at mfitz.net.