We are just now looking into LLM Gateways and LiteLLM was one I was considering looking into. I’m curious to hear more about what makes the code quality garbage.

I've deployed LiteLLM proxy in a number of locations and we're looking to swap it out (probably to Bifrost), we've seen many bugs with it that never should have made it to a release. Most stem from poor code quality or what I'd classify as poor development practises. It's also slow, it doesn't scale well and adds a lot of latency.

Bugs include but are not limited to multiple ways budget limits aren't enforced, parameter handling issues, configuration / state mismatches etc...

What makes this worse is if you come to the devs with the problem, a solution and even a PR it's very difficult to get them to understand or action it - let alone see critical things like major budget blowouts as a priority.

What about forking it for your own use? Not worth it for the bugs you had fixes for?

Not worth the technical debt and architecture of the codebase. To be honest I'd sooner completely rewrite it in Golang/Rust or otherwise.

How do you like bugs where tools are not working, but only for Ollama provider and only when streaming is enabled? This is one of the real instances I had to debug with LiteLLM.

I personally had no issues using the client libs, my only complaint was that they only offer official Python ones would love to see them publish a typescript one