Agree. I think it's improved a bit but Celery is frustrating as the defacto job/queue solution. A lot of the defaults make it unreliable (it can lose jobs if workers crash or don't shutdown cleanly)
I'm hoping the existence of free-threading will push for more first-class concurrency primitives. Concurrent Futures is nice until you need a concurrent-safe data structure besides a queue
Agree that celery is a mess and it doesn't work well with async (Asyncio) python. I think version 6 maybe will support it sometime.
I also had a lot of problem due to async primitives with sqlalchemy - there's some tricky stuff with asyncio.gather vs TaskGroup and how sqlalchemy session works with it to be able to compose code easily.
But weiting a processing pipeline with Python is frustrating if you have worked with C# concurrency.
I figured the best option is Celery and you cannot do it without an external broker. Celery is a mess. I really hate it.