With a similar amount of experience with both languages I found Go much easier to read. I've always been a bit miffed why Python is seen as easy to read for experienced developers. I get the syntax is good for short code or people with little experience but my experience is those readability benefits went away quickly with time or complexity.
Why are you miffed about it? I legitimately hate reading golang with passion and find python to be pretty intuitive, outside of the odd ambitious list comprehensions. I worked in a golang shop for several years, so it's not just an familiarity situation either.
We are just different. That's not something to be mad about.
In my opinion most interpreted languages today tend to produce very dense code. Fancy call chains and closures interleaving. If you look for a subtle bug those are hard to reason about, you have to know the details of a lot of different APIs.
Go is verbose partly for that reason, but a silly loop is a silly loop. The constraints are clear, you only have to do the logic.
Python is a garbage language. Dynamic types are a disaster for maintaining large codebases and we waste enormous amounts of compute running large systems with it.
No we should write one of the many modern programming languages that handle certain projects way better, including kotlin, go, or Java. The only things python is best in class at are scripting and as a harness for high performance c++ or fortran.
I mentioned those explicitly. People are still using it as a backend language for projects like huge SaaS deployments though, with the hypothesis being that dev time is expensive. With modern languages like kotlin and go, I think that gap is much too narrow to justify using a slow and badly designed language with good syntax.
Any language that uses error codes instead of exceptions is a non-starter for me. Produces code that craps all over the happy path.
Python has a different problem: it is slow as f---. I did a micro benchmark comparison against 5 other languages in preparation for my python replacement language. Outside of dictionary lookups, it is 50-600 times slower than C depending on the workload.
Go, Rust etc are fine. They land at 1.25-3x slower than C. But I prefer the readability of python minus its dynamic nature.
Exceptions are not the only alternative to error codes.
Many early programming languages handled errors by providing multiple return addresses to functions.
Thus a function returned to the place of invocation in the normal case, otherwise it returned to one of the error handlers that were grouped at the end of the function that invoked it, away from the happy path.
This was more efficient than the modern way of returning an error code, as it eliminated the superfluous testing of the error code and the expensive conditional jump based on the result of the testing.
Moreover, in my opinion in the majority of the cases maximum information about an error is inside the function that has invoked the function where the error happened and neither in an exception handler several levels above it, nor inside the invoked function, so the place where the error handlers had to be put in this old method is actually optimal for meaningful error messages.
From the point of view of the source text, this old error handling method is equivalent with using exceptions with the constraint of placing the exception handlers in the function that has invoked the function that generates the exception. This limitation enabled a more efficient implementation than for exceptions where the handler can be placed anywhere above the invoked function and a complex stack unwinding may be necessary.
While it sounds interesting, I do not see this gaining traction. My main complaint would be that I do not want to think about errors except when I want to. This system sounds like Java checked exceptions in a way. You have to handle every exception that a function might throw every single time. That eventually resulted in the famous catch-swallow stratagem.
I distinguish between Faults and Errors. Faults are system level events you cannot recover from. Errors are things you may encounter in the course of doing business. Most applications do not really need to do any recovery; simply tell the user something bad happened and ask them to fix the problem.
For workflows that do need error recovery, or functions with logical failure conditions, you are free to use a Result or Optional type.
Also, I have implemented exception handling with stack unwinding for my VM. The code that does it is not really that complex. If you want to retry... then things get messy.
German-style strings is a way to store array of strings for columnar dbs. The idea is to have an array of metadata. Metadata has a fixed size (16 bytes) The metadata includes the string length and either a pair of pointer + string prefix or the full string for short strings. For some operations the string prefix is enough in many cases avoiding the indirection.
> Your coworkers and QA will thank you for learning LINQ and ditching the imperative methods that plague your Python brain.
This is a very unfortunate joke: Python has list (and generator) comprehension expression for a long time (2.3?) which are similar to LINQ. At some point in the history many languages stole useful expressions from other paradigms.
> This is a very unfortunate joke: Python has list (and generator) comprehension expression for a long time (2.3?) which are similar to LINQ.
I love Python, its my main daily driver, both at work and by preference for most of my personal coding, but Python comprehensions and genexps are much more limited than LINQ language level query syntax (Scala’s visually-similar construct is more like LINQ in capabilities) and Python—purely because of core and stdlib convention which also drive convention for the ecosystem, not actual structural features—lacks anything like the method syntax as a common API (unlike, say, Ruby).
EDIT: Thinking about it a little bit, though, it should be possible in theory to implement LINQ in Python without language level changes (including providing something close to but not quite as clean as the language level query syntax[0]) as a library via creative use of inspect.getsource and ast.parse, both for providing the query syntax and for building the underlying expression tree functionality around which providers are built (support for future python versions would require implementing translation layers for the ASTs and rejecting unsupported new constructs). Conceptually, this is similar to how a lot of embedded DSLs in Python for numeric JIT, compiling GPU kernels, etc., from (subsets of) normal Python coded are done.
[0] existing comprehension/genexp syntax looks similar, but relies on simple iteration, not pushing code execution out to a provider which may be doing something very different behind the scenes, like mapping "if..." clauses into SQL WHERE clauses for a database query.
List comprehension is pretty good, but I prefer LINQ method-style because it's executed left-to-right, whereas I keep having to look up the order of Python.
> As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. Which, spoiler alert, is the type we implemented.
Congratulations to the team, Pydantic is an amazing library.
If you find JSON serialization/deserialization a bottleneck, another interesting library (with much less features) for Python is msgspec: https://github.com/jcrist/msgspec
Are there any necessary features that you've found missing in msgspec?
One of the design goals for msgspec (besides much higher performance) was simpler usage. Fewer concepts to wrap your head around, fewer config options to learn about. I personally find pydantic's kitchen sink approach means sometimes I have a hard time understanding what a model will do with a given json structure. IMO the serialization/validation part of your code shouldn't be the most complicated part.
The biggest issue missing from most conversion and validation libraries is creating models from JSON Schema. JS is ideal for central, platform agnostic single source of truth for data structures.
In my use case, I find the lack of features of msgspec more freeing in the long run. Pydantic is good for prototyping, but with msgspec I can build nimble domain specific interfaces with fast serial/deserialisation without having to fight the library. YMMV!
I don't know what were the issues Yelp was facing, I have upgraded several times my past Kafka clusters and really never experienced any issues. Normally the upgrade instructions are documented (e.g. https://kafka.apache.org/31/documentation.html#upgrade) and the regular rolling upgrade comes with no downtimes.
Besides this, operating Kafka never required much effort a part when we needed to re-balance partitions across brokers. Earlier versions of Kafka required to handle it with some external tools to avoid network congestions, but I think this is part of the past now.
On the other hand, Kafka still needs to be used carefully, especially you need to plan topics/partitions/replications/retention but that really depends by the application needs.
I used to work in a project where every other rolling upgrade (Amazon’s managed Kafka offering) would crash all the Streams threads in our Tomcat-based production app, causing downtime due to us having to restart.
The crash happens 10–15 minutes into the downtime window of the first broker. Absolutely no one has been able to figure out why, or even to reproduce the issue.
Running out of things to try, we resorted to randomly changing all sorts of different combinations of consumer group timeouts, which are imho poorly documented so no one really understands which means which anyway. Of course all that tweaking didn’t help either (gunshot debugging never does).
This has been going on for the last two years. As far as I know, the issue still persists. Everyone in that project is dreading Amazon’s monthly patch event.
Check the errors coming back on your poll/commit. The kafka stack should tell you when you can retry items. If it is in the middle of something sometimes it does not always fail nicely but you can retry and it is usually fine. Usually I see that sort of behavior if the whole cluster just 'goes away' (reboots, upgrades, etc). It will yeet out a network error and then just stop doing anything. You have to watch for it and recreate your kafka object (sometimes, sometimes retry is fine). If they are bouncing the whole cluster on you each broker can take a decent amount of time before they are alive again. So if you have 3 and they restart all 3 in quick succession all at once you will see some nasty behavior out of the kafka stack. You can fiddle your retries and timeouts. However, if that is lower than it takes for the cluster to come back you can end up with what looks like a busted kafka stream. I have seen it take anywhere from 3-10 mins for a single broker to restart sometimes (other times it is like 10 seconds). So depending on the upgrade/patch script that can be a decent outage. It goes really sideways if the cluster has a lot of volume to replicate between topics on each broker (your replica factor).
reply