Isn't Apple about to license some variation of this from google for on-device AI? Maybe it’s their sales pitch to Apple and then they will lock it down.
Traditionally those roles of providing "alignment" to overcome organizational inertia were held by program managers. So that seems like IC roles are indeed going away, to be replaced by more "technical program manager" type roles.
Maybe it's my experience, but TPMs were often responsible for coordinating large org-wide or cross-org initiatives. It's prohibitively expensive to have TPMs on anything smaller.
Since engineering with AI is still very technical, I would wager that software engineers would stretch into less technical areas of software development rather than TPMs stretching into technical areas. I only say this as someone with experience with AI and I see how easy it is to write bad code with AI if you're not aware of what it's doing.
Yes, size and performance are not only problems for local LLMs, they are problems for frontier LLM companies like OpenAI and Anthropic. The latter still lose a ton of money on inference and advances in efficient, performant models helps their bottom line.
The argument in the article is backwards. Evals test the stability and boundaries of a concept. They are not created before the concept has been prototyped (which the author acknowledges).
An eval is not somehow breaking silently due to some new capabilities in an LLM. It wouldn't be a good eval if it did. What it does is steer the LLM towards specific goals. If anything, an argument can be made that they restrict creativity and experimentation by narrowing goals.
If the argument is that evals need to written before some new behavior can be devised, that's incorrect. There are an infinite number of evals that test for things which cannot be done. Only when something has been demonstrated to work in a specific context, can an eval be written.
They are addressed but the core of the thesis is still wrong:
> This is the core problem: our entire evaluation infrastructure is structurally reactive. We measure the system after it has changed. We never predict the change.
I applaud the move. It's also a little disingenuous to talk about moral standings when the third opening sentence is "The math hasn’t worked out for a while now." If the numbers were working out, would they continue to turn a blind eye on the privacy tracking?
Too much of code is data transformation. input -> sanitation -> db -> consumer -> api -> client. Business logic defines the shape of that data and some service-level rules but the majority is just shoveling data.
Those are raw numbers. I would look instead at the job changes over total employment numbers. I don't have the numbers but I would wager we have many more people working in tech today (overall) than we did in 2008.
Also, that spike in 21/22 really did a number on people's expectations. The one constant in this industry is its cyclical nature.
Maybe I'm reading the graph wrong, but the decrease comes after years on continuous growth, so total employment numbers in tech should still be absolutely massive, compared to 18 years ago?
If it continues, then yes it could be bad, but so far it seems like a correction for over-hiring in 2021 - 2023. Seems a little weird to be focusing on a decline in 2024 - 2026, without addressing the large increase right in the years before.
There's a lot of dynamics where it's the short-term numbers that matter. If you're a developer who needs a new job after your spouse got transferred to LA or something, it does you no good that the absolute numbers are massive, nor that a different person looking for a job 3 years ago would have found it uncommonly easy.
I had no idea I was in such an exclusive group back in 2000. Everyone I knew was a software engineer or in tech one way or another so I suppose I got a warped sense that I belonged to a larger group.
I'm not sure the nation wide raw statistics are that reliable in the field of software engineering without interpretation.
In the 90s tons of people who were de facto software engineers were listed as "Information Technology Workers". I suspect a lot of that still hasn't been shaken out of the system.
According to the BLS in the year 2000 there were 3.4 million information technology workers.
BLS had some classification changes over the years. I think it's interesting in the "this is how people thought about the role over the decades."
Today there are computer programmers (15-1251), and software developers (15-1252), and web developers (15-1254).
In 2018, there was a reclassification - https://www.dol.gov/sites/dolgov/files/ETA/oflc/Presentation... where 15-1132, Software Developers, Applications and 15-1133, Software Developers, Systems
Software where reclassified into the software developers (15-1252) group.
The other thing that confuses this is that a lot of positions were classified as Computer systems analysts because that's a position that a TN visa can be hired for (there is no software engineer in there... and it wasn't until relatively recently that one could be a "software engineer" in Canada without being an Engineer.
Computer programmers 1010 15-1131
Software developers, applications and systems software 1020 15-1132, 15-1133
Where the "Computer programmer" was the more junior classification and Software developers working on a word processor were classified differently than a software developer working on the operating system... and they were the more senior positions.
Software Developers
Research, design, and develop computer and network software or specialized utility programs. Analyze user needs and develop software solutions, applying principles and techniques of computer science, engineering, and mathematical analysis. Update software or enhance existing software capabilities. May work with computer hardware engineers to integrate hardware and software systems, and develop specifications and performance requirements. May maintain databases within an application area, working individually or coordinating database development as part of a team.
Computer Programmer
Create, modify, and test the code and scripts that allow computer applications to run. Work from specifications drawn up by software and web developers or other individuals. May develop and write computer programs to store, locate, and retrieve specific documents, data, and information.
