A lot of people are unaware of how much husbandry hunter-gatherers still did. They may not have been farmers, with field plantings, weeding, and harvest seasons designed to maximize every acre under production, but they weren't stupid and still spent time to bend the world around them to the task of providing them more resources than it would have on its own. On a bang-for-the-buck basis they often did better than the farmers, it's just that without farming you can't get that density and net return.
There's a number of these problems that end up in a cycle because the solution to some problem is so effective that a later generation forgets what the problem even was. So they remove the solutions. Then they rediscover why they were there in the first place.
It's a Chesterton's Oscillating Fence, on a generation time scale.
Since starting to use AIs seriously for search in the last 3 months I have read and referenced more published papers and academic primary sources then I think I did in the previous 5 years. They're fantastic for pointing at some claim and asking for the primary source for it, and then it does the work of following things through the layers of backref to the original, assuming it's online. Opening the library up to AI access makes it far more accessible and usable then it was before.
AIs, at least in their current form, make you more who ever you were. If you want snap, glib answers of dubious accuracy, they'll give them to you, more easily than ever before. If you want to dig back into primary sources and get the original content, they'll do that for you, more easily than ever before.
Can't speak to how the science infrastructure is going to handle them, but if it takes down the peer review system, which I think has been worthless for probably going on two decades and has just given the entire enterprise a false sense of assurance, it'll probably be a net gain in the end. Peer review is a source of more problems then it is solving right now.
I would at least consider the possibility that they already are, and this is how it is going. It could be a fundamental architecture problem for LLMs. It's not like "slop" is a new problem and I'm sure they'd love to announce a new model that generates much less "slop". The fact they've never so much as mentioned it suggests to me that it's not something they've been able to fix.
It's because of what's in the second paragraph, which I will paste here for convenience:
"Taking a step back, Go manages memory by allocating objects of the same size class (an object’s size is rounded up to the nearest size class) within a contiguous chunk (or span in Go terminology) of one or more 8KiB pages. Size-segregated allocation is common in some malloc implementations (like tcmalloc, which Go’s allocator descends from)."
(No passive-aggressive snark about reading the article intended; it's a good question and I really am just pasting it for our convenience.)
You can't get the inability to allocate some large object because there's a spray of small objects in its way, because the small objects don't share space with the large object. The large objects live in their own space, and the fragmented small objects can be efficiently utilized later by putting other small things in the empty space later.
In the 64-bit world, you also don't have to worry about how you arrange things in physical RAM so one size doesn't end up impacting another. You can always allocate some suitably-sized new chunk of ram that's incredibly distant in the virtual address space and let the OS map that back to the real RAM. It may do something more clever than that because there is still 32-bit Go and that trick is less freeing in that scenario, but the principle would still hold. You don't get the inability to allocate a large object by small objects because they don't live in the same place. You might still "run out of RAM" before you've quite literally run out of RAM, but you'll get closer.
And ultimately that's a problem shared by a lot of memory allocation schemes, not particular to GC or Go. For a lot of reasons, it's a good idea not to run resource usages right up to 100% if you can avoid it and you can expect across a wide range of resources types to encounter problems and expect to do a lot of careful work to make it possible to hit truly full utilization if you need it for some reason. Beyond computers, even... it's rarely a good idea to plan on 100% utilization of anything be it physical or electronic.
> You can't get the inability to allocate some large object because there's a spray of small objects in its way ... The large objects live in their own space
Assume for the sake of argument that the large object space is for objects >=1MB.
Allocate lots of 1MB objects then free every other one (by address).
Unless you're willing to let the large object space grow without bounds....
There is a theorem that is often stated in operating system courses as "For any possible allocation algorithm, there exist streams of allocation and deallocation requests that defeat the allocator and force it into severe fragmentation.". You can ask your friendly local AI about "Bounds for Some Functions Concerning Dynamic Storage Allocation" by J. M. Robson in the July 1974 Journal of the ACM and some various follow-up papers. For any non-compacting memory management algorithm, including traditional malloc/free, there will always be a way of defeating it and forcing it to fragment.
However, consider that Go has been in production for 14 years now, and one of its bread-and-butter applications is network servers, which will collectively exercise quite a bit of the memory allocation pattern space, including some pathological aspects of it. You should expect to need to do better than that to really throw it for a loop.
While the theorem proves some such sequence exists, there's no guarantee that the sequences will be easy to describe in some sort of English sentence.
I haven’t read the article, but for allocations that large, chances are they get allocated as entire memory pages and the garbage collector returns that memory to the OS.
Also, even if it doesn’t, in a 64-bit address space it takes lots of 1MB objects to make that cause problems (there’s room for over 10¹⁶ of such objects)
One of the drawbacks of relying on the virtual memory subsystem is large page tables, that the machine has to manage on your behalf, and that pollute (or at least occupy) caches. That is another reason why Go, like all other software, benefits significantly if you adjust your Linux boxes to use huge pages, or larger-than-default pages, or ARM contiguous page bits, or whatever other features your platform offers to make virtual address translation more efficient. It is unfortunate that Linux usually comes out of the box with all of these post-286 features disabled.
AFAIK, FreeBSD has been doing transparent superpages for decades (I think it was implemented in 2002 for x86 [1], and 2014 for arm [2]) and I don't know of any real issues with it? (I'm sure you could build a test case where it thrashes and causes trouble) Not sure why Linux wouldn't do the same??
Linux also has support for it but it is up to the distribution, or the user, to enable or disable it. The whole discourse was poisoned years ago when the author of Redis told everyone to disable THP on Linux, but this was caused by Redis being a poor program, not by THP being a poor feature. Unfortunately, even though the Redis project finally removed their document about this, many people still carry this bias.
There was also an issue with Linux about 8 years ago where the THP daemon would start busy-looping searching for pages to amalgamate, wasting loads of CPU time (and I believe freezing the processes it was inspecting), and so people were advising switching off THP for that reason until it was fixed. It hit our large Java processes quite badly.
> The whole discourse was poisoned years ago when the author of Redis told everyone to disable THP on Linux
What's the story behind that? Do normal programs really get affected by this?
Normal programs are most likely using glibc's memory allocator, and I'd be surprised if it was incapable of handling arbitrary page sizes. Only reason why I have to care is I implemented my own memory allocator.
Redis uses jemalloc by default, if I recall correctly, but its hostility to the way systems actually work arises from the way that it forks, then changes one bit on every page in the entire virtual space, which causes a lot of kernel work to support copy-on-write by blowing up huge pages into smaller pages. That's what happens when your program is antagonistic to the way the machine actually works.
> then changes one bit on every page in the entire virtual space
Yeah that sucks. Naively implemented garbage collectors have the same problem: they put the live and mark bits in the object itself which spreads those bits all over the address space. This leads to the garbage collector touching every single page when it scans and writes all of those bits.
The proper solution is to allocate separate bitmap pages. This dramatically improves cache efficiency. Machines always want a structure of arrays.
I've always struggled a bit with the fact that "machines want SoA but readability/clarity/etc is easier with AoS". And wonder if it would be possible for a language to have the code representation be AoS but the implementation transparently be SoA.
I would advise reconsidering keeping the current format exactly. Something that would allow conventional email addresses to kept in a list that disambiguates them unambiguously would probably be a good idea.
You don't want to embed the entire email in JSON. Too many parsers past, present, and future tend to want to decode the entire JSON document into memory before doing anything else, and so you'd be forcing entire email documents to be held in memory at scale, which causes you major problems. I'd recommend something more like either a JSON header that defines what parts are in the email, followed by a normal MIME document, which is perfectly normal in HTTP with other things that use the format as well, or having the top level continue to be a MIME document and specify that the first part must be a JSON document containing the headers. JSON replacing the headers is generally something that would be a good idea, though.
That improves the ability of more language environments to be able to handle the email as a stream or in chunks with "normal" libraries and practices. MIME parsers in languages that tend to favor pulling everything into RAM as a string are still more likely to have been forced to face this issue already.
> You don't want to embed the entire email in JSON. Too many parsers past, present, and future tend to want to decode the entire JSON document into memory before doing anything else, and so you'd be forcing entire email documents to be held in memory at scale
Why? Even if your system loads emails into memory a whole email at a time, that doesn't imply that it will load large numbers of emails simultaneously. The kind of processing that can be done by stream-processing large numbers of emails in parallel seems rather limited to me.
> I'd recommend something more like either a JSON header that defines what parts are in the email, followed by a normal MIME document, which is perfectly normal in HTTP with other things that use the format as well, or having the top level continue to be a MIME document and specify that the first part must be a JSON document containing the headers. JSON replacing the headers is generally something that would be a good idea, though.
If we're specifically talking about being able to short-circuit after headers, then why not a JSON header followed by JSON representation of the body?
Compared to the bailout they're going to get at some point, because they're vital to national security and whatever, I think you could easily be looking at a 1000x+ return on investment on those numbers.
Just as an addendum to some of the other fine replies you've got, one of the things I'm interested in for formal verification is security assertions, and those don't necessarily need lots of comprehension to apply. If you can assert "this code never hits the network again" or "this code will never execute arbitrary code", the proof of that may be nontrivial but the comprehension of it is easy.
Proving that some complicated algorithm is actually what you think it is, even after you've proved that it does whatever it is it does, is hard. But to me, and me personally so not necessarily what everyone else thinks, that's not really what I'm interested in from proofs. I'm really more interested in these overarching declarations of what the code definitely doesn't do, or putting bounds on what it does, rather than proving that my billing code bills the customers precisely in the way I expect, when, frankly, the best specification of it I have is already what is in the unproved code base anyhow. I believe those have a very sensible story as to how both humans and AIs can use those assertions.
I would say that especially for books out of copyright, they are unquestionably legally in the right. There is no legal standing for "I like books and it makes me feel squicky when someone disrespects them". Nothing stops you from buying old books and using them as decoupage[1] fodder, using them to start a campfire, or making paper airplanes out of their paper.
Moreover, while it is emotionally appealing to some people to want to add some sort of "responsibility to society" to people who own the old books, it's a very emotional plea that can't really be manifested in the real world. As already pointed out in other places, "an old book" itself doesn't really mean much in terms of what its value is in any particular dimension. Plus, I am always deeply suspicious of anything that expands to "Other people, who are not me, should expend vast quantities of resources so that in the next five or ten times I think about this issue for the rest of my life I feel slightly better about this issue" which is what this really amounts to. I think that as superficially appealing as that may be, it's really a very hostile and demanding position to take.
Personally, to the extent that I would want to lay a "social responsibility" on the AI companies, I'd like to see something like they are either obligated, or ideally, just do it of their own free will, to make the scans of the books that are out of copyright available for some reasonable fee (ideally, "free because we like the PR", but given the scope demanding it be free is not reasonable), and without them trying to lay any further claims on the public-domain results. Trading "one old book somewhere, inaccessible to the world" for "a scan of the book and an OCR of it" I would judge a net win for society for rather a lot of these old books, which are by no means "worthless" sitting in some old collection somewhere but would be a lot more useful for being available.
They could give a copy of the data once to some third party organisation that then seeds it in bittorrent or something like that. Basically, what I want to say is that this doesn't need to be an ongoing obligation for the scanner to be worthwhile for society.
This looks like a good overview paper for interested folk: https://pmc.ncbi.nlm.nih.gov/articles/PMC3048989/
reply