I definitely identify with the "Thinker" described in this post, but I am routinely mulling over ideas and problems (multiple hours, days, and weeks) with the help of LLMs now.
One of the benefits of LLM usage is to figure out the boundaries of your own knowledge and that of humanity's existing knowledge--at least for the LLM's training data.
Enumerating through existing options and existing solutions to problems gets you to the knowledge boundary sooner--where the real work begins! While faster with LLMs, I don't see this process as much different than bouncing ideas off of colleagues (and critiquing your own thoughts).
However, the difference is likely human's unpredictable ability to apply creativity throughout the process...such that a new solution may arise at any point and leap-frog existing solutions/explanations. (Think Einstein taking known data from Lorentz, Michelson, Morley plus Maxwell's equations on light and coming up with special relativity.)
I'm developing a new programming language, so I have to provide a way for LLMs to know about and generate code for a language they have not seen (i.e., have no training data for).
My tooling was previously adding in AI hints with CLAUDE.md, Cursor Rules, Windsurf Rules, AGENTS.md, etc., but I recently switched to using only AGENTS.md and SKILLS. I appreciate the standardization from this perspective.
> Yet, this is our chance. This is our chance to actually build a company. We have to take it. Another chance may never present itself.
Bah! Two weeks or bust? This is a very short-sited viewpoint: think long term! Slow down and think about what another 3 or 6 months of runway would do for your company and product.
You have skills people want. Do some consulting and give yourself a longer runway. If you truly believe you've got a good product and business on your hands, then find a way to keep it going even if it's on the side while you do a consulting gig. (I know split-focus for a few months is not ideal, but "two weeks or bust" seems like a poor alternative.)
I'd like to see some more information about the local setup, including hardware and the postgresql.conf. Otherwise, this tells me very little in terms of comparison.
In anyway this test won't tell you much, just how different systems behave to bigger load.
The local setup was quite usual: PostgreSQL 9.2, Mint 13, default conf in VirtualBox in iMac i5 12GB. (read: home computer, no tuning)
For me the result is that mostly RedShift is on par with local PostgreSQL, sometimes even winning for <5M rows. So with better PostgreSQL tuning you can probably stretch it, but not for as much as RedShift can do for REALLY big data.
Also the big deal was that RedShift scaled linearly.
It scaled linearly, but also went unresponsive for five minutes. (Yeah yeah, it's a new service).
The default Postgres configuration is pretty weak. work_mem is set way to low, for instance, and that's bitten me a few times. I wouldn't say it's unrealistic--lots of people run with it that way in production and never find out how easily they could speed things up. Even me, for years.
But ultimately I'm more swayed by your interaction with it and I hate the endless benchmark tweaking that comes after every blog post about performance testing stuff. The point of this Redshift thing is hugeness first and foremost, so it's interesting.
At least on the small scale, I'd expect realistic "local" hosting to outperform any offering from Amazon.
For example, until your data scales above 20 GB, you'd be able to host it on a $5 SSD-based server from Digital Ocean. Most databases are bottlenecked by I/O. So switching to SSDs gives you the biggest performance boost.
On the high end, Amazon's offer probably will be better. (After all, the major draw to Amazon is "automatic scaling", so that you don't have to worry about Replication or other server administration duties at the high end). But considering how powerful a $5 SSD-virtual machine is today, I think a more realistic test would be with some sort of SSD-based cloud server.
Like others have mentioned, Postgres and Redshift are very different animals, Postgres is a row store and Redshift is a column store. On large data sets analytic queries that return a few columns will significantly outperform a row store DB.
We have found that Redshift is comparable to other columnar databases we work with, while we cannot publish any comparative benchmarks, we did put a blog post on what we found (link in another comment here)
Been there, done that. I rm -rf'd a bunch of important files once, and at the time grep was giving me "memory exhausted" errors. I was able to use strings to grab all of the text of the disk, and then wade through the results with vim.
