> Look at the Railway GCP account ban situation. A literally billion dollar startup running on Google Cloud and Google just randomly snaps their fingers and deletes their account. Zero warning. No phone number to call. No account rep. Poof. Gone. It is actual insanity to me. A billion dollar customer gets the exact same automated middle finger as a low effort spam bot. Your B2B business is completely cooked if that is how you treat people. The enterprise cloud gravy train is right there and Google is standing on the tracks begging to get hit by the train.
If you've been around a while you know that at any business critical scale at all you establish a relationship with your cloud provider and get an account manager. When you do this, you have a number to call.
A billion dollar startup not doing this is a keen lesson for the CTO.
Yes, Google likely screwed up here, but being unprepared for account problems, having no established relationship with your provider is a critical mistake.
The article goes on to talk about Hetzner as an example: their pricing is great for individuals but they literally don't even offer account management relationships - even at scale they actively refuse them. There are equivalent stories of account terminations with Hetzner, which is also a key point: this isn't just a big business problem, at all.
Our team at work uses all three major public clouds and our Google Account Team is by far the worst of the three. Nothing like having to explain the same problem I’m having from the start again on every monthly cadence call because they don’t note anything or try to help resolve it
As a former Googler this doesn’t surprise me at all. Dealing with customers directly is literally seen as failure. Everything at Google is about scale, and automation. If you have to do something manually - anything - then you’re doing it wrong. It’s deep in the culture, or was when I was there. Maybe they’ve gotten better, but no signs this is true.
I had the same experience with AWS. They even assigned principal someone to our case and guy sent us old blogposts and did nothing. I took a month to find "hidden" checkbox. On GCP I always escalate to pass indian support as they are super incompetent.
(I'll take the charitable read and assume you meant an implied poorly trained, low cost Indian support as the descriptor)
Google does and always has treated support as a cost center, rather than a feature or profit center.
This worked for them in the early days and for specific products like Search -- design it well enough and you don't need support*.
Unfortunately, this fails for most other products (and especially all enterprise products) because the magnitude of impacts to a single customer (business instead of user) are so much greater.
F.ex. every conversation I was on with a GCP product team had a weird "Oh, you don't know how anyone who pays for GCP uses it, because you get it for free internally?" flavor.
* Leaving aside the "there's an entire SEO industry to 'support' Search" thing
Microsoft. It more than compensated for Azure not being the best product . They are incredibly more responsive, you have multiple points of contacts and escalation chain of actual humans you can meet
they will even come to your customer call or connect with their rep already working with the mutual customer and so on.
AWS has the best tech and but not as good as Microsoft service wise, they certainly improved a lot last few years and it shows but because they don’t have any enterprise apps like MS their footprint is more limited.
Google keeps talking about GCP being important but doesn’t feel anything has changed on ground
My company also used all 3 (at a very large scale / spends). MS was nice, but useless / incompetent technically. Anything non-trivial took forever to get a straight answer or resolve. We rarely got to speak directly to anyone with real expertise.
AWS, we could speak directly to Sr engineers on the relevant team. Full transparency, highly responsive. They were clearly trying to understand our issues and suggest change for both us and themselves.
Google was mostly useless. There was one team I got to talk directly to, who were great. But that was the exception.
My experience with AWS hasn't been good when we had major problems in redshift becoming unresponsive. Since it was an intermitent issue and not a full blown blackout they just shrugged and we kept having problems for months.
I can confirm. Redshift support is mediocre even for a F100 firm with TAM support if the workload is large and complex and you have some needle in the haystack causing problems like you allude to.
Practically speaking keeping an eye on locks and transactions is a good idea, as is watching out for your statistics on key core columns going bad when they shouldn't. (analyze and vacuum sometimes don't actually do anything when you need them to...)
> You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" – lawnmower doesn’t give a shit about you, lawnmower can’t hate you. Don’t anthropomorphize the lawnmower. Don’t fall into that trap about Oracle. -Bryan Cantrill
A few years back someone at work stuck their hand in the lawnmower. I've seen this happen a few times, but that time it ended with Oracle fining us for every VirtualBox install and the company sent The IT Spanish Inquisition around to make sure we all deleted it off our computers. Fun times.
You honestly just have to treat any Oracle product as malware, and proactively scan for it / block it from being installed on employee laptops in the first place.
The article is incorrect and misleading. Railway did have an account manager and they did call them and they did pick up the phone and work with them to restore service.
An account manager overseeing such a major client should’ve never let this happen. If they don’t, why the hell are they the account manager? What are they even doing to earn their keep? This was such a preventable situation.
I once worked at a company that had their domain lapse because of an internal error at the company that was paid for the domain. There was no alert, there was no attempt to rectify the situation, one day we woke up and we simply did not have control of our website for a full week. There was nothing wrong with our payment method, there was no reason the payment shouldn’t have happened, it was completely their fault. They found out because we called them in a panic. This was a major company. We left them a week later and our CEO talks about it constantly as a horror story to other companies and there is no way our situation was unique.
It’s not just about the value of the contract. This whole situation has been in the news for days now. It’s terrible PR and I guarantee you it is costing them business in the long run. All I have seen for days is people talking about how poor Google’s support is, and I’m not even somebody who makes those decisions.
I get it, “Google is too big to fail.” But eventually, that stops being true
Typically a strong account team builds processes with other teams (compliance, engineering, etc) that enshrines and insulates important accounts from accidents like this.
In this case, I'd expect major accounts (and maybe Railway isn't above this level?) to be in a protected tier that is immune from automated suspensions like this.
If suspicious traffic occurs that _would_ trigger a suspension like that, the account team would be paged. Because this may mean your important account was compromised, shipped a bug, has been hit by something and you should immediately start working _with_ them to figure it out.
Fairly basic for a company with any customer management motion at all.
Then they’re not doing their job OR Google has bad systems in place that make it so their account managers aren’t equipped to do their job. Neither is a good look!
Railway had dedicated reps and everything. Even the number of the head. One of their railway company leads said so on X or their Discord during the outage.
They paid for everything you can pay for to get Google on the horn and ready because they are not a tiny company. Yet it was still terrible.
Spot on. This is the wrong takeaway. The account manager can't do anything to prevent the problem. They can only escalate, and that's what they did.
The right takeaway is you should not use Google Cloud because Google itself doesn't use Google Cloud for anything critical. I don't know how many times I've said this, but it clearly hasn't sunk in. A Googler once responded, "Well ackshually, Google Domains is built on Google Cloud." Google Domains isn't mission critical and has since been sold to Squarespace.
in my experience hetzner does have that. my VP of infra had regular contact with them, was quite important actually because we had scaling needs they couldn't deliver on and we had timelines. hetzner still has issues getting enough intel servers on it seems
subtree is better for this case, you want to encourage actual reading before running. reading won't catch everything but it catches a lot, and the burden isn't as high as people always complain about before they try it.
nope, doesn't help. signatures and removal of script points have zero net effect on the value of the target that the ecosystem has, or how easy/hard it is to write a worm. the package code gets run, this is statistically true, and the exploited developers/environments will sign packages, this is also statistically true.
install scripts are a distraction, just like package signatures are a distraction. adding/removing either feature has no significant impact on the wormability of this package ecosystem. installed npm code is run, with nearly zero exceptions.
The installed code may be run in different settings, under a different user, with different privileges. Say, it may not run in CI/CD at all, or run only with the test user's privileges.
Postinstall scripts run at install time, with installer's privileges.
Trivial relative to which perspective? The distinction matters enough to care. Just because your father might give away their phone pin over the phone doesn't mean we should allow this granting remote access to his phone.
Trivial in the sense that in 99.9% of situations, "npm install" is immediately followed by "npm run", "npm test", or some form of execution. Any execution that imports a dependency is enough for a transitive dependency to execute its malicious payload immediately.
Post-install scripts have a slight edge over executing malicious code on import, i.e. they work 99.95% of the time instead of 99.9% of the time, but removing these scripts wouldn't materially change the situation we're in. You're locking the back door but leaving the front door and all of the windows wide open.
I'm going to suggest that we might be worse off in the short-medium term if post-install scripts are removed because everyone who thought that disabling post-install scripts was a "good enough" standalone security strategy will get caught with their pants down as attackers modify their payloads.
> Post-install scripts have a slight edge over executing malicious code on import, i.e. they work 99.95% of the time instead of 99.9% of the time
The "instead of" depends very much on the exploit and where it's wedged in the code. I doubt it's anywhere near 99%. Plus, getting the exploit to execute on the developer's machine is difficult to manage even in the best cases.
> because everyone who thought that disabling post-install scripts was a "good enough" standalone security strategy will get caught with their pants down as attackers modify their payloads.
Saying "well there are stupid people in the world" seems like a pretty bad justification to leave a hole open.
> The "instead of" depends very much on the exploit and where it's wedged in the code. I doubt it's anywhere near 99%. Plus, getting the exploit to execute on the developer's machine is difficult to manage even in the best cases.
We don't need to guess, it's going to be wedged in index.js, probably on line 1.
Are you aware that all transitive dependencies are executed immediately? You depend on PackageA which imports PackageB, which imports PackageC, which imports a trojanized PackageD. As soon as PackageD is imported, it executes its payload and infects your machine.
All of this happens in a blink of an eye, as soon as you run anything that kicks off an import chain containing a trojanized dependency.
Try it for yourself. This will simulate a malicious transitive dependency: koa > cookies > keygrip > tsscmp. You don't need to do anything except import koa.
mkdir demo && cd demo
npm install --save koa@3.2.0
echo 'import "koa";' > demo.mjs
echo 'console.log("\n\n--- pwned by a transitive dependency ---")' >> node_modules/tsscmp/lib/index.js
node demo.mjs
> Saying "well there are stupid people in the world" seems like a pretty bad justification to leave a hole open
Then you're calling much of the HN audience stupid. I've had this argument on here several times - and this is the top percentile of people who try to do something at all.
The justification for leaving this hole open is that it's a waste of time, resources, and mindshare patching a hole when there's a comparable and unpatchable hole right next to it. Advocate for things that actually work, like sandboxes.
> There's a huge difference, because postinstall scripts are almost guaranteed to run in your CI pipeline. Compromised code probably won't (maybe it will if your test cases test a compromised package). Different attack profile. Worse in some ways (your CI likely has NPM push tokens, which is how this single-package worm become a multi-package self-replicating worm) (your CI pipeline also likely has some level of privileged access to your cloud environment; deployed services are more likely to be highly scoped). But, better in some ways.
> Compromised code probably won't (maybe it will if your test cases test a compromised package).
Code runs automatically on import, you don't have to call dependency.infectMePlease()
Your code imports depA which imports depB which imports depC which imports depD which has been compromised, and boom, malicious code runs before you've even finished resolving the imports.
> your CI likely has NPM push tokens, which is how this single-package worm become a multi-package self-replicating worm
I've never once seen or worked with a CI pipeline that ran "npm install" that would be any safer if post-install scripts didn't exist. They all run "npm run test" or similar.
Zircon is still under development with recent RFC's extending the memory synchronization and attribution model for processes.
There was also more extension added to one of the key disk formats in March which has an eye to more flexible long term evolution and adaptation to particular device form factors.
The publicly available evidence does nothing to support your claims, entirely the opposite.
I used to work on Fuchsia, I have not for many years now and have no idea what their current roadmap looks like, but I do know where to actually look up what's been done recently, which is all public and you could do as well.
Anyway I have no idea if this has any fuchsia code in it.
Okay? Is this impressive? Do you think it shows something? Bizarrely whenever people point out how much of a flop Fuchsia is (relative to the hype a decade ago), there is always someone like you citing commit count. Weird.
The vast majority of the commits are tiny commits to change a version number or rename a test. Or to pass some lint tests. I know tiny two man shops that have much more substantial commits each day.
I didn't say it was dead, though did I? Not quite sure what kicked off your bizarre defensive, asshole-ish screed. I specifically said that it most likely will be a tiny runtime for VM processes in Android.
But Fuchsia, announced A DECADE AGO, remains utterly irrelevant, aside from a couple of poorly received, dogshit Nest devices. And we know that Google massively downsized the team and basically moved on, and from people I've talked to it is now basically a make work project.
Yeah, the chances that Fuchsia powers this device is 0.0000%. I hugely doubt it even appears on the device at all.
So the dream, as constantly restated on here, is pretty clearly absolutely dead.
>Nothing asshole-ish about the grandparent comment
Irony. They wrongly did to me what I could have as easily done to you (anyone speculating that this runs Fuchsia is living in a cave and is probably leaving worthless comments to pretend they're participating), but I'm not obnoxious so I didn't and actually engaged in good faith. But having you stomping in and calling me an asshole in this case, then telling me to "tone it down"...Good god, hilarious stuff.
there are no good reasons we don't do this in the standards themselves, C, C++, and POSIX should all be working on editions that add safer APIs and mark unsafe APIs as deprecated, to start a long term migration. we know how to do this, we've had a lot of success with this. there are real engineering concerns, sure, but they're not reasons to not do it. compilers and library chains can retain support for less safe variants for plenty of time.
The reason this wasn't done by the standards committees is that they spent decades refusing to admit there was even a problem they could help fix. And if there was a problem, it was easily avoided by just writing better code. And if writing better code wasn't enough, well it was certainly too expensive to provide as a debug option. And if it wasn't too expensive to provide as a debug option, the implementors should really lead the way first. And on and on.
The C committee at least seems to get it now. The C++ committee still doesn't, led in large part by Bjarne.
This is a misrepresentation based on a misunderstanding on how standardization works. The C standard committee has long recognized the need for better safety and carefully made it possible so that C could be implemented safely. But the process is that vendors implement something and then come together during standardization so that it is compatible, not that the standardization is the government that prescribes top-down what everybody has to do. Vendors did not bother to provide safer C implementations and safety features (such as bounds checking) did not get much attention from users in the past. So I am happy to see that there is now more interest in safety, because as soon as there solutions we can start putting them into the standard.
(We can do some stuff before this, but this is always a bit of a fight with the vendors, because they do not like it at all if we tell them what to do, especially clang folks)
Stop mixing C and C++, tons of people on Unix still hate C++ (Motif a bit less) for being un-Unixy and megacomplex, even more today. Die had Unix and C people created Plan9 and now Go, which is maybe the other succesor to C before Inferno and Limbo, where programming it's more simpler than the whole C and POSIX clusterfux (even Plan9 and 9front itself can be called a "Unix 2.0").
C++ is something else. Heck, it's often far more bound to a Windows domain (and for a while Be/Haiku) than Unix itself by a huge stretch.
It is probably worth noting that C++, like C/Unix, originated at AT&T Bell Labs and was originally referred to as "C with classes." Classes were implemented using a preprocessor.
Unix creators called Unix "dead and rotten" because the eulogy was done by Perl, and Plan9/9front and Inferno obliterated it. Ditto with C+POSIX against Plan9's C (and 9front) and Inferno vs Limbo, the grandparents of Golang, which is seen from Pike and so as the tool set C++ should have been.
Golang it's like Windows NT. C++ it's like Windows ME, it might have their cases on RT performance and multimedia because of having far less layers than NT, (and much better on single core), but it crumbles down fast and it was really easy to shoot yourself in the foot. Windows 2000 and XP killed it for the good.
Some day Golang would be performant enough (even with CSP) with multiple cores so all the 'performance' advanteges -suppossedly C++ brings- aren't needed at all.
Even C# can be as good as C++ today in tons of cases (AOT and emulators like Ryuyinx are not a bluff), even SBCL for Common Lisp too if you finetune the compiling options.
To clarify, I do agree with you that C and C++ have been two distinct languages for a very long time. And C++ doesn’t have much in common with POSIX.
What I disagree with is the idea that C++ was developed completely independently of C (and Unix) - it originated at Bell Labs and was initially just an extension of C with classes. If you looked at the document I linked to, you would see that Bjarne Stroustrup thanks Dennis Ritchie in it for being a source of good ideas and useful problems. I don’t think I need to explain who Dennis Ritchie was for C and Unix.
Yup, and its not just the standards committees. Look at TR 24731 as an example, an absolute no-brainer for security adding (shock, horror!) bounds checking to long-standing trouble-prone APIs that's been around for 20 years, and the response from most compiler writers/library authors has been "lalalalala I'm not listening I'm not listening". Even then it only got as far as it did due to relentless pressure from Microsoft, anyone else and it'd have been rejected outright.
Having said that, some of it may be due to "it's from Microsoft, we can't ever use it". I'm actually surprised not to have seen any anti-MS diatribes in the discussion so far.
Anything needs to be demonstrated and used in practice before being included in the standard. The standard is only meant to codify existing practices, not introduce new ideas.
It's up to compiler developers to ship first, standardize later.
That produces a bit of a chicken and egg probablem for a stdlib overhaul. Compilers and libc implementations don't have a strong reason to implement safer APIs, because if it is non-standard then projects that want to be portable won't use it , but it won't get standardized unless they do add safer APIs.
So the best hope is probably for a third party library that has safet APIs to get popular enough that it becomes a de facto standard.
I think the real failing is that new language features then must be prototyped by people who have a background in compilers. That's a very small subset of the overall C community.
I don't have any clue how to patch clang's front end. I'm not a language or compiler person. I just want to make stuff better. There needs to be a playground for people like me, and hopefully lib0xc can be that playground.
By adding to the language itself, you mostly make stuff worse. The major reason why C is useful is its quite stable syntax and semantics. Language is typically not the area where you want to add code. It's much better (and much easier) to invent function APIs. See how they shake out, if they're good you might get some adoption.
A vast number of C++ programs import C and POSIX headers directly, so the language level distinction you wish to make isn’t all that relevant to the subject matter.
Lawyers I have spoken to have stated strongly that they believe collective works doctrine will provide strong protections for most mature and sizable software. I see no mention of these considerations here.
reply