Have you ever seen “Jackie Brown”? There's a scene where a criminal shows his colleague the dead body he has in the trunk of his car and then proceeds to explain, in perfectly rational terms, why the guy in the trunk had to be killed.

I'll never forget that lesson. Rationality can be used to justify anything and everything. Invoking rationality as the basis for one's decisions or behavior doesn't mean anything by itself.

This. Perfect rationality, working from insane premises, has to reach an insane conclusion.

Rationality without morality is pretty dangerous, thats for sure, its trivial for a skilled smart manipulator to lead folks into pretty dark places while still feeling superior.

And then there's a non-trivial amount of properly crazy folks who think they are absolutely fine and behaving very rationally, their own version of reality and universe in their head is impenetrable. Psychiatrists have stories to tell, but either can't share or folks prefer not listening such things (same goes for most doctors, ambulance drivers etc.)

I was immediately reminded of Rope by Hitchcock, although the nature of that rationalization is somewhat different.

I can never quite decide if Jimmy Stuart's outrageous hamming at the end of that movie makes it or ruins it ("Did you think you were God, Brandon?")

"Who's that?"

"That's Beaumont."

"Who's Beaumont?"

"An employee I had to let go."

The rest of the perfectly rational explanation will be left out, as it contains strong language not suitable for a website such as this.

> completely normal community apart from the rationalism

I don't think widespread polyamory, group homes, radically different social norms around language use and conversation, or reorganizing your entire life around mitigating AI risk are "completely normal."

I spent years thinking its crazy to get your morals from a random book, but then I read this stuff and I understand the pitfalls of also having it all wide open. This idea we all define our morality works extremely poorly. I used to think it was a product of the old times of almost non existent education and honestly dumb "ancient world" people (lack of nutrition, lead poisoning...).

But I don't know, clearly smart and educated people can believe extremely dumb things. The smarter they are the weirder the setups they get themselves into. One almost comes to appreciate the existence of some religions with a history of many centuries to "chill out" and see what works, instead of any of the new age cults.

I’m not religious and open minded about new stuff but ultimately decided the ancient Stoics system of morals works perfectly for me- and is more time tested than even Christianity. I think the rationalists decision to use consequentialist morality in regular life is a huge mistake, and impossible to get right in practice. Humans cannot predict the future outcomes of our actions very well, but we can easily remember and follow a simple set of core values.

If you look at most great evil perpetrated by mankind, it's almost always someone trying to "do good". Sure there are serial killers or whatever, crimes of passion, but they just kill one or a few. The people who have killed millions, driven whole regions into famine and death, destroyed lives at an industrial scale.... they were 'doing good'.

Utilitarian consequentialism is the most evil of all philosophies. If someone is merely sadistic their sadism will be sated after some finite amount of abuse. If someone is greedy then their harm will at least be limited to what they can profit from and there is no profit in ruling over a cinder. But the harm possible by someone convinced that their actions are good in some abstract sense is without any bound.

The irony in the lesswrongers fixation on consequentialist morality is that their great fear of machine superintendence is derived from an expectation that its evil will arise from the same sorts of reasoning they engage in themselves. They simply fail to pause and ask "Are we the baddies?"

I usually leave it out of my complaints because they're so ineffectual that it's not a real threat. But the LW solution to "unaligned" machine intelligence is to create an AI god in their own image first, so that it can enslave humanity and all other lifeforms within its lightcone for their own best interest, and suppresses the creation of any competing "unaligned" God. So they're literally out to create the very thing they fear, but with the hubris to imagine that if it was theirs it would be "good". The ultimate fantasy of both the utilitarian consequentialist and most authoritarian mass murderers. If their doomsday fears come true my bet is that it will be at their own hand or that of their followers.

It's also the ultimate horseshoe for militant internet atheists-- "there is no God; and this is OK" becomes "there is no God; and we're gonna create one in our own image".

Interesting observation that the rationalists doomsday AI scenarios all are essentially just obvious horrific consequences of utilitarian consequentialism backed by unlimited power- yet they insist on trying to live by acting this way themselves.

> reorganizing your entire life around mitigating AI risk

I'll give the rationalists that: they were out talking about AI ethics and risk way before AI risk was cool. Total hipsters.

And crypto, as early as 2009 with people like Hal Finney posting about Bitcoin on LessWrong.

Philip K. Dick would eventually have written this as a darkly comedic novel. A rationalist death cult plans to unleash a mind-altering drug, meanwhile the protagonist discovers a possibly benevolent AI has just come online in a Berkeley campus basement, all set in an alternative California where Nixon is still president and we have mining colonies on Mars. (The drug disables the empathy center of your brain, but also reveals the true nature of Nixon, who turns out to be a simulacrum, as he was actually assassinated in 1963.)

Scandals with sex pests, minors kept in group houses, partnerships with ponzi scammers, orgies, grit sham charities that pocket the money, etc. isn't normal. It happens elsewhere, sure, but it's not normal.

It's gonzo.

But it's also not new, there are plenty of works of fiction with groups like this in them, perhaps inspired by actual experience with 1960s new wave and prior counterculture groups.

'Once upon a time in San Francisco', perhaps?

> You'd think a group of people devoted to being rational would be a group of monk-like beings, cerebral, disconnected from the concerns of the flesh.

Human beings are fundamentally irrational, and it's a really great way to deceive oneself to believe otherwise.

If you can convince yourself that you're unbiased, objective and "rational", then whatever you happen to believe must be correct! You can rationalize practically anything, so it's no surprise rationalists start getting weird when left to their own devices.

“To do evil a human being must first of all believe that what he's doing is good... it is in the nature of the human being to seek a justification for his actions.” -Aleksandr Solzhenitsyn

“Man is not a rational animal; he is a rationalizing animal.” -Robert A. Heinlein

You know that what you are describing is the central observation of rationalism right ? "Humans rationalize, so you better be skeptical of your own beliefs".

You presenting it as a blind spot of rationalists, but that’s precisely the blind spot that rationalism warn about !

Yes, but step 2 is being able to overcome that slight problem with mind kung-fu. So the result is the same.

Oh come on, the vast majority of LW meetups are about as racy as the IRL comment section of HN.

Even the completely anonymous 18+ only rationalist telegram groups are only just as racy as the normal non-yiff furry groups.

The Berkeley and New York scenes are special in that they lead and guide the community on a whole mainly by example.

I think they're basically secular humanists who decided they wanted a more muscular guiding ideology.

And wound up building a heretical pseudo-Christianity where the messiah is a superintelligence.