Recommended: LessWrong

LessWrong is an online community dedicated to the art of rationality and cognitive biases. It has morphed into a community from what was originally the posts of Eliezer Yudkowsky, who wrote several hundred posts on the above, as well as AI, transhumanism, ethics, Bayseanism, probability, psychology and more. If you have any interest in any of these at all, you should check the site out even if you strongly disagree with the ideas in Yudkowsky’s original post sequences.

EY on generalising from fictional evidence:

When I try to introduce the subject of advanced AI, what’s the first thing I hear, more than half the time?
“Oh, you mean like the Terminator movies / the Matrix / Asimov’s robots!”
And I reply, “Well, no, not exactly. I try to avoid the logical fallacy of generalizing from fictional evidence.”
Some people get it right away, and laugh. Others defend their use of the example, disagreeing that it’s a fallacy.
[...]
You say, “Suppose we build a very smart AI,” and they say, “But didn’t that lead to nuclear war in The Terminator?” As far as I can tell, it’s identical reasoning, down to the tone of voice, of someone who might say: “But didn’t that lead to nuclear war on Alpha Centauri?” or “Didn’t that lead to the fall of the Italian city-state of Piccolo in the fourteenth century?” The movie is not believed, but it is available. It is treated, not as a prophecy, but as an illustrative historical case. Will history repeat itself? Who knows?

EY on fake explanations and worshipping mystery:

Calling “elan vital” an explanation, even a fake explanation like phlogiston, is probably giving it too much credit. It functioned primarily as a curiosity-stopper. You said “Why?” and the answer was “Elan vital!”
When you say “Elan vital!”, it feels like you know why your hand moves. You have a little causal diagram in your head that says ["Elan vital!"] -> [hand moves]. But actually you know nothing you didn’t know before. You don’t know, say, whether your hand will generate heat or absorb heat, unless you have observed the fact already; if not, you won’t be able to predict it in advance. Your curiosity feels sated, but it hasn’t been fed. Since you can say “Why? Elan vital!” to any possible observation, it is equally good at explaining all outcomes, a disguised hypothesis of maximum entropy, etcetera.
But the greater lesson lies in the vitalists’ reverence for the elan vital, their eagerness to pronounce it a mystery beyond all science. Meeting the great dragon Unknown, the vitalists did not draw their swords to do battle, but bowed their necks in submission. They took pride in their ignorance, made biology into a sacred mystery, and thereby became loath to relinquish their ignorance when evidence came knocking.

EY on the use of framing to make the universe appear even more wonderful:

[S]uppose that instead of one eye, you possessed a magical second eye embedded in your forehead. And this second eye enabled you to see into the third dimension – so that you could somehow tell how far away things were – where an ordinary eye would see only a two-dimensional shadow of the true world. Only the possessors of this ability can accurately aim the legendary distance-weapons that kill at ranges far beyond a sword, or use to their fullest potential the shells of ultrafast machinery called “cars”.
“Binocular vision” would be too light a term for this ability. We’ll only appreciate it once it has a properly impressive name, like Mystic Eyes of Depth Perception.