Interesting thoughts, Ben! I won’t pretend to be an expert on this myself, but I agree with a lot of what you wrote, especially about the stereotypical, Terminator-esque conception of killer AIs. You also raise some really good questions about mental illness and how it would manifest itself in an AI. I have severe OCD, and I’m a big sci-fi nerd, so I’ve wrestled with that stuff myself. But your thoughts on the subject are much more coherent than mine.

That being said, I do want to play devil’s advocate a bit and maybe challenge this one part of your piece…

“Nor is anybody likely to do anything so stupid or insane as to build a machine with superhuman intelligence and explicitly malicious or destructive intentions. At least one hopes not.”

I’m not sure it’s that unlikely that someone wouldn’t try to do this. The desire, I think, is certainly there, at least on the part of zealots and extremists who’d rather burn everything down than accept defeat. If, say, Al Qaeda got their hands on such technology, I don’t think they’d hesitate to unleash it on the West even if there was a distinct chance that it could spell doom for the entire world.

I’m also guessing that plenty of governments would jump at the chance to weaponize AI technology. It would be a horrible idea, but they’d justify it by citing the inevitability of a rival government doing the same thing. They’d argue that if they didn’t do it first, some tyrannical regime would have, like North Korea or China.

That being said, I most definitely hope that you’re right and I’m wrong, though I don’t know if either one of us will live long enough to find out.

Anyways, this was a good read. Thanks for posting it! :)