Peter --

Interesting thoughts. I hadn't thought of offering a name for the AI -- Netty is a good one!

I completely agree with your thesis statement. It's tempting to believe we can assert control over whether or not a technological process comes to fruition. I don't see how it could be done. Even in a circumstance where an entire category of basic research is made illegal in a large country (which is unlikely to say the least), the most that such a move seems likely to do is slow the rate of change.

What I'd say though is that I'm not so sure the evolved intelligence will be even loosely analogous to human intelligence. I think the best formulation of that is in the article. Will it be able to ponder the night sky and feel small? Increasing calculation capacity and fomenting creativity are vastly different things. My guess is that the path forward is a symbiosis where humans provide the things they're evolved for like creativity and companionship. And AI provides the things it's evolved for like breadth of awareness and calculation ability.

Still, the scenario you raise is worth being aware of. Thanks for reading!

Will