On self-aware AI

Scott Alexander, Highlights From The Comments On Acemoglu And AI:

Non-self-aware computers can beat humans at Chess, Go, and Starcraft. They can write decent essays and paint good art. Whatever you’re expecting you “need self-awareness” in order to do, I bet non-self-aware computers can do it too. Computers are just going to get better and better at stuff, and at some point probably they’ll be as good as humans at various things, and if you ask them if they’re self-aware they’ll give some answer consistent with their programming, which for all I know is what we do too.