The BBC report that an open letter has been signed by more than 1,300 experts saying

AI is not an existential threat to humanity; it will be a transformative force for good if we get critical decisions about its development and use right.

Those signing the open letter say AI technology is enabling new ways to detect serious illness. It is enabling classifiers which detect cardiac disease or diabetes during eye tests. Others point to agricultural uses such as robotic pollination of plants and weed eradication using lasers.

A former council member of the Open Source Society, employer and friend of mine, Igor Portugal, is writing a book about a positive AI future called "The Planet Fortht: Enslaving Freedom", which according to an online description presents

Utopian world where AI and other advanced technologies safeguard individuals, uphold justice, and deter crime. Nobody is above the law, and justice is administered to anyone who causes harm to another human being.

So perhaps let me begin by steel manning this argument. It is a proposition that the AI will not destroy us, but rather will serve us and be obedient. Consequently society will benefit greatly on many levels.

In this vision the outcome is a society where the needs and desires of people are met. It is a future of abundance, where people will no longer be constrained by the need to work, where we will have access to entertainment, amazing food, health care, and education.

It would be a golden age, where people would have no need to work. Virtual reality could deliver any sensory experience you desire in complete safety.

Enabling this future would be obedient intelligent machines doing all the work for us. From growing food, to providing health care, running power plants, designing and manufacturing goods, even educating children. All would be done by artificial intelligence and robots that are both more intelligent than humans while also being perfectly obedient.

Perhaps this story sounds familiar. Agent Smith describes it in The Matrix.

Did you know that the first Matrix was designed to be a perfect human world. None suffered, where everyone would be happy. It was a disaster. No one would accept the program. Entire crops were lost.

This theme of the best outcome also features in my 2005 article "Rise of the Immortals".

Once the revolution begins there will be very real benefits to those companies leading the revolution in production of thinking machines.
What is more, this may actually benefit us. Will the effect be dramatically more efficient and effective products? Humans may actually end up without much of a role at all, as computers and robotics will be able to perform all the jobs humans currently perform.

Let us assume for the sake of argument that we do in fact ensure excellent alignment, that machines serve humanity, and that it brings about unparalleled human physical well being. Is this the utopia we dream of?

Agent Smith continues

The Matrix was redesigned to this, the peak of your civilization. I say your civilization because as soon as we started thinking for you it became our civilization.

The Matrix was very much like the perfect human utopia. All your needs taken care of. Your nutrition, your health, and you could have any experience you wish, without any danger to your physical body.

It is also difficult to avoid comparison with the Christian Heaven, with perfect happiness, perfect health, forever, with one loving 'father', or if you prefer 'owner'. Because that vision too denies personal free will and agency once there.

In this vision of utopia we are relieved of all responsibility, have no need for intellectual or physical labour, rather live a life devoid of responsibility, competition or status. In such a utopia what do we strive for?

Our personal struggles, our determination, our tragedies, our victories all make us who we are. As Star Treks Captain Kirk once said

Pain and guilt can't be taken away with a magic wand. They are the things we carry with us, the things that make us who we are. If we lose them we lose ourselves. I don't want my pain taken away, I need my pain!

Perhaps our purpose is less about the pain and more about how we respond to it. How we decide to be good people in spite of it. Pain, suffering, personal trauma, they give us the opportunity to display courage, fortitude, compassion. Living in a utopia relieves us of any need to fight against injustice.

There is little difference between technological, religious or fictional utopias. They all have common themes. We become the playthings of other entities. We lose our agency. We will be deprived of the opportunity to exhibit the good. We will no longer be sovereign over our own future.

And so the perfect utopia is also a living hell, one where there is no real control over your life, where we live only for naked self gratification.

In such a future we would be little more than pets kept by machines. They would tend to our needs, feed us, entertain us, care for us in a clinical and detached way.

In such a situation I think the human race would have a limited future. What human would engage in actual procreation when you could have unlimited sexual pleasure at a whim without any of the consequences? What would the point be of having kids if they had no real future other than to exist?

What role would education have in a world without work? Would humans be educated at all, or would they simply be entertained?

It is easy to see how humans would quickly decline in population while losing any political or economic power they may have once had. How long before we would be all but extinct?