Some rough notes on The Precipice: Existential Risk and The Future of Humanity

Toby Ord ( https://en.wikipedia.org/wiki/Toby_Ord ) sent me a copy of his book:

 The Precipice: Existential Risk and The Future of Humanity 

 

Thank you Toby! Some notes and highlights:

Summary. Page 31: "Given everything I know, I put the existential risk this century at around one in six: Russian roulette."

In general, things are going very well in the world. Decrease in extreme poverty, child mortality, increase in literacy, and life expectancy.  Figure 1.3

Page 21: "Earth is the cradle of humanity, but mankind cannot stay in the cradle forever...": Konstantin Tsiolkovsy who gave us  (Delta V) 

The author Toby Ord is optimistic and seems to take it as a given that we will inhabit the stars, "our species learns quickly" but "Houston we have a problem" development in rocketry had been pitiful and even gone backward since the Apollo mission as SpaceShipTwo designer Burt Rutan explained in 2006: https://www.ted.com/talks/burt_rutan_the_real_future_of_space_exploration 

Page 29: "Technology has proved itself immensely valuable in improving the human condition...Without it, we would be doomed by the accumulated risk of natural disasters such as asteroid impacts."

Technological progress without an equivalent progress in wisdom and human institutions can doom us. 

Our neglect of existential risk: (When I was a reserve officer in the Dutch Field Artillery, we prepared for ABC risk: Atomic, Biological, Chemical. Nowadays Warren Buffett likes to joke that CNBC is the biggest risk: Cyber, Nuclear, Biological, Chemical.)

Page 57 Neglect of risks: "The international body responsible for the continued prohibition of bioweapons (the Biological Weapons Convention) has an annual budget of just $1,4 million - less than the average McDonald's restaurant." 

The risks: https://theprecipice.com/faq#how-high the biggest is unaligned Artificial Intelligence. There is a risk I won't mention here because as Toby Ord writes on page 138: "Information hazards are especially important for ...risk, due to its high ratio of misuse risk to accident risk... While exploring society's current vulnerabilities or the dangers from recent techniques, the ... community also emits dangerous information (something I've had to be acutely aware of while writing this section)." emphasis added. 

Dystopia + Fear the memesphere. Page 156 "The historical record is rife with examples of seriously defective ideologies and moral views that gripped large parts of the world."

page 158 "A key problem is that the truth of an idea is only one contributor to its memetic potential --its ability to spread and to stick. But the more that rigorous and rational debate is encouraged, the more truth contributes to memetic success. So encouraging a culture of such debate may be one way we can now help avoid this fate."   Something Elon Musk pointed out during the 2020 Coronavirus toilet paper panic.

The Risk Landscape through time

 Page 146 As part of the Great Conversation Ord, comments on something Steven Pinker wrote in Enlightment Now: Pinker: "Our ancestors were powerless to stop these lethal menaces, so in that sense, technology has not made this a uniquely dangerous era in the history of our species but a uniquely safe one."

Ord responds: "While Pinker is is quite correct that we face many natural threats and that technology has lowered their risk, we can't conclude that makes our time uniquely safe." 

That first became clear with the nuclear explosions during the Second World War and especially with the Castle Bravo hydrogen bomb, which was much more powerful than expected. 

We might kill ourselves through technology: the risk not named here plus unaligned AI, unforeseen risk, etc, that danger of killing ourselves in the next century is orders of magnitude higher  ( 1 in 6) than a natural demise for humankind (1 in 10 000) according to Ord's guesstimates.

Page 194 SECURITY AMONG THE STARS: Lowering risk by spreading out human consciousness among the stars. (Paraphrasing Elon Musk: We must make consciousness multi-planetary to ensure the candle continues to burn.) 

"The idea is based on an important statistical truth. If there were a growing number of locations that all need to be destroyed for humanity to fail, and if the chance of each (planet) suffering a catastrophe is independent...unfortunately, this argument only applies to risks that are statistically independent. Many risks such as disease, war, tyranny and permanently locking in bad values are correlated across different planets.... A few risks such as unaligned AGI and vacuum collapse are almost completely correlated... Space settlement is thus helpful..but it is by no means sufficient. Becoming a mult-planetary species is an inspirational project--- and may be a necessary step in achieving humanity's potential."

Appendix G: By leaving the cradle, humanity becomes a Type II (K1) civilization on the Kardeshev Scale. (Alternatively if you believe like Elon Musk we are living in a sort of computer game, then Mars is Level 2.) 

There is a Planet B, but we need to get there. 

The LONG REFLECTION: because technology is going so fast, we need to implement measures to quickly address short-term existential risk. We would then have moved back from The Precipice and then take a break to reflect for a while. How long? Ord: "My best guess is that it would be worth spending centuries (or more) before embarking on major irreversible changes to our future...The ultimate aim of the Long Reflection would be to achieve a final answer to the question of which is the best future for humanity." This reminds me of the search to find an answer to the Ultimate Question of Life, The Universe, and Everything. The answer is 42. https://www.google.com/search?q=42 See The Hitchhiker's Guide to the Galaxy (not mentioned in the book). 

The short reflection called for in 2023 after ChatGPT intro: https://futureoflife.org/open-letter/pause-giant-ai-experiments/

Page 221 The threat of running out of CO2... "...scientists expect the brightening of the Sun to also slow the Earth's plate tectonics, dampening volcanic activity. Life as we know it requires such activity; volcanoes lift essential carbon dioxide into our atmosphere...Without carbon dioxide from volcanoes, scientists estimate that in about 800 million years photosynthesis will become impossible in 97 of plants, causing an extreme mass extinction." 500 million years later everything is dead. But maybe not?..

"For all of the myriad species that inhabit our Earth, only we could save the biosphere from the effects of the brightening Sun. Even if humanity is very small in your picture of the world, if most of the intrinsic value of the world lies in the rest of the eco-system, humanity's instrumental value may yet be profound. For if we last long enough, we will have a chance to literally save the world."

What isn't in the book? 

The book calls for reflection and seeking wisdom. That is the core business of the world's religions, but little mention is made of their possible role except in a negative light: a fundamental religion taking over the world could result in a dystopia. 

The role of love is touched on and the idea that spending time with your child could be the most valuable thing in the world. Contrary to contemporary thinking, rationality, enlightenment, and science can go hand in hand with love and faith.  

"Faith (love) and reason are like two wings on which the human spirit rises to the contemplation of truth."  Fides et Ratio: http://www.vatican.va/content/john-paul-ii/en/encyclicals/documents/hf_jp-ii_enc_14091998_fides-et-ratio.html 

---------

Personally going forward: The book has strengthened my resolve to set up this "project" (in practice a savings account): https://sinaas.blogspot.com/2020/05/missionary-on-mars-project-rough-notes.html

I like The Long Now foundation and Longbets.org for example: https://longbets.org/768/ 

Thanks again Toby Ord for the book and https://www.effectivegiving.nl/ for linking us.