(2024-06-07) Zvim Quotes From Leopold Aschenbrenners Situational Awareness Paper

Zvi Mowshowitz: Quotes from Leopold Aschenbrenner's Situational Awareness Paper. This post is different. Usually I offer commentary and analysis. I share what others think, then respond. This is the second time I am importantly not doing that. The work speaks for itself. It offers a different perspective, a window and a worldview. It is self-consistent. This is what a highly intelligent, highly knowledgeable person actually believes after much thought. (2024-06-01) Leopold Aschenbrenner's AGI Situational Awareness Paper

this is only quotes and graphs from the paper, selected to tell the central story while cutting length by ~80%, so others can more easily absorb it

Then there will be one or more other posts, where I do respond.

Introduction

Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness.

Section 1: From GPT-4 to AGI: Counting the OOMs

AGI by 2027 is strikingly plausible

it is strikingly plausible that by 2027, models will be able to do the work of an AI researcher/engineer

Section 2: From AGI to Superintelligence: The Intelligence Explosion

AI progress won’t stop at human-level. Hundreds of millions of AGIs could automate AI research, compressing a decade of algorithmic progress (5+ OOMs) into 1 year. We would rapidly go from human-level to vastly superhuman AI systems. The power—and the peril—of superintelligence would be dramatic.

Section 3a: Racing to the Trillion-Dollar Cluster

Historical precedents

1T/year of total annual AI investment by 2027 seems outrageous. But it’s worth taking a look at other historical reference classes:

In their peak years of funding, the Manhattan and Apollo programs reached 0.4% of GDP, or ~$100 billion annually today (surprisingly small!). At $1T/year, AI investment would be about 3% of GDP.

Between 1996–2001, telecoms invested nearly $1 trillion in today’s dollars in building out internet infrastructure. • From 1841 to 1850, private British railway investments totaled a cumulative ~40% of British GDP at the time. A similar fraction of US GDP would be equivalent to ~$11T over a decade.

Many trillions are being spent on the green transition.

Section 3b: Lock Down the Labs: Security for AGI

Section 3c: Superalignment

Section 3d: The Free World Must Prevail

Section 4: The Project