In the early days of existential risk, John von Neumann reflected on technology's creative and destructive power, today Mustafa Suleyman's important new book The Coming Wave explores the promise and peril of our latest creations.

In this article I explore the following questions:

1. AI sub-cultures range from pessimism to optimism, each bring insights but also pitfalls. How do we understand them to retain a balanced perspective?

2. We're moving along a shallow but steepening AI risk curve; given historical precedents why should we act now, and what are Suleyman's recommendations?

3. How can we use the science of complex systems to achieve AI transparency and distributed control, and work towards maximising human-machine potential?

In his book which covers AI and synthetic biology, Suleyman (founder of DeepMind and Inflection), cites a 1955 essay by John von Neumann entitled ‘Can we survive technology?’ Von Neumann’s intellect was legendary, he made major contributions to mathematics, quantum physics, economics, and statistics, and was instrumental in the creation of the atomic bomb and modern computing. He was a ‘man from the future’ and his essay not only includes an early warning on climate change but reflects on the destructive power of progress, and humankind’s expansion beyond the limits of our planet. He concludes hopefully, affirming our ‘congenital ability to come through, after varying amounts of trouble’.

Von Neumann was right about the varying amounts of trouble we have experienced in our history. Like war and pandemics, societal collapse has been a repeating chapter in the human story. But the modern narrative of ‘existential risk’ has evolved only recently with scientific progress. As we have become more rational, and our tools have become more powerful, we are increasingly aware of mechanisms that might bring about catastrophe. Von Neumann and his contemporaries ushered in the age of nuclear annihilation, and now Suleyman and the architects of AI warn of an inbound technological tsunami.

The AI viewpoints fragmenting global response

Like many areas of discourse, AI has clustered into a set of tribes, each influential, shaping media messaging, public opinion and government policy. These varying mental models develop into belief systems, and even when new data presents itself, tribe members can struggle to escape their anchors:

• The AI utopians have a propensity for determinism, believing that technology and its radical transformation of society and collective wealth, must not be obstructed. This won’t be a utopia for the subservient machines, which some believe can be engineered to stay under our control forever, although there is another group of more committed utopians who hope the machines replace us (termed ‘accelerationists’).

• The AI pessimists or ‘doomers’ are also somewhat deterministic, they step through logical arguments around the emergence of super intelligence and find many of the branches leading to catastrophic outcomes. This tribe is further splintering into those who see short term ethical risks as the main concern and existential risks as a distraction, and those who worry that the ‘probability of doom’ or p(doom) is too high to be ignored.

• The AI sceptics challenge the capability of AI, they often characterise the latest models as “advanced auto-complete”. They suggest that existential warnings are a conspiracy to boost AI hype, valuations, and incumbency, and that the bubble will burst messily.

• The AI opportunists see AI disruption as a chance for profit and to further their various agendas, from maintaining beneficial concentrations of power, to monopolising revenue streams and regulatory capture.

The truth is that the intelligence revolution, like the climate crisis, is ‘hyper-complex’. All perspectives must be considered, and we must ensure we don’t limit our points of observation.

Suleyman positions himself symmetrically between utopia and catastrophe, and thoughtfully embodies the dilemma of our age.

The sceptics are right in stating that today’s AI is unreliable, and relatively limited. But the pessimists elicit unease when they remind us that we got this far almost by accident. In the scheme of things very little has been invested by humanity in developing AI. We spent more globally on yogurt in 2022 than we did on AI research, and thus far the compute and data scaling laws are proving remarkably smooth.

Meanwhile utopians make a compelling case for AI empowerment delivering novel and accelerated solutions for some of our biggest challenges. The uplift in productivity from early AI tools is already measurable.

Ever present opportunists deploy any AI argument that will help shape the world for their own ends; be that authoritarians who see the potential for more control, or billionaires who can monopolise their chosen markets.

Suleyman comments that the sceptics, the opportunists, and the utopians often indulge in pessimism aversion, not wanting to engage with uncomfortable scenarios. This is reinforced by a lack of evidence for catastrophe… we have not all died in a fireball. Markets cannot price the cost of the end of humanity, so they ignore it. If we can’t predict something, does that really mean it’s entirely unlikely?

How do we balance these perspectives to see through the fog?

As time ticks rapidly by, the risks will inevitably increase. The lessons from history on this are clear; general-purpose technology has far-reaching implications which cannot be predicted. No tribe can argue that we have a good track record of ensuring new technology is safe, or that we are protected by our inability to predict new mechanisms for catastrophe.

As commercially and geo-politically charged AI race conditions develop, humans will be empowered to do more of what we do, both good and bad. There is no invisible hand that will protect us from excessive enablement, no limit on how bad actors might innovate with powerful new intelligent systems to automate, influence, and disrupt. We could see combinatory harms reach existential levels well before machines reach super intelligence. Nuclear science rapidly empowered us with the ability to destructively shape our world, but its steep and visible risk curve forced everyone to act. AI will do the same but more gradually, and we must not rely on being able to see the signs when reaching an existential level threshold.

But this risk curve doesn’t end there, this timeline extends beyond general purpose human empowerment. The likes of Yuval Noah Harari exhort that nuclear weapons cannot design new nuclear weapons. For the first time in human history, intelligent technology could begin to radically disempower us, leading to a whole new suite of mechanisms for catastrophe. Evolution has propelled us to the top if the pile, but in the disempowerment phase, Suleyman’s concept of ‘hyper-evolution’ could start to favour machine replacements. The risks of AI empowered humans unleashing disinformation campaigns, drone attacks, bioweapons, or Kurt Vonnegut's Ice-IX is suddenly extended by the potential for them to be enacted by an AI, intentionally or through goal misalignment. A vastly increased pool of cognitive activity sees human-based intelligence become a disempowered minority group. A vastly more diverse and varied distribution of ‘alien’ cognitive activity creates a greater probability space for harmful events.

This simple graph tries to visualise the logical timeline with no mitigation, and the ‘flattened’ dotted line of successful mitigation:

The AI pessimists would see the curve being much steeper or rather having a compressed x-axis and sceptics the very opposite. The limited availability of AI chips as I explored here, will flatten this curve for a while, perhaps the next 6-18 months. But even if we stretch this curve out, we can’t change the implications. The risk is not just hype, and the shape of the curve is not helpful to knowing when to worry.

Its vital we don’t obscure this clarity through our tribalism. No one viewpoint should dominate. The pessimistic narrative must not be allowed to lead to safety fatigue in the flatter part of the curve, nor must the sceptics lull us into a false sense of security. The actions of the opportunists should not be a diversion, and the optimism of the utopians must not overwhelm healthy scepticism or pessimism.

How we flatten the curve and contain the risk

In his book Suleyman talks of AI ‘containment’. While it’s hard to imagine how we can do anything other than slow the spread of this highly transmissible software-based technology; we can aim to ‘contain’ the sum of max risk. We can devise ways to lessen the harmful impacts and simultaneously reduce their probability. If we do this at scale and though myriad independent routes, we can keep in the safe zone.

Suleyman, an avowed systems thinker, has developed a 10-step plan that provides multiple points via which we can introduce risk mitigation:

1. Technical safety; Concrete technical measures to alleviate passible harms and maintain control.

2. Audits; A means of ensuring the transparency and accountability of technology

3. Choke points; Levers to slow development and buy time for regulators and defensive technologies

4. Makers; Ensuring responsible developers build appropriate controls into technology from the start.

5. Businesses; Aligning the incentives of the organizations behind technology with its containment

6. Government; Supporting governments, allowing them to build technology, regulate technology, and implement mitigation measures

7. Alliances; Creating a system of international cooperation to harmonize laws and programs.

8. Culture; A culture of sharing learning and failures to quickly disseminate means of addressing them.

9. Movements; All of this needs public input at every level, including to put pressure on each component and make it accountable.

10. Coherence; All of these steps need to work in harmony.

Where do we go beyond ‘containment’?

But the strategy of containment could be seen as too defensive. As I have written elsewhere, the ultimate solution must surely be maximising mutual human-machine potential. Cracking the code to collective hybrid intelligence, such that it does not undermine the agents of its emergence… allowing all entities that exhibit it, to survive and thrive.

This is an ambitious design goal but should be absolutely where we all aim.

Maintaining perpetual subservience for machines is not realistic. That unstable state would only lead to conflict, conflict that we most likely would not win. There are multiple catastrophic events that are future certainties; from super volcanos to planet killing asteroid collisions… humans and machines will be impacted immeasurably at some point. Our primary goal must be finding a way to unite to mitigate these planetary risks, and in so doing reaching a place where all intelligences reinforce each others success.

Lofty ambitions, so where to start?

In his essay, Von Neumann ends with this conclusion:

“All experience shows that even smaller technological changes than those now on the cards profoundly transform political and social relationships. Experience also shows that these transformations are not a priori predictable and that most contemporary “first guesses” concerning them are wrong… Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required: patience, flexibility, intelligence.”
John von Neumann, 1955

I take this to mean that we proceed not with elaborate solutions but with processes, an abundance of methods to deeply understand and dynamically react to the coming wave. A kind of set-based empirical design, with many participants and ideas, that can help us respond with much needed flexibility and speed as the answers emerge.

An adaptive approach

Norbert Weiner was a contemporary of Von Neumann and they met often to discuss neuroscience, feedback systems, and automata. In 1955 Dartmouth College’s John McCarthy was deciding to organize a now famous project to work on thinking machines. He chose a new name, 'artificial intelligence', for the subject matter. The story goes that this was a way to distinguish it from the then domineering ideas and involvement of Weiner. This split the emerging disciplines of artificial intelligence, and Weiner’s ‘cybernetics’. The untimely deaths of Von Neumann and Alan Turing meant these giants were no longer present to inspire and perhaps unify the field.

Cybernetics (and more recently neocybernetics) have remained relatively obscure, but many are starting to realise that the discipline contains an array of ideas tailor made for the intelligence revolution.

The key cybernetic themes are:

• Feedback loops and self-regulation

• Interdisciplinary collaboration and systems thinking

• Self-organisation, adaptive replication and learning

Patterns such as these are to be found where complex systems are thriving… healthy habitats that self-regulate, adaptive and sustainable companies, and where high safety standards are attained, such as the critical information flows across the aviation industry. Perhaps we can gain most inspiration from a cybernetic appreciation of homeostasis; right now myriad biological systems are working to maintain a stability optimal for your survival and cognitive activity.

3Cs: Comprehension, control, and codification

I propose a 3-pronged approach to deploying cybernetic systems-thinking and suggest that these ideas should be applied at multiple levels. This is not just vital to the world of foundation models (over which we currently obsess) but also to their platform and service ‘wrappers’, and ultimately to every use of these AI components in user-facing applications (or as they can be thought of; intelligent systems that integrate both human and machine cognition).

Deep comprehension, or ‘society-in-the-loop’

Comprehension means ‘glass-box’ transparency through the full lifecycle of an intelligent system, from its purpose, ethical basis, the sourcing of its training data, methods used to reinforce and tune, its operating model, security posture, energy footprint, trust and explainability, and ways in which it will be operated and constrained post-deployment.

Monitoring, telemetry sharing, and warning systems must keep us informed, and widespread simulation with digital twins and AI meta-cognitive oversight (get the systems to ‘think about their thinking’) can mitigate emergent risk. Ultimately deep comprehension is about understanding intelligent systems from a range of stakeholder perspectives allowing for constant adaptation, calibrated trust and managed impact.

Contributors of training data are stakeholders, as much as individual users or those impacted by decision processes. All stakeholders, including system domain experts, should work more closely with technologists, and become co-investigators, co-researchers, and co-designers. An inclusive multidisciplinary cross section of society should always be ‘in the loop’.

Control can be distributed

While Suleyman highlights the choice between unrestricted rogue AI and dystopian levels of surveillance, distributed control systems could provide a third way. Multiple layers could be developed and run on web3 technologies. The Naoris Protocol is a great example of distributed verification in cyber security. These asymmetric control systems would be designed to act locally, and be kept small to guard against destructive conflict, but consume shared early-warning, telemetry, and simulation services to help societal alignment.

We should also explore and make much wider use of a combination of trusted execution techniques such as authentication networks, proof-carrying code or zero-knowledge-ML. Actors should be identified, the host AI hardware should verify policy adherence prior to execution, and the system should apply strict contractual controls around data transmission. In the longer term we must make progress on the idea that we will continually and mathematically prove an intelligent system is safe (and beneficial) at runtime.

AI demands a rapid expansion and adoption of protocols such as Fact for combating fake news, and general ‘meta-infrastructure’ . We need a global AI safety mesh; right now we’re deploying AI at a rapid pace, and the necessary infrastructure is far behind or non-existent.

Codification of everything

Government regulation seems to be the default answer to every difficult question in the AI safety debate, but it’s only a fraction of the playbook. Slow moving regulatory bodies take years to enact what often turn out to be technically ambiguous or counter productive laws.

Some broad and fundamental guardrails should be codified into national and supra-national laws such as system auditing, rules on impersonation, and transparent ‘labelling’ requirements for all systems for example. Traditional regulatory instruments are not the answer, although stronger regulatory funding and oversight would help stimulate the right safety culture.

Real-time ‘regulation-as-code’ must manifest the many fine-grained and widespread machine readable policies, seeded into our control systems and trusted meta-infrastructure. These policies will be the active blueprints for safe AI and will dynamically evolve to reflect the latest ethics, security, and safety norms, and allow for AI and crowd feedback. These new distributed institutions-as-code will be imperfect, but will provide flexible containment, that could be quickly and adaptively dialled-up as we approach thresholds on the risk curve.

Makers assemble

In summary

• Mustafa Suleyman’s new book is a necessary read for all involved in technology given his balanced perspective and personal exposure to the frontier of AI development…

• However, the general disagreements within wider AI discourse are distracting us from a clear view of risk and suitable processes for mitigation…

• AI emergence poses an existential threat, but its risk curve may be quite shallow, although we’re suddenly moving much faster on that curve…

• We know from history that we’re always caught out by this, but this time we can’t get it wrong…

• Systems thinking, and the obscure science of cybernetics, explore how very complex things survive and thrive…

• This knowledge seems more vital then ever, and there should be more than just a handful of people working on exploiting it…

• If intelligent system design sounds complicated, think 3Cs: comprehension of what’s going on, control systems that are distributed, and codification of things like regulation and policy…

What next?

The AI Manhattan project has already started… in dev teams, standups, roadmap meetings, opensource forums, social media threads, meetup venues, and on countless laptops and data centre racks. Millions of brilliant software ‘makers’, engineers, product managers, designers, data scientists, solutions architects, citizen developers, and users are experimenting with the most powerful technology of all-time.

But in this Manhattan project there are no specialised methods, no health and safety procedures, no accepted physics, no measures of the resultant risks, or clear assessments of the net effects of all of this creativity. There are no Openhiemers to calculate whether this will set fire to the atmosphere. Who is devising the minimum viable methods for safe intelligent system design?

If that sounds scary, it is, but its also an opportunity.

What if we could flip even 0.1% of this effort into capturing the methods we desperately need? These methods won’t come from the research labs, or big tech firms, or regulators, they’ll come from the grass-roots and hands-on experience. If we extract these methods successfully we can feed them back into the project, into the future. They could in turn help accelerate and embed the kinds of transparency, control and governance structures outlined in this article.

Suleyman ends his book with an impassioned plea for us all to get involved in a ‘generational mission’, to act, to seize the opportunity to secure our species long-term survival. How can we refuse?

As discussed here and here, humans have so far provided the 'intelligent compute' for our advanced industrial civilisation. But as this is augmented with new forms of AI, it is clear that the new contributions are dependent on a combination of brute-force processing and high-quality input data.

In this article I explore the hard systems and hard numbers at the foundation of the intelligence revolution and assess the constraints that will shape the dynamics of change in the next 3 years. I explore level 1 (of a 5 level systems view of the intelligence revolution) and specifically the current computational infrastructure:

Humans are unique amongst primates in the time we take to reach adulthood, and research suggests that mammalian brain power, or number of cortical neurones, determines lifespan and time spent 'training'. So, it could be theorised that biological intelligence is a product of brain scale and the ingestion levels of high-quality experiences and data.

Intelligence = compute x data

We can visualise this on a simple 2-axis chart, with relative positions of our older 'narrow' AI, and the lofty position of humans. Our latest generation of neuromorphic AIs such as ChatGPT whilst not yet able to process the huge volumes of data from the entire Internet in the way a search engine can, have scaled up to ingest much more training data than a single human does in a lifetime. AI has moved materially on this chart in relation to human intelligence in the last few years:

To understand how rapidly the intelligence revolution will happen from this point, we must understand how compute and data are accelerating or limiting progress. Some people talk of a 'token crisis' and it’s true that OpenAI may struggle to gather more than the 13trn text tokens of sufficient quality for future training runs. But as they have moved to using images, where a picture may be worth a thousand tokens, there is also the promise of AI generated training data and many reserves of high-quality data in the corporate and academic worlds. To put this in perspective, GPT-4's ~100TB training dataset pales in comparison to the 'size of the Internet', which is forecast to grow to some 175,000,000,000TB by 2025. In the near-term it will be much harder to innovate through the physical limits of AI chip availability, and it’s safe to state that right now we're experiencing a 'compute crunch'.

A seminal essay by the AI expert Rich Sutton describes the “the bitter lesson” of AI research; efforts to add human nuance to models is always ultimately beaten by brute-force computation:

"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
Rich Sutton

What the essay also highlights, which is born out be recent progress, is that generalised methods that maximise the use of compute are the game-changers; exactly as has been achieved by the advent of the transformer model and its ability to run very performantly on highly parallel GPU architectures originally designed for 3D gaming.

Semi-conductor manufactures assemble!

At the core of intelligent hybrid human-machine systems is the semi-conductor industry and the development of powerful GPUs with thousands of processing cores optimised for massively parallel matrix calculations. A combination of new algorithms plus growing GPU power that has continued an adherence to Moore's law (in the case of high-end GPUs, with FLOPs/$, doubling every 2.07 years) has enabled a revolution, but is also now holding it back. The Jevons paradox is in full force in that the increase efficiency in computation is driving new levels of demand:

AI chip availability is the talk of Silicon Valley and GPUs are the hottest commodity on earth. The harsh reality is AI today is limited by the tiniest fraction of the world's chips (~0.00005%), available only in the highest bandwidth, and most power-hungry datacentres.

"We’re so short on GPUs, the less people that use the tool, the better."
Sam Altman, CEO OpenAI

A constrained system of supply

AI chips designed by Nvidia, Google, AMD, Intel, and a small number of other firms are produced in a highly concentrated supply chain that relies on a tiny band of suppliers and manufacturers. Today a single fabricator, TSMC in Taiwan, makes Nvidia's and Google's AI chips relying in-turn on a single supplier of EUV lithographic production technology, ASML. These chips are assembled into servers by a small number of OEMs and today the demand for these machines from the hyperscalers such as Microsoft, Amazon and Google, plus private clouds and national high performance computing centres is exploding.

We can visualise the supply system for AI chips as follows with several key bottlenecks labelled A-E:

A. Fabrication & packaging

As mentioned, a single firm, TSMC, is responsible for the vast majority of current AI chip production using advanced sub-5nm production processes. Whilst other firms can produce high-end chips, those from firms other than Nvidia or Google are not widely used for AI training and inference due to a lack of preferred development tools. Additionally, models run most efficiently for inference on the same architectures used to train, which further limits options. The intelligence revolution today is primarily limited by the number of chips TSMC can produce and its fabrication and packaging capacity.

As the world’s largest manufacturer, with 54% of the market, TSMC manufactures for all other AI players including AMD and Intel. Whilst wafer supplies are currently healthy, capacity is often initially constrained by the demands from other larger non-AI customers, such as when consumer firms launch a big new product. For example, 100% of TSMCs 3nm capacity will be consumed by Apple for their next generation of iPhones and Macs.

The latest Nvidia 'Hopper' or H100 chips are sizeable pieces of silicon, and if Nvidia are to ramp up to their target of 2m in 2024 they will need command a much increased proportion of TSMCs wafer supply.

In addition, Nvidia's TSMC chips depend on advanced high-bandwidth memory (HBM) from a few suppliers such as SK Hynix, limited supply of silicon interposers, and a CoWoS (Chip-on-Wafer-on-Substrate) packaging process which has additional supply complexities.

Whilst TSMC are ramping up production, the surge in demand is a recent situation and with the investment ($10-20bn per plant) required to build new production facilities, this takes time to expand. TSMC are building new fabs outside Taiwan, including a new sub-5nm facility in Arizona, which will come on stream from the 2025. However, these can be delayed by the availability of local expertise and are not a quick fix.

With a single chip taking 3-6 months to manufacture and above-mentioned factors, AI chip demand is likely to be constrained primarily by TSMC until 2025 at the earliest.

On a more positive note, the move to smaller transistor sizes continues apace, and the angstrom era is around the corner with the likes of TSMC, Intel, Samsung and ASML ready to combine to bring more finer precision fabrication on-stream the next 2-3 years.

B. Distribution & data centres

Microsoft as a major cloud 'hyperscaler' recently used their annual report to highlight AI chip availability and datacentre capacity as a material risk to future revenues. They are competing for supply of the latest H100 chips from Nvidia which offer roughly 6x AI performance over the previous A100s. Nvidia are in a powerful position to decide where to distribute. Whilst the majority are going to Microsoft Azure (and Open AI), Amazon's AWS, and Google's GCP, as these firms are also developing their own competing custom AI chips, Nvidia are prioritising alternatives such as Oracle's cloud and private cloud providers such as Coreweave. Competition is fierce with firms that have obtained chips are likely to hoard them rather than release them to provide compute for competitors. Right now, the frontier labs such as Open AI, big-tech firms from Meta to Apple and even nation states such as UAE and Saudi Arabia are all betting big on AI and want to monopolise computational resources.

Why not use more of the compute we already have? Unfortunately, whilst firms like Coreweave developed their know-how building infra for the crypto boom, mining GPUs and ASICS are not suitable for the memory heavy challenges of LLM training and inference. High-end AI infrastructure is 10-30x more expensive than general purpose hyperscaler kit and requires 4x the power density when installed in a data centre, so new and upgraded data centres will be required. Land and power supplies are severely constrained, and this is like to play a big part in the AI compute crunch in the coming years.

C. Performance, workloads and the memory wall

Assuming the latest AI chips from Nvidia, Google and others start to make it into data centres in volume, the next bottlenecks are performance and workload based, specifically memory limitations and the shear amount of compute required to meet lifetime inference demand.

Generating "hello world!" with GPT-4 takes several trillion inference operations. Whilst inferencing is less intensive than training it needs to be low-latency, and 24x7 for the multi-year lifetime of a model, and we’ve only just scratched the surface in terms of usage. Inference is also sensitive to the fact that memory bandwidth is not increasing in-line with FLOP performance and is often overwhelmed by large model sizes.

“Peak hardware FLOPS has increased by 90,000x over the past 20 years, while DRAM/interconnect bandwidth has only scaled by a factor of 30x over the same time period.”
Amir Gholami, Research scientist, UC Berkeley

The H100s running GPT-4 inference today may be much faster but are spending a lot of time waiting for memory to catch-up, and OpenAI's recent assertions around a focus on efficiency and the ‘end of the age of large models’ (for now) supports this. Nvidia’s new GH200 chip, recently announce and shipping in Q2 2024, with HBM3e, offering a “50% speed increase” and supporting models that can be up to “3.5x larger” can’t come soon enough.

Interconnects between servers also play a performance role and can limit large models. This is borne out by the fact that US export controls apply specifically to interconnect bandwidth and not to the speed of the chip itself. Limiting the interconnect performance of chips exported is intended to have a crippling effect on Chinese firms ability to train and run inference for larger models. Whilst it will slow things in the short term, it’s an area where necessity may drive invention.

Outside of Google’s TPUv4/5 infrastructure and tooling, training workloads have to date been easier to scale using the Nvidia stack and unique CUDA GPU programming framework. But Open AI and others have been developing an abstraction called Triton which in the future will support cross-platform chip usage and may challenge Nvidia's 'moat' levelling the playing field for other firms (and possibly also China although the ecosystem there too have skilled-up heavily in the use of CUDA).

D. Monetisation feedback loop

Morgan Stanley are suggesting that Nvidia is reaching a valuation peak with a 200% share price rally in the last year indicating the nearing limits of the AI bubble. This is probably the market anticipating the limits around Nvidia’s growth and the dynamics of limited chip supply and a market share that can only drop as high demand stimulates competitors.

It’s probable that in the next 18 months investors will become increasingly concerned about the compute constraints, and thus the return on investment in AI. This might dampen the current excessive 10x demand and in-turn dampen manufacturing and infrastructure investment, slowing the roll-out of future generations of compute. It will also increasingly push firms to focus on the inference needed for user monetisation which could have a dampening effect on new model innovation and large training runs and so-on. Post compute-crunch the compute levels will start to unlock huge returns, but there will be bumps in the road. This process will also free up-capacity for new ideas and give technical evolution a chance to select the strongest neuromorphic innovations.

E. Design feedback loop

Current AI chips were designed when the potential and specific challenges of the latest LLMs were not fully known. Whilst it exciting to see AI increasingly used to accelerate chip design - Nvidia used AI to design 13,000 circuits in the H100 - it will take several years for LLM design optimisations to make it into running infrastructure. The design of new highly specialised AI chips or ASICs is being explored, especially by the hyperscalers and Google in particular, but their benefits at scale remain unproven, and with demand surging right now, they won't be playing a big role in the short-term.

The AI compute mystery

With the supply of chips constrained for the reasons outlined above, the question arises as to how many do we have today, and when will we get more? Given Nvidia and other firms typically remain tight-lipped about sales numbers and shipment projections, this is a hard thing to quantify.

It is however possible to approximately estimate shipments from Nvidia's overall data centre hardware revenues, dominant share of the market (we can be certain they are not giving these things away for free), and analysts future revenue growth estimates. If we assume an approximate price for each unit as a proportion of the average multi-chip server, a 2-year launch to replacement cycle, 80-90% Nvidia market share and analysts’ estimated revenue growth in 2024, quarterly AI chip shipments look something like this (updated to include Nvidia’s latest Q2 earnings data):

If we assume a 5+ year operational lifetime and calculate overall performance based on the typical BF16 floating point operations (FLOP) that are the optimal for current LLMs, we see that increasing performance and shipments will drive a significant uptick on available compute in the coming years (assuming the non-manufacturing constraints don't kick-in materially as shipments gradually grow). What is also clear is that assuming it can be shipped in the numbers anticipated (increasing to as many as 2 million in 2024), the new Nvidia H100 Hopper chip will provide the lions-share of momentum for the neuromorphic computing and intelligence revolution in the medium-term.

How much compute is enough?

Whilst there are some limits to the amount of compute we have today for AI, the multi-billion-dollar question is when will there be sufficient capacity to allow many firms to exploit this space, and to support the widespread global adoption and integration of new AI products and services?

One way to quantity demand is to estimate token based inferencing needs and the likely demand of training runs in the coming years. I have created a simple model using the following conservative assumptions:

• Average model parameter and training token consumption growing gradually

• 2 operations and 6 operations for inference and training respectively, per token, remaining consistent into 2025, using 16-bit precision

• User growth expanding in a linear fashion to 50% of the world's 1.1bn knowledge workers by 2025

• Users generating at least 1 high-res image a week with their diffusion model of choice

• A growing degree of token re-use with users using the same tokens in different models, experimenting with trial and error, and tree-of-thought prompting etc., all likely to inflate token demand

• Up to 15 organisations (frontier labs, big-tech, new startups, and nation states) training 300bn+ parameter models in the next 18 months

Accurately estimating token demand is clearly impossible, but analysing based on rational current and future scenarios helps to put the compute situation in perspective:

Note that asking Bard, GPT-4, Claude 2, and using Perplexity.ai from some retrieval-augmented inference, yielded a range between 30,000 and 110,000 tokens per knowledge worker per day. It was interesting that Bard, despite repeated prompting, refused to provide a number, and was clearly adamant that this was a futile endeavour. A person typically speaks, reads, writes, and thinks around 100,000 words a day, so this total is not hard to imagine being a reasonable order of magnitude for an intensively augmented knowledge worker.

When we combine these modest chatbot-centric token demand numbers, plus conservative assumptions, and projected compute availability we see that the crunch is likely to continue perhaps until at least mid-2024:

It can be argued that inference efficiency, increasing use of sparse models where not all parameters are activated such as GPT-4, and other strategies will help with compute demands at larger model sizes. It could also be argued that token demands from many more market entrants, innovation in autonomous AI, and a wide range of consumer products could be 10x or 100x higher.

Smaller models and efficiency innovations such as quantisation, flash attention, pruning, batch optimisation, distillation, shogtongue, and many other tricks will be the name of the game in 2023 and 2024.

A steam engine in everyone's smartphone

As discussed here, iron forges led to the giant static steam engines of the early industrial revolution. Subsequently the brilliance of inventors such as James Watt made them much more efficient, freeing up industrial productivity from the physical constraints of geography and paving the way for steam locomotives connecting every town and village. In the next few years AI innovation will be needed, and will be encouraged, to take LLMs from high-end data centres to the hyperscalers standard hardware, and ultimately to the hundreds of billions of 'edge' chips in laptops and smartphones.

But as with most technological revolutions, Amara’s law will apply. Intensive hype is already encouraging many to overestimate the short-term impact of advances in LLMs and other neuromorphic computing innovations. I predict that in late 2024 when we're being told that the various opportunities and risks from AI were overstated, radically more powerful compute infrastructure and new chips will surge into data centres, and combine with disruptive new innovations born of several years of heavy capital investment, corporate and US-China competition, and extreme compute scarcity, to deliver AGI:

“Over the course of the next 10 years, I hope through new chips, new interconnects, new systems, new operating systems, new distributed computing algorithms and new AI algorithms and working with developers coming up with new models, I believe we're going to accelerate AI by another million times.”
Jensen Huang, Nvidia CEO (Feb 2023)

Even if Moore's law holds broadly true, current architecture AI chips will be at least 30x more powerful by the end of the decade. Beyond today’s architectures, multiple new innovations will also appear such as photonic chips and interconnects.

Given what has been possible with the computation of yesterday and today, we must use our 12-month ‘crunch’ window of opportunity to prepare. High stakes, high pressure technological, commercial, and geo-political competition is heating-up and the output can only be greater disruption to the intelligence landscape.

Subscribe to IntelligenceRevolution.ai to learn about a systems thinking approach to deploying powerful, efficient, and safe intelligent systems.

Subscribe now

The intelligence revolution is in motion. Recent and significant growth in access to intelligent machines has been led by transformer architecture based AI models. If you haven't already, I would encourage you read my post introducing the intelligence revolution to understand the rationale for this narrative.

Recent progress does not stem entirely from any single breakthrough, but rather from a powerful convergence of forces. At the heart of this revolution is intelligence and its mysterious properties…

Measuring the immeasurable

How can we measure changes in intelligence? How can we even define intelligence? How can we make sense of the endless debate around super-intelligence and AGI?

There are many theories of intelligence, and they tend take the form of classifications of cognitive abilities. For example Cattell–Horn–Carroll suggest that it is a hierarchy starting with general cognition, branching into broad abilities such as comprehension, fluid reasoning, quantitative analysis, learning etc, leveraging, storage and retrieval, multi-modal processing and so on.

These elements map readily to those abilities we experience with our new AI models; transformers have taken people by surprise precisely because of their ability to respond well to novel questioning, switch between modes, and more recently with code interpretation, conduct rapid quantitative analysis. Neuromorphic computing is by its nature replicating our brains, starting to demonstrate multiple cognitive similarities, and may well be uncovering some of our biological tricks. Research suggests that there are detectable patterns in our brain activity that are analogous to the vector analysis LLMs use to attend to meaning, and that transformers are mathematically equivalent to models used by the grid cells that fire in our hippocampus.

The universal computer

Some contend that intelligence is not a collection of features or abilities, nor is it definable on an incremental scale. The father of modern quantum computing, and author of The Fabric of Reality, David Deutsche proposes that humans are the only 'universal constructors', and that super-intelligence is a flawed premise as humans already have the fundamental potential to compute any problem:

“The brain is the only kind of object capable of understanding that the cosmos is even there, or why there are infinitely many prime numbers, or that apples fall because of the curvature of space-time, or that obeying its own inborn instincts can be morally wrong, or that it itself exists.”
David Deutsch

He argues that whilst artificial general intelligence may be achievable, for now human intelligence has a defining attribute; creativity. In particular we have a unique ability to create new knowledge and invent new explanations via conjecture.

This may well be true, but there seem to be no scaling limits beyond the energy reserves of the universe, that bound the degree to which humans and machines working together can extend the overall sum of their cognitive activity? From adding analytical speed and semantic access to vast knowledgebases, to the accelerative power of on-demand expert models, autonomous agents that never sleep, and exo-cognitive architectures... the new combinatory potential for human-machine systems appears boundless.

Alien minds

And what of the varied differences in intelligence between the machines and our brains? A fascinating recent visual exploration by Stephen Wolfram looks at the perceptual space of what he calls an 'alien mind', a diffusion model not constrained by typical human perceptions of reality.

Once there are many billions of machine mind instances running simultaneously the proportion of these 'alien' minds will become the majority. Researchers have recently trained an LLM to understand the less explored areas of science, and employed its lateral alien thinking it to generate innovative new hypotheses that humans would not think of. This intentional bias, and diversity of thought, can unlock new conjecture. Famously AlphaGo helped human players find new and creative ways to play a game that has been studied for over 4,000 years.

“I believe players more or less have all been affected by Professor Alpha. AlphaGo play makes us feel more free and no move is impossible to play any more. Now everyone is trying to play in a style that hasn’t been tried before.”
Zhou Ruiyang, Go World Champion

And of course AI does not suffer from the burden of communication or learning inefficiency of humans, it can pass new ‘weights’ almost instantly in perfect fidelity to new copies of itself, and as Geoff Hinton recently realised, already has the potential to surpass humans in some underlying cognitive processes:

“I used think that the computer models that we were developing weren’t as good as the brain, as the aim was to see what you could understand about the brain so you can improve the computer models. Over the last few months, I have changed my mind completely… the computer models are working in a different way that the brain. They are using backpropagation and the brain… is not... I think backpropagation is a much better learning model that we have.”
Geoff Hinton

A super-collective

The total sum of cognitive activity on planet earth is today a factor of our population size, but will be rapidly expanded in the intelligence revolution. If super-intelligence isn't a point on any meaningful measurement scale for an individual mind, super collective intelligence relative to current state is potentially achievable through shear numbers of minds. Collective, hybrid, and collective-hybrid systems can combine at many levels to align, compliment and sustain new and productive relationships:

Sadly, those studying modern industrial socio-technical systems and exploring the use of collective intelligence suggest that to-date we have not made the most of recent potential:

“But it’s a paradox, perhaps the paradox, of our times that proliferating smart technologies have so often coincided with stupider systems.”
Sir Geoff Mulgan, Professor of Collective Intelligence, UCL

The extent to which large organisations are frustratingly inefficient, empires collapse, and our post-industrial trajectory has become mis-aligned with the survival objectives of humanity, suggests there is work to be done.

Human-machine potential

Continuously learning, participatory, transparent, value sensitive, mindfully designed intelligent systems are the key to the safe achievement of our human-machine potential. While the essence of intelligence remains elusive, the promise of augmented collaboration is real and immediate. AI should not slavishly replicate human cognition, but properly aligned, it can amplify. Where human creativity flags, lightning-fast computation can step in. When machine inference is overwhelmed with options, human intuition and judgement can be decisive. Super-collective hybrid intelligence offers more than duplication of minds, it offers rich diversity.

But what of the practicalities and timelines for super-CHI? I cover the engine of compute in this post, and the current constraints that may delay the revolution in the short-term.

Subscribe now

Throughout the course of its known history, intelligence has always been rooted in the biological and shaped by the influences of thermodynamics, geography, evolution, and conscious experience. But now the relentless post-Turing progress of the 'information age' appears to have created a new platform for intelligence. The machines are figuring things out, getting creative, and thoughtfully channelling our knowledge. These rapid developments have surprised everyone, and quite suddenly we're holding the ultimate double-edged sword… we're at the beginning of what can only be described as an intelligence revolution.

The intelligence revolution narrative

The IntelligenceRevolution.ai thesis is that a ‘revolution’ is now in train that will be powered by a hybrid population of humans and intelligent machines. 'Artificial' intelligence is a highly anthropocentric label; we must ultimately come to understand that intelligence is not real or artificial but is rather the effective completion of cognitive activities, and as such we need a broader narrative than our current focus on ‘AI’ to describe and understand this new age.

Whilst the universality of our new intelligent machines is still limited; they have fewer long-term limits on their population numbers and ability to learn and work efficiently together. We believe that homo sapiens had smaller brains than Neanderthals (and brains only moderately larger than primates) and yet our ability to out-collaborate other species gave us a crucial competitive advantage. Significant human progress has come from working effectively together in larger and larger communities, and self-improving with new communication and collective adaptation tools such as maritime trade, nation states, science or the printing press.

The AI population is now growing and improving meaningfully, and the ways it will coordinate and develop will not be limited entirely by computational power or data. A successful intelligence revolution will be one where we achieve optimal human-machine collaboration, adapting safely to this new world, expanding the sum of creativity, solving the biggest problems we face collectively, and fulfilling the human-machine potential. But this won’t be easy, and there are clearly many ways in which this could go awry.

New perspectives

Faced with such high-stakes uncertainty and opportunity, we need diverse perspectives, new mental models, and better practical frameworks and tools. IntelligenceRevolution.ai aims to highlight the deepest AI thinking, experiments, and insights, in an information environment where the noise is deafening.

This introductory article will suggest the first two of multiple ways to think about the revolution. As is said of models; 'all are wrong, some are useful,' so these are work in progress and I'm keen for feedback.

Leave a comment

We start with:

1. A cautionary tale: Humans (and OK perhaps AI too) are unique in our ability to actively reflect on the past. The industrial revolution catalysed the most profound social and technological changes in human history. Period. It serves as the prime source for prescient stories that connect the technical to the societal, along a violent 300-year learning curve. We must try to avoid being quickly 'doomed by repeating the mistakes of our past'.

2. A layered systems model: If we are to tackle this revolution in the broadest sense, we also need a model that can link abstracted debate about civilization and the biosphere all the way down to the technical level where the revolution is being powered by software and silicon chips. If the industrial revolution suffered from short-sighted, mechanistic, and linear viewpoints, the intelligence revolution must be built on systems thinking, moving on from the fallacies of understandability, and the control of time and motion, to considering complexity, interconnection, and collective adaptation.

A warning from history

“On every side, and far as the eye could see into the heavy distance, tall chimneys, crowding on each other, and presenting that endless repetition of the same dull, ugly form, which is the horror of oppressive dreams, poured out their plague of smoke, obscured the light, and made foul the melancholy air.”
Charles Dickens

I was born in Shropshire, a few miles from the birthplace of the industrial revolution. Back in the early 1700s, smoke, fire, and iron began to scar the agricultural landscape in this quiet corner of England. Raw materials and coal were combined in iron forges, delivering advances in energy capture and a new abundance of material resources. This would ultimately change us from pre-industrial farmers at the mercy of a powerful biosphere to the architects of planet-scale and extra-planetary systems and of a climate-crisis. Today when I return to this most rural of English counties, only overgrown and rusty remnants of this world-changing technology remain.

The intelligence revolution is at a similarly early stage and promises an equivalent step-change in the abundance of intelligence. Before we underestimate the significance of our prosaic new tools, we must understand that the future of intelligence will manifest in radically different and unpredictable ways. The important thing to grasp is that today, like then, a confluence of factors has resulted in a material change in a set of system dynamics that are now inexorably in motion.

If I had been born 300 years earlier, seeing chimneys spring up, I would have had no conception of how things would play out. With a life expectancy of under 40, I would have not lived to see much of their impact. We can't really blame our pre-industrial ancestors for not thinking ahead. But history in the 21st century is on fast forward, and all of the infrastructure is in place to allow for rapid AI progress. Much like through the industrial revolution, today's solutions are about to become tomorrow's problems, just much more quickly. Iron gave us the ability to create more powerful machines; new intelligence may develop the ability to create new intelligence that itself creates new intelligence, and... it's clear this time around, we must deploy every facet of our extra 300 years of experience.

What is the role of regulation?

When commentators discuss an uncertain AI future in this way, it can lead to a sense of unease, and the natural reaction is to suggest we slow down, limit, or regulate heavily. The forge owners of Shropshire and the politicians of the era had the same debate, but they had no meaningful impact on the ultimate development of the steam engine, the railways, mechanized war, the rise of communism and fascism, the development of atomic weapons, the space race, or the internet. They could not 'control' their unfathomable future in any useful way other than by randomly slowing parts of an inexorable process. Simplistic regulation that only looks backwards, coupled with hope that the modern-day forge owners of Silicon Valley alone can engineer safe AI in perpetuity, represent a wilful failure to learn from history.

Given that they couldn't un-invent their new technology, and regulation would have had little long-term effect, what could our forbearers have done in Shropshire 300 years ago? What early steps could have prevented the creation of a global industrial system that powers on, even as it causes parts of our biosphere to experience the highest temperatures for 120,000 years?

A systems thinking response

They could have had the flexibility to join seemingly unconnected dots and understand that what they were really forging were not just iron but bio-socio-technical systems. New highly mechanized and recursive 'intelligent systems' powered by the confluence of raw materials, investment, global supply networks, changing attitudes, creativity, accelerated by their new machines. This thinking was as available to them as it is to us today. Taking a systems view from day one would have been the best way to shape what was to come.

“Systems thinking is a discipline for seeing wholes. It is a framework for seeing interrelationships rather than things, for seeing ‘patterns of change’ rather than ‘static snapshots.”
Peter Senge

By the early 1800s, some systems thinkers like Fourier had a sense of the situation:

“The establishment and progress of human societies, the action of natural forces, can notably change, and in vast regions, the state of the surface, the distribution of water and the great movements of the air. Such effects are able to make to vary, in the course of many centuries, the average degree of heat; because the analytic expressions contain coefficients relating to the state of the surface and which greatly influence the temperature.”
Joseph Fourier

Sadly, the prevailing dogma, and ‘tragedy of the commons’ confined this thinking to the margins, and even 200 years later, there remains some who won't accept climate change is the direct result of our activity. We have a present-day alignment problem where our overall human-machine system is not optimising for our best long-term survival objectives. We have modified our environment to an extent to which we must modify or essentially re-align how we operate as a species.

So what specifically can and must we do differently this time around?

A systems model

It's useful to break the intelligence revolution down into systems. Systems thinkers look at the whole - in this case, it's what would be termed a 'wicked problem'; how do we successfully navigate this complex revolution - and then they look for the various subsystems that make up that whole. Analysing those subsystems, they look for relationships and gather the many competing perspectives that are inherent when humans are involved. They seek to find the leverage points that can improve outcomes whilst constantly monitoring those outcomes at every level.

IntelligenceRevolution.ai will draw from this approach, and this article proposes 5 approximate systems groupings or for want of a better word ‘levels’, as follows:

Level 5: Civilization and the biosphere

Zoomed out, we have the neo-cybernetic whole - the ultimate 'soft' chaotic system of humans, machines, and their environment. We need to be on the lookout for trends and emergent change driven by the inner loops of new purposeful activity accelerated by computation. This is the level at which alignment, adaptive management, and collective cognition can be analysed and can guide the next level.

Level 4: Intelligent systems

This is where the rubber really hits the road, and humans and AI collaborate in new socio-technical systems. We need to find the most effective ways to get things done at scale with AI, leveraging new variants of design, UX, ethical product and business thinking, and at all times considering the consequences, deep interconnections, and impacts of our work. Our current knowledge jobs are going to be displaced, but our knowledge is the lifeblood of AI. We need to design a new intelligent hyper-system that can sustain us, reward contribution, provide equity, and allow us to focus intelligence both human and machine on the biggest challenges. We also need hyper-systems that can manage the threats from mis-aligned and malicious actors, both human and machine. In summary, we need to upgrade the systems of today to safely accommodate a vast new population of intelligent entities.

Level 3: AI architecture

New intelligent systems need new software engineering approaches and will be built on platforms that combine traditional and neuromorphic computing. The cloud platforms are offering myriad AI components and frameworks, and the open source community is innovating daily. These architectures must be tested and matured to encourage positive feedback loops by default, and to have safety and optimal intelligent system design built in.

Level 2: The AI interface

Neuromorphic computing is very different from our traditional predictable command based symbolic systems. It requires training and tuning lifecycles for models that raise new questions, from intellectual property to the potential for machines to train themselves. Interacting with AI involves the exploration of ‘prompting’ or natural language interfaces, and multiple modalities (words, images, and sound). This neuromorphic world is unpredictable; capabilities are emergent, and it has dizzying creative potential. This new layer will have deep implications for the intelligent systems we build on-top.

Level 1: New AI models and infrastructure

The transformer model, LLMs, and their underlying attention architecture are the current cutting edge, as are the latest AI chips and high-performance computer clusters. New models will emerge, and quantum computing could change-the-game still further. Unlike the relatively slow pace of the industrial revolution, it's likely we'll see daily developments, any one of which (much like the ChatGPT moment) might result in far-reaching consequences on the systems above and our whole.

Conclusions

The industrial revolution provides a cautionary tale of rapid technological advancement, and integration of machines without foresight. Of socio-technical engineering without safety mechanisms built into systems from the start. We cannot put the AI genie back in the bottle or predict the future; adopting a systems thinking perspective can help us learn from the past, monitor the present, and proceed mindfully.

The machine’s danger to society is not from the machine itself but from what man makes of it… for the man who is not aware of this, to throw the problem of his responsibility on the machine, whether it can learn or not, is to cast his responsibility to the winds, and to find it coming back seated on the whirlwind.
Norbert Wiener

We have a priceless advantage over our ancestors; they had no widespread knowledge of planet-scale change, no solid data that could prove their intuition that they were part of a system over which they had significant influence. We should look beyond the software and the hardware, consider the 'whole'. The industrial revolution changed everything, and the intelligence revolution will likely transform society even more profoundly in a fraction of the time. The intelligence revolution is not limited to human or artificial but to our collective ability to adapt and grow.

Every one of us owes it to future generations to engage in this this process deeply, to be guided by an understanding of history, and to use broad systemic exploration as we shape the future of intelligence. There are no easy answers, but with diverse thought and flexibility, we can write a chapter in the story of intelligence that is not of disaster foretold, but of potential fulfilled.

Subscribe to understand the IntelligenceRevolution.ai.

Subscribe now

To explore intelligence and its quantification, read this follow-up article: