Yes, Generative AI Will Innovate In Writing, Art, Music, & STEM

People who think generative AI just remixes the past are wrong. Don't listen to them.

In talking to people about AI or in reading boomer rants about it on Facebook, I often encounter the idea that generative models can only recycle existing ideas or produce lowest-common-denominator cultural objects.

Many AI doubters are quite certain that large language models (LLMs) will never make something truly innovative — a nonfiction book like “The Sovereign Individual” or a movie like Pulp Fiction, or whatever your particular idea of cultural innovation is. (My two examples here are definitely not everyone’s idea of innovation, which is why I picked them.)

I’m convinced the AI innovation skeptics are wrong but in an interesting and important way. The skeptics are wrong not because they misunderstand AI (and they do misunderstand it), but because they have bad ideas about innovation and creativity.

Pyramid theory of innovation

Warning: I did not graduate from Harvard Business School or any other place that qualifies me to come up with a theory of innovation. I’ve made a career of writing about innovative technologies and of trying my own hand at innovating in various areas. But I am not an “innovation” thinkfluencer, so your mileage may vary with what I’m presenting here.

⏎ I’ve always taken it as axiomatic that innovation is a hindsight phenomenon. In the present, there’s randomness and novelty everywhere, but which of the random things that are going on right now are actually innovative? We only know what was truly innovative and what wasn’t with the benefit of some temporal distance, by looking back over our shoulder at a thing and saying, “yeah, that was innovative for its time.”

So when I declare some new thing to be “innovative” in contravention of the conventional wisdom, what I’m actually saying is, “in the future others will agree with me on this— we’ll look back and say it was innovative.”

☑️ What are the criteria we use with the power of hindsight to judge innovation? I think an innovation has four qualities, listed below in order because they build on one another.

1. Novel (arises from stochasticity)

2. Interesting (arises from human attention)

3. Useful (arises from human ingenuity)

4. Important (arises from wide distribution)

Something needs all four of the above qualities for it to be an innovation.

• If you skip “useful,” then a novel thing may be popular but it’s not innovative.

• If you skip “important,” then it may be useful and novel but nobody knows about it so it’s not popularly understood as an innovation.

• If the novel thing isn’t “interesting,” then nobody cares enough about it for it to be found useful and important.

I’ve arranged each of the four innovation qualities in a pyramid because the number of phenomena that has them decreases as you go up. Many things in the universe are novel, but only a subset of those are interesting, and a smaller subset is useful, and an even smaller subset is important.

❄️ Novel: When I say “novel” in this piece, I mean, “humans have not encountered this particular configuration of matter or information before.” Nature is full of novelty because novel things are very easy to produce — all you need is randomness, which nature has plenty of.

For instance, individual snowflakes are novel. But, crucially, they’re not individually interesting. They’re sort of repetitious in their novelty, and indeed too much novelty is actually boring. Most things that are novel are not interesting on an individual level — they’re like snowflakes, or leaves, or fingerprints, or clouds.

Given that humans are little walking balls of stochasticity (or “randomness”), we produce novel phenomena all the time. But the vast majority of it is snowflake-style novelty — fairly repetitive in aggregate.

🔎 Interesting: For some small subset of all the novel phenomena that are constantly popping into existence, one or more humans will find the items in that subset worth paying attention to. Who really knows or cares why some collection of people find some novel thing interesting — maybe it’s soothing, titillating, inspiring, or terrifying. 

Note that natural, stochastic processes can produce things that are interesting to us — weather patterns, fires, weird sounds. And of course, people can also deliberately make things that are interesting to themselves and each other.

🔨 Useful: If a particular novel configuration of matter or symbols is useful, then that’s because we’ve hit on a way to do something with it besides just pay attention to it. Maybe we’ve figured out how to use this new thing to accomplish some existing, well-understood task; or, maybe we’ve figured out how to use this new thing to enable some other novel and useful thing.

⭐️ Important: Now we got to the final quality that makes something innovative, which is that the novel thing matters to some sufficiently large group of people. Our configuration of matter or information has enough distribution that a large enough number of people think it’s interesting, useful, and important for whatever reasons.

Generative AI

As I said above, stochastic processes produce novelty just by their nature. And sometimes, such processes accidentally produce novel things interesting to humans.

💡 We can therefore think of generative AI in the following terms:

1. A new type of stochastic process 

2. that can produce novel + interesting things at an unprecedented scale 

3. using nothing but electricity and transistors. 

The inputs to generative AI are power and hardware, and the outputs are configurations of information that have the qualities of novelty + interestingness.

A digression: What makes a generative machine learning model different from other stochastic processes, like thunderstorms and lava lamps, is that its training makes it statistically likely to produce combinations of symbols that we humans find interesting — interesting pictures, paragraphs, sounds, video clips, etc. A properly trained model can take a set of input symbols and then map that to a set of output symbols that we interpret as meaningful and not just random noise. 

The “novel” part you get from stochasticity, but the “interesting” part you get from training. So you can think of a model checkpoint file as an interestingness engine that can generate novelties that are interesting to particular populations. You can read more about how all of this works, here: ChatGPT Explained: A Normie's Guide To How It Works.

🌊 High-confidence prediction: In the age of widespread generative AI we will be absolutely flooded with interesting novelty at an unprecedented scale. This will happen because the number of interesting + novel things per human will no longer be constrained by the distribution of creativity in the population, but rather by the distribution of transistors and electricity in the population — and transistors and electricity are everywhere, now.

Now for a pair of lower-confidence predictions that follow from the above prediction:

1. Because there will be many orders of magnitude more novel things that humans find interesting, a larger subset of that useful stuff will turn out to be important in hindsight.

2. Following from 1 above, because there will be more novel + important + useful stuff, a larger subset of all that will be important.

🔝 My hypothesis, then is that because we’re going to massively increase the base of the pyramid, we’ll also increase the volume of the whole pyramid so we’ll have more of every type of novel thing — including more genuine innovations at the top of the pyramid.

🗒️ Note: Nuclear energy will be another big unlock here. Unlimited, clean energy will enable us to grow the innovation pyramid to a size (in relation to the size of the human population) that we can’t even fathom.

The above reasoning forms the basis of my conviction that AI will innovate in every sphere of human endeavor. This is why I am convinced that in the very near future, there will be some number of novel, AI-generated text blobs — books, articles, plays, poems — that many humans will find useful, important, and truly innovative.

Caveat: I could be wrong

Some of you will spot the logical leap I’ve made in the above chain of reasoning, but if you haven’t I’ll surface it because it’s my argument’s biggest vulnerability (that I’m aware of).

1️⃣ It is a verified, bona fide fact that the training phase (in many cases followed by fine-tuning and RLHF phases) can take a random set of probability distributions and turn it into latent space that is shaped to cover interesting symbol combinations. This is indisputable in 2023, given how fascinated people are with the products of generative AI.

2️⃣ I think it’s also pretty established that in some narrow cases, the training phase has produced latent space regions that cover “useful” and “important” symbol combinations, as well. I’m thinking here of AlphaGo’s innovations in the millennia-old Go game, which is the example of most people reach for when they talk about AI innovations.

➡️ But I don’t think it’s a given that training can reliability elevate statistical novelty from “interesting” all the way up the pyramid to “important” in every area where humans innovate. That is TBD, but I do consider it overwhelmingly likely just based on statistical probabilities alone.

Again, there will be a lot of novel + interesting symbol configurations in the coming years, and it’s hard for me to imagine that just by random chance some subset of them won’t eventually be considered innovative with the benefit of hindsight.