How large are LLMs in Generative AI? In particular how big is the training data used to create these models. I wanted to poke at this question as it’s hard to get a sense of scale when there are so many “billions” thrown around.

Here’s one attempt at putting it in context. All of the books ever written are in the order of 100M. Let’s assume -to make our task simple- that each book has 100K words. Most books have less, maybe 50K words. So that means if you add up all the words in all the books ever written you get about 10 trillion words. In ML/AI this translates to 10 trillion “tokens”.

We know that some recent LLM models were trained with about 1 trillion tokens. To put that in perspective this would be comparable to creating an AI that has embedded in its “parameter weights” around 10% of our entire civilizations’ book corpus. That’s a lot of knowledge!

The Growing Knowledge in AI Models

I played this out and was surprised at how sizeable the training data is. So much, we’re able to use in training some of these models.

Also, you have to add the speed at which we are increasing our compute/datasets to train LLMs. In terms of compute it’s growing at about 6x per year, maybe a bit more. It’s hard to pin this number down. I haven’t seen the comparable for size of datasets. You can see where this is going and at what speed. That’s a huge amount of knowledge in these AI models.

I hope this “back of the napkin” calculation had give you some sense of scale around the amount of knowledge in LLM’s.

Of course, to train these models they don’t use books only. They often use web crawl data, and other types of public datasets.

For those in different business domains, could you imagine the power of these AI’s if you could train them with that type of scale in your own business domain?