Generative AI development is moving fast, and that matters. The figures I’ve seen is that it’s developing at somewhere between 3x-6x Moore’s law. This is commonly framed as 2x number of transistors in a chip in ~18-24 months.
Why does this matter? Because unlike classical AI, the way generative works is that emergent capabilities appear when certain thresholds in the power of the model are met.
The way you see this, if you look at journal papers, is that they chart on the x axis the power of the model (typically by 10x each notch in the axis). And on the y axis is it’s ability to do something (pass a benchmark). And typically you see flat lines on the 0 axis and suddenly the line just hockeysticks towards 90% quickly, like a J curve. Meaning the model can’t do something at all, and suddenly it can do it quite well, even exceptionally well.
What are Benchmarks?
These benchmarks I mention are tasks that people are able to do, things like, for example:
- order the sentences in a paragraphs,
- recognize what is in a picture,
- discover a word masked in a sentence.
- etcetera.
There’s tons more on various modalities of cognition, including language, speech, vision, etc.
The classical benchmark is the Imagenet challenge from 2010. In this challenge the teams competed to get high scores in a well defined task in image recognition. People were wowed when in 2011, a model called Alexnet beat all others improving the accuracy substantially, in large part by using a deeper model (see below). The AI community then created a large number of benchmarks to be able to measure progress on AI capabilities.
For the technical folks, you can describe the power of models:
- the size of the Large Language Model in “parameters” (think transistors or “neurons”),
- or as amount of compute used to create the model.
These “power” counts miss of course important work such as calibration and expertise from Machine Learning engineers (ML), or feedback from users. But they are valuable metrics nevertheless.
I find it surprising that this discussion of speed of Gen AI development a la Moore’s law hasn’t passed from the AI community to the mainstream press. But it should. As when people say “AI is not good enough yet”, they don’t realize the improvement is probably around the corner.
Sorry I haven’t posted lately. It’s been busy!
What do you think about this topic?