This post is on the recent history of AI, in particular the role of standard datasets and competitions. People inside the machine learning culture know the competitions and benchmarks well. By now, many important problems have standard datasets which teams compete on to get better scores.

History of AI: Imagenet

For example, there is Imagenet. This is a dataset of millions of images that have been labeled with what is shown on each image. This could be an animal, person, object, etc.

Teams have been competing in building AI models for Imagenet with a higher score every year. That shows they are improving in recognizing things in images never before seen by the AI.

Glue: General Language Understanding Evaluation

Similarly, there is Glue, which is a benchmark for measuring how well an AI understands written language. And many more benchmarks exist for other problems.

What people don’t know is that it took a lot of smarts to realize that creating these standard datasets, their benchmarks, and wrapping competitions around them would accelerate advances in the field. If you posit a challenge, and can keep score, AI people want to participate and try to win. Imagenet is credited with being the first such challenge. And there are a few articles about it.

Here is one such article if you’re interested in its history:
ImageNet: A Pioneering Vision for Computers

So it’s interesting to note that people think that AI scientists focus most of their time on finding the right algorithm. In truth, AI scientists spend a large amount of time building the right dataset to train their AI. This focus on the dataset was a key ingredient in the advances we’ve made in the last decade. When we hit a wall in solving for a benchmark, new ML infrastructure/capabilities are deployed which enable new solutions for it.

If you’re looking at building AI applications, you should reflect on what area or use case your AI needs to be good at (language understanding, vision / images, speech, tabular data) and then look at some standard benchmarks. It will help you in getting a sense of how good current approaches are for roughly what you’re trying to do.