Why slow experiments are the core problem in ML research
From How Two Brothers Outperformed Big Tech for $10K in 1 Month on Deeptech Decoded, with Sergio Charles.
Transcript
It's like a sort of approximation of how well would the model perform when you test it in the real world. So if you have a data set, you split it into three parts. You split it into the train. That's what you train the model on to like, you basically showed a bunch of images of like cats and dogs and what can it predict? cats and dogs in the end, after you like do all of the gradient descent stuff. ⁓ and then you have the validation data set.
you
That validation data set is what you really use to fiddle with hyperparameters. And so all of the neural architecture search methods were using reinforcement learning actually to try to basically find optimal architectures from that validation loss, which was the sparse signal. And I think you actually mentioned that there's like this sort of reward. in research. think the reward in a lot of research is very much sparse. actually, that's what makes it fundamentally difficult to do the research. But there's this one caveat with machine learning in that you actually get a number signal. And I think that's what attracted BlueG and I to this problem in particular. It's a scientific domain, but it's one for which you actually have in silico the actual
Mm.
the metric that tells you how good this model is. And then came about agents and suddenly, know, with cloud code and all of these various iterations on top of the frontier models, you could use coding agents to sort of devise ⁓ architectures. But there's a still this, there's this really hard problem in ML, which is that you need high experiment throughput in order to determine what are good architectures. So you need to run a ton of different experiments and you need to do that in a way that allows you to then determine what is the next best experiment to take. But each experiment in machine learning takes hours because it takes a long time to train these models, right? ⁓ Think even for instance, GPT, it takes a very long time. The current version of GPT-5, it takes a very long time to even
Hmm.
go one gradient step to train for one step. And so you have this really interesting problem, which is that we want to make the machine learning process an engineering one, but simultaneously it takes a long time to run these experiments. anyway, I don't want to ramble on, but that is sort of the fundamental problem that we are solving.
