What MLE-Bench measures and why benchmarks matter
From How Two Brothers Outperformed Big Tech for $10K in 1 Month on Deeptech Decoded, with Luigi Charles.
Transcript
Mm.
Get better today. We just stop here I think that like over the next few years the world is gonna see like incredible benefits and You know the kinds of agents that people are are creating the kinds of systems most white-collar work that can be done in a computer will like 100 % for sure like go go this way, which is a pretty pretty wild reality and It's all happened in the last three years, which is like the I think even
day. Amazing.
most incredible thing. ⁓
But do you find benchmarks productive? Does it make you have to say in a way, yes, that was a very meaningful benchmark, or that was really motivating and keeping the team more productive? Because they all depends on a certain way of evaluating. I was wondering your opinion on this.
Yeah, benchmarks are important insofar as they are good benchmarks to be going after and measuring. Things like MLE Bench, which are just straight taking Kaggle competitions and testing how well your agents do on them, I think are pretty reasonable benchmarks. There is a bit of hill climbing and perhaps in academia, some gaming to it. ⁓
Mm.
Yeah, like overall, Emily Bench, think when it started, the percentage was 20, 30%. Right now, people are submitting models, agents that perform at 80 % of all tasks. So the jump is incredible. And going after it as a company probably doesn't make a ton of sense. think focusing on your customers and use cases is pretty paramount. But I have no doubt that if we for example, went back to these benchmarks, we can improve a little bit on what's there.
