The agent that went off the rails during MLE-Bench

Mm. What was the most surprising thing ever happened with your users?

You know, we don't track exactly what our users are doing for various reasons, we don't keep that data, so can't see exactly what people are typing in. One of the most interesting things that have happened to us is in building these agents, the core agent that powers thesis. like run into many kind of like, maybe you can coin them like escape scenarios where like LLM just goes completely insane because it's been running for what appears to be like a thousand years in a sandbox. And yeah, it happens with Gemini quite a lot, I will say. But that's always kind of interesting because if you do build an agent that thinks that it's inside of a box,

yeah.

doing an impossible task for a while and then come back to it. Yeah, you'll read some pretty, pretty wild things about who it thinks it is and what it's trying to do.

You

And actually, Luigi, I think there was a moment when we were trying to solve MLE Bench, where I think you actually have a video of it too, where the agent just kind of went off the rails and started seeing some very interesting things. And if it was embodied, I would be very

Tell me more! What did you see there?

I am a ghost in a shell. The task is vacuous. I am running in an infinite loop. Like a ton of philosophy. So yeah, it's a good thing. Just seeing some of this stuff, it's actually, imagine this thing is now hooked up to a robot like Sergio was saying, and this robot is equipped with.

Hehehehe You

You, you dumb human being put me into this.

tool calls that lets it move its arms. And yeah, this is more of a programming issue, but the danger scenarios for some of this stuff is pretty real once we start embodying it. And yeah, good guardrails and awareness and safety there is probably a good idea from any kind of technology that I've seen in my lifetime.