The trade-off: higher latency in exchange for efficiency

For the old one, that was a power simulation of like, what is the maximum flops we can produce on our chip and what is the maximum power drain. That's a very simple thing. It doesn't mean that those computations are useful, but it's the maximum peak of the old chip. What the benchmark we're doing now, with our compiler, that's also using MLIR, we have a profiler. So if you're running like a transformer model, for example, we can see like, what is the speed up of X, Y, and Z. And that will always be dependent on how much bandwidth you essentially have like your effective bandwidth. And then what we actually do with our chips, because they are reconfigurable, not in a spatial sense, as in like we don't take this node and this node and put them together. What we actually do is we have a different execution model. So when you compute your, your tensor, like a matrix, you should do a matmul and then you have like an only new activation. And then you sort of have to like move off the matmul results to some other circuit that's doing the your activation, whatever. But there is a way to fuse them together to essentially just be in one go. And the less times you have to read from memory and the less time you have to write from memory and those intermediate steps, the faster your chip becomes, the more throughput you get and the more energy efficient you become. So that is what we're benchmarking on like a transformer. Like how much fusion can we do in a transformer given these constraints? And then it's like a bunch of like information dependence. constraints which limits your fusion, but we have some cool neat tricks around those as well.

Tell me more about it. What sort of trade-offs do you take then? Because you can't maximize on everything.

It's basically higher latency. That's it. But if you're doing like large like training runs and even large inference with a big batch size, that doesn't really matter. But if you're doing stuff on the edge, if you want to have like a... I don't know, running your whatever model on the edge for like a drone or something or on your phone, then the high latency is not really good for that. But as also notice, that's not really the workloads we care about because we care more about the actual computational things that will lead us to ASI or whatever you want to call it.

You mean the spray intelligence? Yeah. What is the definition of intelligence for you?

being able to adapt and getting what you want. I think like... For humans, if you get the AI to get what they want, then I know that's not do that actually, but for humans at least, I would think it would be like there's a lot of very smart people who aren't intelligent enough to sort of achieve their goals and they sort of like burrow down in this one direction and he gets stuck and then they like obsess over details that doesn't really matter and then they waste time. And then they end up writing. 50 papers and they end up with a PhD and that's it versus someone who actually built something that's very useful for a of people and you move the needle for humanity. That is the major difference between the two.