WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/gpu-matmul/

1
00:00:00.251 --> 00:00:15.475
Almost every layer of a model is a matrix multiplication whose arithmetic is fixed, so what really separates CUDA cores, tensor cores, and a TPU systolic array is how many bytes each one has to move for every multiply-add.

2
00:00:15.475 --> 00:00:30.762
Take a 1024 by 1024 product, about a billion multiply-adds: every tile of the floor C needs one row-block of A and one column-block of B, so everything turns on how often those blocks cross the bus.

3
00:00:30.762 --> 00:00:46.982
Here is the failure mode, plain CUDA cores with no tiling: each output re-reads its row, and its column, from HBM, so eight megabytes of input become eight gigabytes of traffic, and the cores sit idle.

4
00:00:46.982 --> 00:01:01.495
Tiling is the fix: each block is loaded into shared memory once and reused thirty-two times, traffic falls to about 270 megabytes, and the same cores climb to around forty percent busy.

5
00:01:01.495 --> 00:01:18.713
Now, the part most explainers miss: a tensor core does sixty-four multiply-adds per instruction, yet it needs nearly three hundred flops per byte, so the same tiling leaves it starved, idle about ninety-five percent of the time.

6
00:01:18.713 --> 00:01:31.089
A TPU systolic array solves this in hardware: values flow from neighbour to neighbour, so one read from memory feeds a whole row, while the sums build along a diagonal wave.

7
00:01:31.089 --> 00:01:42.604
So, check the roofline before you buy compute: below the ridge, bigger tiles and more reuse are what speed you up, and only past it do tensor cores pay off.