WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/parallelism/

1
00:00:00.070 --> 00:00:13.809
A seven billion parameter model needs 130 gigabytes to train, more than one card holds, so we will compare data parallel, pipeline parallel, tensor parallel and FSDP.

2
00:00:13.809 --> 00:00:23.893
Most of that footprint is not the 14 gigabytes of weights but the Adam optimiser, which burns 84 gigabytes on master weights and two moments.

3
00:00:23.893 --> 00:00:34.578
Strategy one, data parallel, is the surprise: every card still carries all 130 gigabytes, so it buys speed and saves no memory.

4
00:00:34.578 --> 00:00:46.006
Strategy two, pipeline parallel, cuts the layers into stages, 16 gigabytes a card, but with six micro-batches over half of each step is bubble.

5
00:00:46.006 --> 00:00:55.882
Strategy three, tensor parallel, splits every matrix down the middle, and stitching the halves back after each block only pays over NVLink.

6
00:00:55.882 --> 00:01:08.957
Strategy four, FSDP, shards everything and gathers each layer just in time, cutting a card from 130 to 33 gigabytes for half again the traffic.

7
00:01:08.957 --> 00:01:19.467
So, reach for FSDP first; keep tensor parallel inside one node; and save pipelines for crossing nodes, where InfiniBand runs eighteen times slower.