Thirty-two threads, one instruction

A ground-up introduction to CUDA for programmers who have never written a kernel, where every number was measured on a real GPU.

18 min readcudagpuperformance

The first CUDA program I ever benchmarked spent more than 98% of its wall-clock time not running on the GPU. The kernel itself was fine. It was everything around it — copying data in, copying data out — that ate the clock. That one measurement taught me more about GPUs than any diagram of streaming multiprocessors ever had, and it’s a good place to start this post.

What follows is the introduction to CUDA I wish I’d had: the programming model, a first kernel, and then four small experiments — vector addition, reduction, matrix multiplication and kernel fusion — that each teach one idea about how GPUs actually spend their time. All of the code comes from my lab notebooks (kernel-labs-cuda, systems-under-load and paper-kernel), and every number was measured, not estimated.

Why a GPU at all

A CPU is built to make one thread go fast. Most of its silicon is spent on things that aren’t arithmetic: branch predictors, out-of-order execution machinery, and large caches that hide memory latency. A GPU makes the opposite bet. It strips each core down to almost nothing and spends the transistors on more of them.

Left: a CPU with a large control unit, four cores and big caches. Right: a GPU drawn as a grid of forty small streaming multiprocessors.
Figure 1. A CPU spends its transistors on making a few threads fast. A GPU spends them on running thousands of threads at once.

The consequence is that a GPU doesn’t hide memory latency by being clever — it hides it by having so much other work in flight that, while one group of threads waits for memory, another group is ready to compute. That only works if you give it a lot of parallel work. A GPU running a hundred threads is a very expensive, very slow CPU.

The programming model

CUDA asks you to write one function, the kernel, from the point of view of a single thread. You then launch it across thousands of threads at once, organised in a two-level hierarchy:

A grid made of nine blocks; one block is expanded to show four warps of threads and a strip of shared memory; one thread is expanded to show it owns its registers.
Figure 2. A launch is a grid of blocks; each block is a group of threads that runs on one SM and can share fast memory. The hardware executes threads 32 at a time, in warps.
  • A thread runs your kernel once. It has its own registers and its own index.
  • A block is a group of up to 1,024 threads that are guaranteed to run on the same SM. Threads in a block can share a small, fast scratchpad (shared memory) and wait for each other with __syncthreads().
  • A grid is all the blocks in a launch. Blocks are independent; the GPU runs them in whatever order it likes, on whichever SMs are free.
  • A warp is the hardware’s unit of execution: 32 consecutive threads in a block that issue the same instruction at the same time. You don’t declare warps, but you can’t write fast code without thinking about them.

Every thread can ask where it is through a handful of built-in variables:

VariableMeaningTypical use
threadIdxthis thread’s index within its block (x, y, z)which element within the tile
blockIdxthis block’s index within the gridwhich tile of the problem
blockDimthreads per blockstride between blocks
gridDimblocks in the gridgrid-stride loops
warpSizealways 32 on NVIDIA hardwarewarp-level tricks

For a one-dimensional problem, the global index of a thread is

i=blockIdx.x⋅blockDim.x+threadIdx.xi = \texttt{blockIdx.x} \cdot \texttt{blockDim.x} + \texttt{threadIdx.x}

and that line appears, in some form, in nearly every kernel ever written.

A first kernel: adding two vectors

Here’s the whole thing — the kernel from my first lab, plus the host code that drives it:

__global__ void vectorAdd(const float* a, const float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {            // the last block may hang off the end
        c[i] = a[i] + b[i];
    }
}

int main() {
    const int n = 1 << 30;                  // ~1 billion floats
    const size_t bytes = size_t(n) * sizeof(float);
    std::vector<float> h_a(n, 1.0f), h_b(n, 2.0f), h_c(n);

    float *d_a, *d_b, *d_c;
    // 1. allocate on the device
    CHECK_CUDA(cudaMalloc(&d_a, bytes));
    CHECK_CUDA(cudaMalloc(&d_b, bytes));
    CHECK_CUDA(cudaMalloc(&d_c, bytes));

    // 2. copy the inputs in
    CHECK_CUDA(cudaMemcpy(d_a, h_a.data(), bytes, cudaMemcpyHostToDevice));
    CHECK_CUDA(cudaMemcpy(d_b, h_b.data(), bytes, cudaMemcpyHostToDevice));

    // 3. launch: round the grid up so every element is covered
    int block = 256;
    int grid  = (n + block - 1) / block;
    vectorAdd<<<grid, block>>>(d_a, d_b, d_c, n);
    CHECK_CUDA(cudaGetLastError());

    // 4. copy the result out
    CHECK_CUDA(cudaMemcpy(h_c.data(), d_c, bytes, cudaMemcpyDeviceToHost));
}

Three things to notice. __global__ marks a function that runs on the GPU but is called from the CPU. The triple-chevron <<<grid, block>>> syntax is the launch configuration: how many blocks, and how many threads in each. And the if (i < n) guard matters, because the grid is rounded up to a whole number of blocks, so the last few threads have nothing to do.

CHECK_CUDA is a three-line macro that turns a returned error code into a crash with a file and line number. Use it on every call. CUDA errors are sticky and asynchronous: a bad launch often doesn’t complain until some unrelated call several lines later.

Lesson one: the PCIe tax

The GPU can’t read your program’s memory directly. Data has to cross the PCIe bus into the card’s own VRAM, the kernel runs there, and the result has to cross back.

Host (CPU and system RAM) and device (GPU and VRAM) connected by a PCIe link; arrows show copy in, kernel launch, and copy out.
Figure 3. Every CUDA program pays for the round trip. Measured on the T4: PCIe moved data at ~4.4 GB/s, while the kernel read VRAM at ~256 GB/s — a 58× difference.

I timed each of the three phases separately with CUDA events, across five problem sizes. Times are in milliseconds, medians across block sizes of 64–512:

ElementsH→D copyKernelD→H copyKernel shareKernel GB/s
2¹⁰0.0190.0100.01523.8%1.2
2¹⁶0.2310.0140.1174.0%54.8
2²⁰4.4640.0631.2001.1%198.6
2²⁴31.3280.79315.4991.7%253.9
2³⁰1,927.8950.2741,049.651.7%256.3

At a billion elements, the kernel took 50 ms; moving the data took almost three seconds. The kernel itself, meanwhile, is doing great: it moves 3×43 \times 4 bytes per element (two reads, one write), so at 2302^{30} elements it streams 12.9 GB through memory in 50 ms. That’s 256 GB/s, about 80% of what the T4’s memory can physically deliver.

Vector add: effective kernel bandwidth
GB/s moved by the kernel alone, excluding PCIe copies
01002003004002¹⁰2¹⁶2²⁰2²⁴2³⁰T4 peak · 320 GB/sNumber of elements256.3 GB/s
Figure 4. Small launches are dominated by fixed overhead (~10 µs per launch). Bandwidth only saturates once there's enough work to fill all 40 SMs.

The left side of that curve is the other half of the lesson. With 1,024 elements, the kernel finishes in about ten microseconds no matter what it does — that’s the fixed cost of launching anything at all. A GPU needs millions of elements before it stops being bored.

Lesson two: how a warp reads memory

Vector add reached 80% of peak bandwidth almost by accident, and the reason is worth understanding. When a warp executes a load, the hardware looks at all 32 addresses together and fetches the 32-byte memory segments that contain them. If neighbouring threads read neighbouring addresses, those requests collapse into a few wide transactions. This is called coalescing.

Top: eight threads read eight adjacent floats that fall in a single memory segment. Bottom: eight threads read every third float, touching three segments and wasting most of each.
Figure 5. The same eight loads, two access patterns. Coalesced loads use every byte fetched; strided loads pay for bytes nobody asked for.

In vectorAdd, thread ii reads a[i], so a warp reads 32 consecutive floats: four perfectly used segments. Flip the indexing so each thread walks down a column of a row-major matrix, and the same instruction suddenly fetches 32 different segments, using 4 bytes out of each 32. Same arithmetic, an eighth of the useful bandwidth.

My notes from that session put it bluntly: coalescing means combining many small memory accesses into one big memory transaction. Almost every optimisation in the rest of this post is a way of arranging for that to happen.

Lesson three: making threads cooperate

Adding two vectors is “embarrassingly parallel”: no thread needs anything from any other. Summing a vector is not. Every element has to end up in a single number, so threads have to combine their work. The standard approach is a tree reduction inside each block, in shared memory, followed by a final sum over the per-block results.

There are two obvious ways to draw that tree, and they perform very differently:

Two reduction trees over eight values. Left, interleaved: active threads 0, 2, 4, 6 then 0, 4 then 0. Right, sequential: active threads 0 to 3, then 0 to 1, then 0.
Figure 6. Both trees sum the same eight numbers to 25. On the left, the busy threads are scattered across every warp; on the right, they're packed together so entire warps can go idle.

My first version used interleaved addressing: at step ss, thread tt is active if tt is a multiple of 2s2s.

for (unsigned int stride = 1; stride < blockDim.x; stride *= 2) {
    // active: every 2nd thread, then every 4th, then every 8th…
    if ((tid % (2 * stride)) == 0) {
        sdata[tid] += sdata[tid + stride];
    }
    __syncthreads();
}

The problem is warp divergence. In the first step, half the threads in every warp are active, so every warp has to execute the instruction while half its lanes sit masked off. The % operator doesn’t help either; integer modulo is slow on a GPU.

The fix is to flip the tree so the active threads are contiguous — sequential addressing:

for (unsigned int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
    // active: threads 0 .. stride-1, packed together
    if (tid < stride) {
        sdata[tid] += sdata[tid + stride];
    }
    __syncthreads();
}

Now, at each step, the active threads fill whole warps and the inactive ones form whole warps that simply retire. The third version goes further: each thread adds two elements while loading from global memory (halving the number of blocks), and once only 32 threads remain, it stops calling __syncthreads() because a single warp is already in lockstep:

__device__ void warp_reduce(volatile float* sdata, unsigned int tid) {
    sdata[tid] += sdata[tid + 32];
    sdata[tid] += sdata[tid + 16];
    sdata[tid] += sdata[tid + 8];
    sdata[tid] += sdata[tid + 4];
    sdata[tid] += sdata[tid + 2];
    sdata[tid] += sdata[tid + 1];
}

Here’s how the three compare on 2242^{24} floats, swept across block sizes:

Reduction: effective bandwidth by block size
2²⁴ floats on a Tesla T4, higher is better
  • Two loads + warp unroll
  • Sequential
  • Interleaved
010020030040064128256512T4 peak · 320 GB/sThreads per block108.7 GB/s54.3 GB/s38.2 GB/s
Figure 7. Fixing divergence buys about 30%; doing more work per thread and unrolling the last warp more than doubles it again. 128 threads per block was the sweet spot for both faster versions.
KernelBest blockTime (ms)Elements/s% of peak BW
Interleaved (naive)641.47611.4 G14%
Sequential addressing1281.11215.1 G19%
Two loads + warp unroll1280.45437.0 G46%

A reduction does one addition per four bytes loaded, so it’s entirely memory-bound; the only question is how close to 320 GB/s you can get. The best version reaches 46%, against vector add’s 80%. The remaining gap comes from each thread doing too little work. The usual next steps are a grid-stride loop so each thread sums many elements before touching shared memory, float4 vectorised loads, and replacing the shared-memory warp stage with __shfl_down_sync, which passes values between registers directly.

Lesson four: reuse is everything

Matrix multiplication is the opposite of a reduction. For C=ABC = AB with N×NN \times N matrices, there are 2N32N^3 floating-point operations but only 3N23N^2 numbers. In principle every number loaded from memory can be reused NN times. Whether a GEMM is fast is almost entirely a question of how much of that reuse you capture.

The naive kernel gives each thread one output element and lets it walk the whole dot product by itself:

__global__ void sgemm_naive(int M, int N, int K,
                            const float *A, const float *B, float *C) {
    const uint row = blockIdx.x * blockDim.x + threadIdx.x;
    const uint col = blockIdx.y * blockDim.y + threadIdx.y;
    if (row < M && col < N) {
        float acc = 0.0f;
        for (int k = 0; k < K; ++k)
            acc += A[row * K + k] * B[k * N + col];  // both from VRAM
        C[row * N + col] = acc;
    }
}

Each thread loads 2K2K values to do 2K2K FLOPs. The caches rescue some of that, but the kernel still treats VRAM as if it were free. The fix is shared-memory tiling: a block of 16×1616 \times 16 threads cooperatively loads a 16×1616 \times 16 tile of AA and one of BB into shared memory, synchronises, and then every thread reads its operands from the fast on-chip copy.

Matrices A, B and C. A tile from a row panel of A and a tile from a column panel of B are loaded into shared memory arrays As and Bs, which produce one tile of C.
Figure 8. One step of a tiled GEMM. The block loads one tile from each input, every thread reuses it 16 times, then the tiles slide one step along K.
template <const int TILE>
__global__ void sgemm_shared(int M, int N, int K,
                             const float *A, const float *B, float *C) {
    __shared__ float As[TILE * TILE];
    __shared__ float Bs[TILE * TILE];

    const uint tRow = threadIdx.x / TILE, tCol = threadIdx.x % TILE;
    // point A, B and C at this block's row panel, column panel and output tile
    A += blockIdx.x * TILE * K;
    B += blockIdx.y * TILE;
    C += blockIdx.x * TILE * N + blockIdx.y * TILE;

    float acc = 0.0f;
    for (int k0 = 0; k0 < K; k0 += TILE) {
        // each thread loads one element of each tile
        As[tRow * TILE + tCol] = A[tRow * K + tCol];
        Bs[tRow * TILE + tCol] = B[tRow * N + tCol];
        __syncthreads();  // wait until both tiles are complete

        A += TILE;
        B += TILE * N;
        for (int k = 0; k < TILE; ++k)
            acc += As[tRow * TILE + k] * Bs[k * TILE + tCol];
        __syncthreads();  // don't overwrite tiles still in use
    }
    C[tRow * N + tCol] = acc;
}

Two __syncthreads() calls per step, and both are load-bearing: the first makes sure nobody reads a half-loaded tile, the second makes sure nobody overwrites a tile someone else is still reading. Forgetting either one doesn’t crash — it silently gives you almost the right answer, which is much worse.

Here’s how the versions compare on a 1024×1024×10241024 \times 1024 \times 1024 FP32 GEMM:

SGEMM throughput, 1024³ FP32
GFLOP/s on a Tesla T4, higher is better
01,0002,0003,000Naive (Triton)43Naive (CUDA)376Tiled 16×16 (CUDA)867cuBLAS (PyTorch)2,594
Figure 9. Tiling alone is a 2.3× win over the naive kernel. cuBLAS is another 3× beyond that, using larger tiles, register blocking and hand-tuned scheduling.
ImplementationTime (ms)GFLOP/svs. cuBLAS
Naive (Triton)49.9824360× slower
Naive (CUDA)5.7053766.9× slower
Tiled 16×16 (CUDA)2.478*8673.0× slower
cuBLAS via PyTorch0.8282,594—

* The tiled timing comes from my kernel-fusion experiment below, and includes a fused bias-add and ReLU, which cost almost nothing next to the GEMM.

The Triton result is the surprise. Triton is a fantastic tool, but my naive Triton kernel launched one program per output element — a million tiny programs, each computing a single dot product — and Triton can’t fix a bad decomposition for you. A proper Triton GEMM assigns each program a whole output tile and uses tl.dot, at which point it’s competitive with cuBLAS.

Seeing it all on one chart

There’s a single picture that explains every result above. For any kernel, compute its arithmetic intensity II: FLOPs performed per byte moved from VRAM. With peak compute PpeakP_{\text{peak}} and memory bandwidth BB, a GPU can deliver at most

P=min⁡(Ppeak, I×B)P = \min\left(P_{\text{peak}},\ I \times B\right)

On the T4, that’s min⁡(8.1 TFLOP/s, 320 GB/s×I)\min(8.1\ \text{TFLOP/s},\ 320\ \text{GB/s} \times I). The two terms are equal at I≈25I \approx 25 FLOP/byte, the ridge point. Below it you’re memory-bound, and the only way to go faster is to move fewer bytes. Above it you’re compute-bound.

A log-log roofline plot for the Tesla T4 with a sloped 320 GB/s memory roof meeting a flat 8.1 TFLOP/s compute roof at about 25 FLOP per byte. Vector add and reduction sit far left, naive and tiled GEMM further right, and cuBLAS past the ridge.
Figure 10. A roofline for the T4 with each kernel from this post. Vector add and reduction will always be memory-bound; tiling moves GEMM up the slope by raising its intensity.

Vector add does 1 FLOP per 12 bytes (I≈0.08I \approx 0.08); no amount of cleverness will ever make it compute-bound, so the goal is just to saturate bandwidth. A T×TT \times T shared-memory tile raises GEMM’s intensity to roughly T/4T/4 FLOP/byte in FP32 — about 4 for T=16T = 16 — which is why tiling helps, and why serious GEMMs use much larger tiles plus a second level of blocking in registers.

Lesson five: the fastest byte is the one you never write

The last experiment comes from reproducing a kernel-fusion idea from a recent paper. A neural-network layer rarely ends at the matrix multiply: there’s usually a bias add and an activation afterwards. The straightforward implementation launches three kernels, and each one writes its full output to VRAM for the next one to read back.

Top row: GEMM, bias and ReLU run as three separate kernels, each writing C to VRAM. Bottom row: one fused kernel applies bias and ReLU in registers and writes C once.
Figure 11. Fusing the epilogue keeps the intermediate result in registers. For a 1024×1024 output, that removes about 16 MiB of VRAM traffic per layer.

The fused version applies the bias and ReLU to the accumulator while it’s still in a register, then writes CC exactly once. Same tiled GEMM main loop in both versions, timed over 100 runs after warm-up:

Fused vs. separate epilogue, per-run latency
ReLU(A·B + bias), 1024³ FP32, 100 timed runs on a Tesla T4
  • Separate kernels
  • Fused epilogue
2.42.452.52.552.6120406080100Run2.56 ms2.48 ms
Figure 12. The gap is small but consistent: 2.529 ms vs. 2.478 ms on average, and the fused kernel was faster on all 100 paired runs.

A 2% improvement doesn’t sound like much — until you check it against the roofline. The separate path does one GEMM write plus two read-modify-write passes over the 4 MiB output, about 20 MiB in total; the fused path writes 4 MiB. The difference is 16 MiB, and at 320 GB/s, moving 16 MiB takes

16×220 bytes320×109 bytes/s≈52 μs\frac{16 \times 2^{20}\ \text{bytes}}{320 \times 10^{9}\ \text{bytes/s}} \approx 52\ \mu\text{s}

The measured saving was 51 µs. That’s the whole improvement, explained to the microsecond by bytes not moved. Here the GEMM dominates, so the win is small; in memory-bound parts of a network (normalisation layers, activations, residual adds) the same trick often cuts runtime by half or more, because there almost all of the time is memory traffic.

A beginner’s checklist

When a kernel is slow, these are the questions I now ask, in order:

SymptomLikely causeFirst thing to try
cudaMemcpy dominates the profileData crossing PCIe every iterationKeep tensors resident in VRAM; batch the copies
Runtime flat no matter the sizeLaunch overhead (~10 µs each)Do more work per launch; fuse kernels
Elementwise kernel far below peakUncoalesced or strided accessMake threadIdx.x index the contiguous dimension
Reduction far below peak bandwidthDivergent warps; too little work per threadSequential addressing, grid-stride loops, shuffles
GEMM stuck below 1 TFLOP/sNo data reuseShared-memory tiling, then register blocking
Many small kernels in a rowIntermediates round-tripping through VRAMFuse the epilogue into the producing kernel

Where to go next

Everything here is the foundation, not the frontier. The next rungs on the ladder are register blocking (each thread computes a small tile of outputs, not one), vectorised float4 loads, warp-level tiling, and finally Tensor Cores, which do a 16×16×1616 \times 16 \times 16 matrix multiply as a single warp-wide operation. Those are the steps I’m working through now in systems-under-load, on the way to FlashAttention and mixture-of-experts kernels.

But the core idea never changes. A GPU is a machine with an absurd amount of arithmetic attached to a comparatively narrow pipe to memory. Almost all of CUDA performance work is figuring out how to send fewer bytes down that pipe, and how to keep thirty-two threads at a time all doing the same useful thing.