← Journey · Lab notes · since Apr 2026 Ongoing

Why is the same maths faster?

Notes from writing GPU kernels in Triton on a laptop, checking them against PyTorch, and asking the profiler what was really holding each one back.

GPU: RTX 3060 laptop · 30 SMs · compute 8.6
Baselines: PyTorch eager and torch.compile
Profiler: Nsight Compute (ncu)

01 · Softmax

Beating PyTorch, then trying to do better and failing

My first real kernel was a row-wise softmax: each program loads a whole row, finds its max, exponentiates and normalises, then writes once. At large sizes it beat both baselines.

8192 × 32768, time per call
PyTorch eager	torch.compile	My Triton kernel
13.55 ms	20.20 ms	7.20 ms

The profiler said it was memory-bound (91% of peak memory throughput, 10% compute), which is what a softmax should be. But it also flagged 93 registers per thread, capping occupancy at 33%. That looked like the obvious thing to fix.

So I rewrote it as a tiled online softmax that walks each row in 2048-wide chunks. Registers dropped to 26 and occupancy jumped to 97%.

It ran 73% slower. 12.16 ms against 7.02 ms. Memory throughput barely moved (92%), but the tiled version reads the data about 1.5 times. Occupancy was never the bottleneck: bytes moved were. The profiler’s loudest warning isn’t always the one that matters.

02 · Matmul

Tile sizes, shared memory and tensor cores

Matmul is the opposite case: compute-bound. My first tiled kernel was slightly slower than PyTorch at 2048 × 2048 (2.40 ms vs 2.25 ms). Sweeping tile shapes found 128 × 64 × 32 at 2.15 ms, about 4% ahead. The bigger 128 × 128 × 64 tile wouldn’t even launch: it needed 131 KB of shared memory and the card has 99 KB.

The profiler showed compute at 73% of peak, 192 registers per thread, and one surprising line: the tensor pipe at 0%. In strict IEEE fp32 mode, the tensor cores weren’t being used at all.

2048 × 2048 matmul
PyTorch eager	Best fp32 tile	TF32 (tensor cores)
2.25 ms	2.15 ms	1.55 ms

Allowing TF32 made it 1.45× faster than eager, about 31% less time, with a small, known loss of precision. Compute utilisation fell to 46% because the work moved to the tensor cores.

03 · Fusion

Six trips to memory become two

In PyTorch, bias + GELU + dropout is three separate operations, and each one reads its input from memory and writes its output back: six passes. Fusing them into one kernel means one read and one write.

PyTorchbiasGELUdropout6 passes
Fusedbias + GELU + dropout2 passes

4096 × 1024, dropout p = 0.3
PyTorch eager	Fused Triton	Speed-up
0.370 ms	0.113 ms	3.3×

Two details mattered. GELU needs tanh, which I wrote in terms of exp. And dropout is random, so I checked it statistically instead: 30.05% of outputs were zeroed for p = 0.3.

So far

What I’m taking away

Find out whether a kernel is memory- or compute-bound before changing anything.
For memory-bound kernels, count bytes moved. Occupancy is a means, not the goal.
Check which hardware is actually doing the work. A 0% tensor pipe is easy to miss.
Fusion is the cheapest big win: the fastest memory trip is the one you skip.

The question I keep coming back to: could signals like these, rather than runtime alone, guide tuning automatically?

Every number here comes from the day-by-day journal, with the code and profiler readings: kernelbench-journal on GitHub ↗