Skip to content
Computational Lab.

Gradient Descent

Small steps downhill, one hyperparameter from disaster.

Beginner / optimization / gradient-methods

Gradient descent fits the same noisy line as the Least Squares experiment, but iteratively: the parameters step against the MSE gradient. The contour plot shows the loss surface and the descent path - shrink the learning rate and it crawls, raise it past stability and the path flies off the plot. That contrast with the closed-form page is the entire lesson of first-order methods.

Tagged: learning rate, loss surface, divergence, MSE

Live simulation

Learning rate past stability and the descent path leaves the plot.

Speed

Learning rate past stability and the descent path leaves the plot.

Mathematics

Gradient descent minimizes a loss by stepping against the gradient: theta <- theta - lr * grad L(theta). For the line fit, L(a, b) is the mean squared residual - a bowl in (a, b) space - and the path of the descent is a curve drawn on that bowl.

L(a, b) = (1/N) sum (y_i - a x_i - b)^2

dL/da = -2/N sum r_i x_i; dL/db = -2/N sum r_i

stability: lr < 2 / (largest eigenvalue of the Hessian)

The stability bound is the slider's danger zone: push the learning rate past it and the iterates oscillate outward - divergence, visibly. Below it, the path converges to the same (a, b) the normal equations compute in closed form on the Least Squares page. Same problem, two philosophies, one shared test: both must agree on the answer.

How it works

The loss surface is precomputed on a 72x48 grid in (a, b) space and rendered as banded shading; the descent path is drawn over it with markers every few epochs. Data is generated from a true line with seeded noise; the truth marker shows where the optimum lies.

Each tick performs a fixed number of full-batch steps - no stochasticity, so the path is smooth and the divergence clean.

Implementation

src/engine/experiments/gradient-descent.ts · 189 lines · the module running above

TypeScript · fixed-timestep engine

1import { BaseExperiment } from "../core/base-experiment";
2import { gaussian } from "../core/rng";
3import type { ParameterDef, ThemeColors } from "../core/types";
4
5/**
6 * Gradient descent fitting the same noisy line as the Least Squares
7 * experiment - but iteratively: (a, b) take small steps down the MSE loss.
8 * The contour plot shows the loss surface and the path; a learning rate
9 * that's too large visibly diverges. The end state should agree with the
10 * closed-form solution the other page computes exactly.
11 */
12export class GradientDescentExperiment extends BaseExperiment {
13 readonly id = "gradient-descent";
14
15 private xs: number[] = [];
16 private ys: number[] = [];
17 private a = 0;
18 private b = 0;
19 private epoch = 0;
20 private loss = 0;
21 private path: { a: number; b: number }[] = [];
22 private readonly trueA = 1.2;
23 private readonly trueB = 0.4;
24 // Contour cache in (a, b) space.
25 private contour = new Float64Array(0);
26
27 protected params(): ParameterDef[] {
28 return [
29 { key: "lr", label: "Learning rate", min: 0.01, max: 1.2, step: 0.01, defaultValue: 0.25 },
30 { key: "steps", label: "Steps per tick", min: 1, max: 20, step: 1, defaultValue: 2 },
31 { key: "noise", label: "Data noise", min: 0, max: 0.5, step: 0.02, defaultValue: 0.15 },
32 ];
33 }
34
35 protected onReset(): void {
36 const noise = this.num("noise");
37 this.xs = [];
38 this.ys = [];
39 for (let i = 0; i < 80; i++) {
40 const x = this.rng();
41 this.xs.push(x);
42 this.ys.push(this.trueA * x + this.trueB + gaussian(this.rng) * noise);
43 }
44 this.a = 0;
45 this.b = 0;
46 this.epoch = 0;
47 this.path = [{ a: this.a, b: this.b }];
48 this.computeLoss();
49 this.buildContour();
50 }
51
52 protected onParameterChange(key: string): void {
53 if (key === "noise") this.reset();
54 void key;
55 }
56
57 private mse(a: number, b: number): number {
58 let sum = 0;
59 for (let i = 0; i < this.xs.length; i++) {
60 const r = this.ys[i] - (a * this.xs[i] + b);
61 sum += r * r;
62 }
63 return sum / this.xs.length;
64 }
65
66 private computeLoss(): void {
67 this.loss = this.mse(this.a, this.b);
68 }
69
70 private buildContour(): void {
71 const w = 72;
72 const h = 48;
73 this.contour = new Float64Array(w * h);
74 let min = Infinity;
75 let max = -Infinity;
76 for (let gy = 0; gy < h; gy++) {
77 for (let gx = 0; gx < w; gx++) {
78 const a = (gx / w) * 3;
79 const b = (gy / h) * 1.6;
80 const value = this.mse(a, b);
81 this.contour[gy * w + gx] = value;
82 min = Math.min(min, value);
83 max = Math.max(max, value);
84 }
85 }
86 // Store log-scaled for contrast.
87 for (let i = 0; i < this.contour.length; i++) {
88 this.contour[i] = Math.log10(Math.max(1e-6, this.contour[i] - min) + 1);
89 }
90 void max;
91 }
92
93 protected onUpdate(): void {
94 const lr = this.num("lr");
95 const steps = this.num("steps");
96 const n = this.xs.length;
97 for (let s = 0; s < steps; s++) {
98 // Gradients of MSE over the whole batch.
99 let ga = 0;
100 let gb = 0;
101 for (let i = 0; i < n; i++) {
102 const r = this.ys[i] - (this.a * this.xs[i] + this.b);
103 ga += -2 * r * this.xs[i];
104 gb += -2 * r;
105 }
106 ga /= n;
107 gb /= n;
108 this.a -= lr * ga;
109 this.b -= lr * gb;
110 this.epoch += 1;
111 }
112 this.computeLoss();
113 this.path.push({ a: this.a, b: this.b });
114 if (this.path.length > 600) this.path.shift();
115 }
116
117 render(ctx: CanvasRenderingContext2D, theme: ThemeColors): void {
118 ctx.fillStyle = theme.bg;
119 ctx.fillRect(0, 0, this.width, this.height);
120 const w = 72;
121 const h = 48;
122 // Loss contours (banded log-MSE)
123 const cw = this.width / w;
124 const ch = this.height / h;
125 for (let gy = 0; gy < h; gy++) {
126 for (let gx = 0; gx < w; gx++) {
127 const value = this.contour[gy * w + gx];
128 const t = Math.min(1, value / 2.2);
129 ctx.fillStyle = t > 0.55 ? theme.accent : theme.fgSecondary;
130 ctx.globalAlpha = 0.05 + t * 0.22;
131 ctx.fillRect(gx * cw, gy * ch, cw + 0.5, ch + 0.5);
132 }
133 }
134 ctx.globalAlpha = 1;
135 const toX = (a: number) => (a / 3) * this.width;
136 const toY = (b: number) => (b / 1.6) * this.height;
137 // Descent path
138 ctx.strokeStyle = theme.accent;
139 ctx.lineWidth = 1.8;
140 ctx.beginPath();
141 this.path.forEach((point, index) => {
142 if (index === 0) ctx.moveTo(toX(point.a), toY(point.b));
143 else ctx.lineTo(toX(point.a), toY(point.b));
144 });
145 ctx.stroke();
146 ctx.fillStyle = theme.fg;
147 this.path.forEach((point, index) => {
148 if (index % 6 !== 0 && index !== this.path.length - 1) return;
149 ctx.beginPath();
150 ctx.arc(toX(point.a), toY(point.b), 2.4, 0, Math.PI * 2);
151 ctx.fill();
152 });
153 // True parameters marker
154 ctx.strokeStyle = theme.viz[1];
155 ctx.setLineDash([4, 3]);
156 ctx.beginPath();
157 ctx.moveTo(toX(this.trueA), 0);
158 ctx.lineTo(toX(this.trueA), this.height);
159 ctx.moveTo(0, toY(this.trueB));
160 ctx.lineTo(this.width, toY(this.trueB));
161 ctx.stroke();
162 ctx.setLineDash([]);
163 ctx.lineWidth = 1;
164 ctx.font = "11px ui-monospace, monospace";
165 ctx.fillStyle = theme.fgTertiary;
166 ctx.fillText("LOSS SURFACE over (a, b)", 10, 16);
167 ctx.fillStyle = theme.viz[1];
168 ctx.fillText("TRUE a, b", this.width - 84, 16);
169 }
170
171 getMetrics() {
172 return {
173 epoch: this.epoch,
174 a: this.a.toFixed(4),
175 b: this.b.toFixed(4),
176 loss: this.loss.toExponential(3),
177 trueValues: `${this.trueA} / ${this.trueB}`,
178 };
179 }
180
181 describe(): string {
182 return `Gradient descent at epoch ${this.epoch}: a = ${this.a.toFixed(3)}, b = ${this.b.toFixed(3)}, loss ${this.loss.toExponential(2)} - descending the same surface least squares solves in one step.`;
183 }
184
185 entities(): number {
186 return this.xs.length;
187 }
188}
189

Performance

Time per tick
O(N) per step (one pass over the data)
Space
The data plus a path buffer
Notes
The convergence test asserts the descent end state agrees with least squares within 5% - two pages, one answer.

Live numbers (FPS, frame time, steps per second, microseconds per step, entity count) are in the performance overlay on the simulation above: the pulse icon in its top-right corner.