4.5 Inner products and orthogonality

The vector-space axioms — addition and scalar multiplication — let you add and stretch arrows but say nothing about length or angle. Pythagoras’ theorem, the cosine rule, the very idea of “perpendicular” — none of these are part of the bare vector-space structure. To recover them, you need one extra piece of equipment: an inner product.

This lesson defines the inner product, develops the geometric notions it makes possible (length, angle, projection, orthogonality), and walks through the Gram–Schmidt procedure that turns any linearly-independent set of vectors into an orthonormal basis. The same machinery, lifted to function spaces, is what makes Fourier expansion and modal projection work — it is the algebraic structure underlying Foundations 6.5.

The dot product

In $\mathbb{R}^n$ the inner product is the familiar dot product:

\mathbf{u} \cdot \mathbf{v} \;=\; u_1 v_1 + u_2 v_2 + \cdots + u_n v_n \;=\; \sum_{k=1}^n u_k v_k.

This is just a single number — the inner product of two vectors is a scalar, not another vector. Three properties pin it down and generalise to other inner-product spaces:

Symmetric. $\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}$ .
Bilinear. $(\alpha \mathbf{u} + \beta \mathbf{w}) \cdot \mathbf{v} = \alpha (\mathbf{u} \cdot \mathbf{v}) + \beta (\mathbf{w} \cdot \mathbf{v})$ , and similarly in the second slot.
Positive definite. $\mathbf{v} \cdot \mathbf{v} \geq 0$ , with equality if and only if $\mathbf{v} = \mathbf{0}$ .

Any operation on a vector space with these three properties is called an inner product, often written $\langle \mathbf{u}, \mathbf{v} \rangle$ in more abstract settings. The integral $\int_0^L f(x) g(x)\, dx$ , for instance, satisfies all three properties (with functions $f, g$ as the “vectors”), and so it is an inner product on the space of integrable functions on $[0, L]$ . That fact is what makes the Fourier-projection formula in Foundations 6.5 work mechanically the same way as the projection of a 2-D vector onto an axis.

Length and angle

The inner product gives you a norm (length):

\|\mathbf{v}\| \;=\; \sqrt{\mathbf{v} \cdot \mathbf{v}}.

In $\mathbb{R}^2$ this is just Pythagoras: $\|(x, y)\| = \sqrt{x^2 + y^2}$ . The norm satisfies the triangle inequality $\|\mathbf{u} + \mathbf{v}\| \leq \|\mathbf{u}\| + \|\mathbf{v}\|$ — a consequence of the Cauchy–Schwarz inequality $|\mathbf{u} \cdot \mathbf{v}| \leq \|\mathbf{u}\|\, \|\mathbf{v}\|$ , which is one of the most-used inequalities in mathematics.

The inner product also encodes the angle between two vectors:

\cos \theta \;=\; \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\, \|\mathbf{v}\|}.

Two vectors are orthogonal (perpendicular) if their inner product is zero. In $\mathbb{R}^n$ orthogonality is the algebraic condition $\sum_k u_k v_k = 0$ ; in function-space settings it becomes $\int f(x) g(x)\, dx = 0$ . The integral version is what makes the Fourier sine series of Foundations 6.3 work:

\int_0^L \sin(m \pi x / L)\, \sin(n \pi x / L)\, dx \;=\; \tfrac{L}{2}\, \delta_{mn},

— different-mode sines are orthogonal as elements of the function space $L^2[0, L]$ .

Projection

If $\mathbf{u}$ is a unit vector (i.e. $\|\mathbf{u}\| = 1$ ), the projection of any vector $\mathbf{v}$ onto $\mathbf{u}$ is

\text{proj}_\mathbf{u}(\mathbf{v}) \;=\; (\mathbf{v} \cdot \mathbf{u})\, \mathbf{u}.

The scalar $\mathbf{v} \cdot \mathbf{u}$ is the length of the projection — how much of $\mathbf{v}$ points along $\mathbf{u}$ . The vector $(\mathbf{v} \cdot \mathbf{u})\, \mathbf{u}$ is the projection itself, a vector parallel to $\mathbf{u}$ .

The residual $\mathbf{w} = \mathbf{v} - \text{proj}_\mathbf{u}(\mathbf{v})$ is the component of $\mathbf{v}$ perpendicular to $\mathbf{u}$ . Two checks: $\mathbf{u} \cdot \mathbf{w} = \mathbf{u} \cdot \mathbf{v} - (\mathbf{v} \cdot \mathbf{u}) \cdot 1 = 0$ (so $\mathbf{w} \perp \mathbf{u}$ ✓), and $\mathbf{v} = \text{proj}_\mathbf{u}(\mathbf{v}) + \mathbf{w}$ (so the decomposition is exact ✓).

Projection is the bread-and-butter geometric operation of linear algebra. If $\{\mathbf{e}_1, \ldots, \mathbf{e}_n\}$ is an orthonormal basis, then any vector decomposes as

\mathbf{v} \;=\; \sum_{k=1}^n (\mathbf{v} \cdot \mathbf{e}_k)\, \mathbf{e}_k.

That formula is the Fourier expansion of $\mathbf{v}$ in the basis $\{\mathbf{e}_k\}$ . When the basis is the modes of a PDE rather than $\mathbb{R}^n$ basis vectors, the same formula extracts modal coefficients from initial data; that is the Fourier-projection step in Foundations 6.5.

Gram–Schmidt orthogonalisation

Given a set of linearly-independent vectors $\{\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_k\}$ , we want to produce an orthonormal set $\{\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_k\}$ spanning the same space — orthogonal to each other and each of unit length. The Gram–Schmidt procedure does this iteratively, by subtracting projections.

The recipe for two input vectors $\mathbf{a}$ and $\mathbf{b}$ :

$\mathbf{e}_1 = \mathbf{a} / \|\mathbf{a}\|$ . (Normalise the first vector.)
$\mathbf{w} = \mathbf{b} - (\mathbf{b} \cdot \mathbf{e}_1)\, \mathbf{e}_1$ . (Subtract from $\mathbf{b}$ its component along $\mathbf{e}_1$ .)
$\mathbf{e}_2 = \mathbf{w} / \|\mathbf{w}\|$ . (Normalise.)

For three or more inputs the procedure continues: subtract from each subsequent vector its components along all previously orthonormalised vectors, then normalise.

Initial input: two linearly-independent vectors a and b.

Drag the blue and red circles to set the input vectors a and b. Step through the procedure: normalise a, subtract its component from b to get a perpendicular remainder w, then normalise w. The small right-angle marker confirms that the green w is perpendicular to e₁. The output {e₁, e₂} is an orthonormal basis for the same plane that {a, b} spanned.

Drag the blue and red input vectors $\mathbf{a}$ and $\mathbf{b}$ . Step through the three operations. The little right-angle marker in step 2 confirms that the green residual $\mathbf{w}$ is perpendicular to $\mathbf{e}_1$ ; in step 3 it gets normalised to $\mathbf{e}_2$ . The output basis $\{\mathbf{e}_1, \mathbf{e}_2\}$ spans the same plane as the input but with orthogonal axes.

Worked example

▶ Worked example: Gram–Schmidt on three vectors in R³

The problem. Find an orthonormal basis for the span of

\mathbf{a}_1 = (1, 1, 0), \quad \mathbf{a}_2 = (1, 0, 1), \quad \mathbf{a}_3 = (0, 1, 1).

These three vectors are linearly independent (verify by computing $\det = 2 \neq 0$ ) and so span all of $\mathbb{R}^3$ ; the output will be an orthonormal basis of $\mathbb{R}^3$ .

Step 1 — Normalise $\mathbf{a}_1$ .

\|\mathbf{a}_1\| = \sqrt{1^2 + 1^2 + 0^2} = \sqrt{2}, \qquad \mathbf{e}_1 = \frac{1}{\sqrt{2}} (1, 1, 0).

Step 2 — Subtract the $\mathbf{e}_1$ -component from $\mathbf{a}_2$ to get $\mathbf{w}_2$ .

\mathbf{a}_2 \cdot \mathbf{e}_1 = \frac{1}{\sqrt{2}} (1 \cdot 1 + 0 \cdot 1 + 1 \cdot 0) = \frac{1}{\sqrt{2}}.

\mathbf{w}_2 = \mathbf{a}_2 - (\mathbf{a}_2 \cdot \mathbf{e}_1)\, \mathbf{e}_1 = (1, 0, 1) - \frac{1}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}} (1, 1, 0) = (1, 0, 1) - \tfrac12 (1, 1, 0) = (\tfrac12, -\tfrac12, 1).

Check that $\mathbf{w}_2$ is orthogonal to $\mathbf{e}_1$ : $\mathbf{w}_2 \cdot \mathbf{e}_1 = \tfrac{1}{\sqrt{2}} (\tfrac12 - \tfrac12 + 0) = 0$ . ✓

Step 3 — Normalise $\mathbf{w}_2$ .

\|\mathbf{w}_2\| = \sqrt{(\tfrac12)^2 + (\tfrac12)^2 + 1^2} = \sqrt{\tfrac{3}{2}} = \frac{\sqrt{6}}{2}.

\mathbf{e}_2 = \frac{2}{\sqrt{6}} (\tfrac12, -\tfrac12, 1) = \frac{1}{\sqrt{6}} (1, -1, 2).

Step 4 — Subtract $\mathbf{e}_1$ - and $\mathbf{e}_2$ -components from $\mathbf{a}_3$ .

\mathbf{a}_3 \cdot \mathbf{e}_1 = \frac{1}{\sqrt{2}} (0 + 1 + 0) = \frac{1}{\sqrt{2}}.

\mathbf{a}_3 \cdot \mathbf{e}_2 = \frac{1}{\sqrt{6}} (0 \cdot 1 + 1 \cdot (-1) + 1 \cdot 2) = \frac{1}{\sqrt{6}}.

\begin{aligned} \mathbf{w}_3 &= \mathbf{a}_3 - (\mathbf{a}_3 \cdot \mathbf{e}_1)\, \mathbf{e}_1 - (\mathbf{a}_3 \cdot \mathbf{e}_2)\, \mathbf{e}_2 \\ &= (0, 1, 1) - \frac{1}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}} (1, 1, 0) - \frac{1}{\sqrt{6}} \cdot \frac{1}{\sqrt{6}} (1, -1, 2) \\ &= (0, 1, 1) - (\tfrac12, \tfrac12, 0) - (\tfrac16, -\tfrac16, \tfrac13) \\ &= (-\tfrac23, \tfrac23, \tfrac23). \end{aligned}

Check orthogonality: $\mathbf{w}_3 \cdot \mathbf{e}_1 = \tfrac{1}{\sqrt{2}} (-\tfrac23 + \tfrac23 + 0) = 0$ ✓, and $\mathbf{w}_3 \cdot \mathbf{e}_2 = \tfrac{1}{\sqrt{6}} (-\tfrac23 - \tfrac23 + \tfrac43) = 0$ ✓.

Step 5 — Normalise $\mathbf{w}_3$ .

\|\mathbf{w}_3\| = \sqrt{(\tfrac23)^2 \cdot 3} = \tfrac{2}{\sqrt{3}}.

\mathbf{e}_3 = \frac{\sqrt{3}}{2} \cdot (-\tfrac23, \tfrac23, \tfrac23) = \frac{1}{\sqrt{3}} (-1, 1, 1).

Step 6 — Final answer. The orthonormal basis is

\boxed{\;\mathbf{e}_1 = \tfrac{1}{\sqrt{2}}(1, 1, 0), \quad \mathbf{e}_2 = \tfrac{1}{\sqrt{6}}(1, -1, 2), \quad \mathbf{e}_3 = \tfrac{1}{\sqrt{3}}(-1, 1, 1).\;}

Each has length 1; each pair is orthogonal; together they span $\mathbb{R}^3$ .

Inner products on function spaces

The bridge to PDEs: the same recipe — projection, orthogonality, Gram–Schmidt — works on infinite-dimensional spaces of functions, provided the inner product is the integral

\langle f, g \rangle \;=\; \int_a^b f(x)\, g(x)\, dx

(possibly weighted; the Sturm–Liouville theorem describes which weight functions are appropriate for which differential operators). The functions $\sin(n \pi x / L)$ on $[0, L]$ are pairwise orthogonal under this inner product, with $\|sin(n \pi x / L)\|^2 = L/2$ . Therefore the Fourier sine series

f(x) = \sum_n c_n \sin(n \pi x / L), \qquad c_n = \frac{2}{L} \int_0^L f(x)\, \sin(n \pi x / L)\, dx,

is exactly the orthonormal-basis expansion $\mathbf{v} = \sum_k (\mathbf{v} \cdot \mathbf{e}_k)\, \mathbf{e}_k$ from the finite-dimensional case, lifted to the function space. The arithmetic is identical; the only difference is that “sum” becomes “integral” in the inner product, and " $\sin(n\pi x/L)$ " plays the role of $\mathbf{e}_n$ . This is the deep reason separation of variables and Fourier expansion work: the natural eigenfunctions of self-adjoint differential operators are orthogonal under an appropriate inner product, and the spectral theorem in 4.6 guarantees their completeness as a basis.

What we use this for

Inner products are everywhere across the bookshelf:

Fourier-projection for mode coefficients (Foundations 6.5) — the $A_n$ formula is an inner product divided by a norm squared.
Orthogonality of mode shapes in Sound 7.6 — tube modes, Sound 7.7 — cavity modes, Sound 7.8 — room modes — distinct cavity modes are orthogonal under the appropriate volume integral.
Vector projection in 3-D geometry — every plane-wave acoustic problem involves projecting a wavevector $\mathbf{k}$ onto direction vectors, computing inner products like $\mathbf{k} \cdot \mathbf{r}$ to find the phase at a position.
Inner products of complex signals in Hearing 5.3 — phase-locking — vector strength is a circular-mean inner product of $e^{i \phi}$ unit vectors, generalising the dot product to complex unit phasors.
Optimal control and matched filters — the matched filter that maximises signal-to-noise ratio for detecting a signal $s(t)$ in noise is the inner product of the received waveform with $s(t)$ itself, just as projection extracts the component of one vector along another.

The next lesson, 4.6, ties together everything in the chapter: the eigenvalues of 4.4, the orthogonality of this lesson, and the spectral theorem that guarantees a self-adjoint operator’s eigenvectors form a complete orthonormal basis. That theorem is what makes mode expansions for PDEs not just convenient but mathematically complete.