Orthogonal Bases and Projections

Computation

Display

Basis \(\beta\)
Vector \(\mathbf{x}\)
\(x_1\) = 2
\(x_2\) = 4
Finding the coordinates of a vector with respect to a basis $\beta = \{\mathbf{v}_1, \mathbf{v}_2\}$ normally means solving a linear system or inverting the basis matrix $M_\beta$ (see the Change of Basis page). However, when the basis vectors are pairwise orthogonal, there is a much quicker way: the coordinates are simply (orthognal) projections, which provide significant computational efficeincy. \subsection*{Orthogonal sets and bases} Recall that $\mathbf{u}, \mathbf{v} \in \R^n$ are \textbf{orthogonal} if $\mathbf{u}^\mathsf{T}\mathbf{v} = 0$, and that the norm $\|\mathbf{u}\| = \sqrt{\mathbf{u}^\mathsf{T}\mathbf{u}}$. \begin{definition} A set $S = \{\mathbf{v}_1, \ldots, \mathbf{v}_p\} \subset \R^n$ is an \textbf{orthogonal set} if its vectors are pairwise orthogonal, $\mathbf{v}_i^\mathsf{T}\mathbf{v}_j = 0$ for $i \neq j$. It is an \textbf{orthonormal set} if, in addition, $\|\mathbf{v}_i\| = 1$ for all $i$. A basis of a subspace $W$ that is also an orthogonal (orthonormal) set is called an \textbf{orthogonal basis} (\textbf{orthonormal basis}) of $W$. \end{definition} Orthogonality is a stronger requirement than linear independence: if the vectors of $S$ are nonzero and pairwise orthogonal, then $S$ is linearly independent. (The proof uses the same trick as the theorem below.) So in $\R^2$, any two nonzero orthogonal vectors form an orthogonal basis (resp. in $\R^m$, any $m$ nonzero orthogonal vectors form one too). \subsection*{Coordinates by projection} Recall that the projection of $\mathbf{x}$ onto a nonzero vector $\mathbf{v}$ is $$P_\mathbf{v}\mathbf{x} = \frac{\mathbf{v}^\mathsf{T}\mathbf{x}}{\|\mathbf{v}\|^2}\,\mathbf{v}.$$ \begin{theorem} Let $\beta = \{\mathbf{v}_1, \ldots, \mathbf{v}_p\}$ be an orthogonal basis of $W$ and $\mathbf{x} \in W$. Then the coordinates of $\mathbf{x}$ with respect to $\beta$ are $$c_i = \frac{\mathbf{v}_i^\mathsf{T}\mathbf{x}}{\|\mathbf{v}_i\|^2}, \qquad i = 1, \ldots, p.$$ \end{theorem} \begin{proof} Write $\mathbf{x} = c_1\mathbf{v}_1 + \cdots + c_p\mathbf{v}_p$ and multiply on the left by $\mathbf{v}_1^\mathsf{T}$. By orthogonality every cross term $\mathbf{v}_1^\mathsf{T}\mathbf{v}_j$ with $j \neq 1$ vanishes, leaving $$\mathbf{v}_1^\mathsf{T}\mathbf{x} = c_1\,\mathbf{v}_1^\mathsf{T}\mathbf{v}_1 = c_1\|\mathbf{v}_1\|^2.$$ Since $\mathbf{v}_1 \neq \mathbf{0}$ (it belongs to a basis, so it needs to be linearly independent with the rest of the vectors), we may divide by $\|\mathbf{v}_1\|^2$. The same argument holds with $\mathbf{v}_i^\mathsf{T}$, giving each $c_i$. \end{proof} Each term $c_i\mathbf{v}_i$ is exactly the projection of $\mathbf{x}$ onto $\mathbf{v}_i$. In other words, for an orthogonal basis, $$\mathbf{x} = P_{\mathbf{v}_1}\mathbf{x} + P_{\mathbf{v}_2}\mathbf{x} + \cdots + P_{\mathbf{v}_p}\mathbf{x}.$$ This is the purple arrow in the graph landing on $\mathbf{x}$ (when the basis is, in fact, orthogonal). \begin{remark} If $\beta$ is orthonormal, then $\|\mathbf{v}_i\| = 1$ and the formula simplifies even further: $c_i = \mathbf{v}_i^\mathsf{T}\mathbf{x}$. \end{remark} \begin{example} Let $$\mathbf{v}_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \qquad \mathbf{v}_2 = \begin{bmatrix} -1 \\ \phantom{-}1 \end{bmatrix}, \qquad \mathbf{x} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}.$$ Since $\mathbf{v}_1^\mathsf{T}\mathbf{v}_2 = -1 + 1 = 0$, $\beta = \{\mathbf{v}_1, \mathbf{v}_2\}$ is an orthogonal basis of $\R^2$, with $\|\mathbf{v}_1\|^2 = \|\mathbf{v}_2\|^2 = 2$. We can tehn directly calculate $$c_1 = \frac{\mathbf{v}_1^\mathsf{T}\mathbf{x}}{\|\mathbf{v}_1\|^2} = \frac{3}{2}, \qquad c_2 = \frac{\mathbf{v}_2^\mathsf{T}\mathbf{x}}{\|\mathbf{v}_2\|^2} = -\frac{1}{2}.$$ As is good practice, we should verify with simple vector scaling and addition $$\frac{3}{2}\begin{bmatrix} 1 \\ 1 \end{bmatrix} - \frac{1}{2}\begin{bmatrix} -1 \\ \phantom{-}1 \end{bmatrix} = \begin{bmatrix} 2 \\ 1 \end{bmatrix} = \mathbf{x}.$$ \end{example} \subsection*{Without orthogonality} When $\mathbf{v}_1^\mathsf{T}\mathbf{v}_2 \neq 0$, the cross terms in the proof no longer vanish, and the projections do not sum to $\mathbf{x}$. \begin{example} Let $$\mathbf{z}_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \qquad \mathbf{z}_2 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, \qquad \mathbf{x} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}.$$ These form a basis of $\R^2$, but $\mathbf{z}_1^\mathsf{T}\mathbf{z}_2 = 1 \neq 0$. The true coordinates are $c_1 = c_2 = 1$, since $\mathbf{z}_1 + \mathbf{z}_2 = \mathbf{x}$. The projections, however, are $$P_{\mathbf{z}_1}\mathbf{x} = \frac{2}{1}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 0 \end{bmatrix}, \qquad P_{\mathbf{z}_2}\mathbf{x} = \frac{3}{2}\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 3/2 \\ 3/2 \end{bmatrix},$$ and their sum is $$\begin{bmatrix} 7/2 \\ 3/2 \end{bmatrix} \neq \mathbf{x}.$$ \end{example}