From b2778f9f38bf04fe2e801b10019c02d57f2e7062 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:06:48 +0800 Subject: [PATCH 1/9] remove numba, add motivation and matrix convention --- lectures/eig_circulant.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index 270aa6046..d1cdf6b5a 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -17,21 +17,21 @@ kernelspec: This lecture describes circulant matrices and some of their properties. -Circulant matrices have a special structure that connects them to useful concepts -including +Building on eigenvalues and eigenvectors from {doc}`linear_algebra`, we use circulant matrices to connect several useful concepts, including * convolution * Fourier transforms * permutation matrices -Because of these connections, circulant matrices are widely used in machine learning, for example, in image processing. +These connections appear elsewhere in this series: {doc}`hoist_failure` uses Fourier transforms to compute convolutions, while {doc}`svd_intro` develops related matrix decomposition tools. + +Circulant matrices are also widely used in machine learning, for example, in image processing. We begin by importing some Python packages ```{code-cell} ipython3 import numpy as np -from numba import jit import matplotlib.pyplot as plt ``` @@ -66,7 +66,6 @@ first column needs to be specified. Let's write some Python code to generate a circulant matrix. ```{code-cell} ipython3 -@jit def construct_cirlulant(row): N = row.size @@ -90,7 +89,8 @@ construct_cirlulant(np.array([1., 2., 3.])) Here are some useful properties: -Suppose that $A$ and $B$ are both circulant matrices. Then it can be verified that +Suppose that $A$ and $B$ are both circulant matrices of the same order and constructed +using the same cyclic shift convention. Then it can be verified that * The transpose of a circulant matrix is a circulant matrix. @@ -200,7 +200,6 @@ $$ Let's write some Python code to illustrate these ideas. ```{code-cell} ipython3 -@jit def construct_P(N): P = np.zeros((N, N)) From 5e58696792cc9af7d89e73d263a19499a1e95d23 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:14:34 +0800 Subject: [PATCH 2/9] add def --- lectures/eig_circulant.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index d1cdf6b5a..bc5335e27 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -45,6 +45,19 @@ To construct an $N \times N$ circulant matrix, we need only the first row, say, $$ \begin{bmatrix} c_{0} & c_{1} & c_{2} & c_{3} & c_{4} & \cdots & c_{N-1} \end{bmatrix} .$$ +```{prf:definition} Circulant matrix +:label: def-circulant-matrix + +An $N \times N$ matrix $C$ is **circulant** if there are numbers $c_0, \ldots, c_{N-1}$ such that + +$$ +C_{ij} = c_{(j-i) \bmod N}, +\qquad 0 \leq i,j \leq N-1. +$$ + +Equivalently, each row is obtained from the previous row by shifting entries one step to the right. +``` + After setting entries in the first row, the remaining rows of a circulant matrix are determined as follows: @@ -89,8 +102,9 @@ construct_cirlulant(np.array([1., 2., 3.])) Here are some useful properties: -Suppose that $A$ and $B$ are both circulant matrices of the same order and constructed -using the same cyclic shift convention. Then it can be verified that +Suppose that $A$ and $B$ are both circulant matrices of the same order and constructed using the same cyclic shift convention. + +Then it can be verified that * The transpose of a circulant matrix is a circulant matrix. From 811382d507cdc9041779d7b128f1d0c385327b72 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:17:02 +0800 Subject: [PATCH 3/9] notation update --- lectures/eig_circulant.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index bc5335e27..0ef14b8e1 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -125,16 +125,18 @@ Now consider a circulant matrix with first row The **convolution** of vectors $c$ and $a$ is defined as the vector $b = c * a $ with components $$ - b_k = \sum_{i=0}^{n-1} c_{k-i} a_i + b_k = \sum_{i=0}^{N-1} c_{k-i} a_i $$ (eqn:conv) +Here and below, indices such as $k-i$ are interpreted modulo $N$. + We use $*$ to denote **convolution** via the calculation described in equation {eq}`eqn:conv`. It can be verified that the vector $b$ satisfies -$$ b = C^T a $$ +$$ b = C^\top a $$ -where $C^T$ is the transpose of the circulant matrix defined in equation {eq}`eqn:circulant`. +where $C^\top$ is the transpose of the circulant matrix defined in equation {eq}`eqn:circulant`. From 133d16d9677ebbca251fc4ebea557c1bf18719e2 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:21:18 +0800 Subject: [PATCH 4/9] update formula and notation --- lectures/eig_circulant.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index 0ef14b8e1..dadf40546 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -192,7 +192,7 @@ $$ and solving $$ -\textrm{det}(P - \lambda I) = (-1)^N \lambda^{N}-1=0 +\textrm{det}(P - \lambda I) = (-1)^N(\lambda^N - 1)=0 $$ @@ -205,7 +205,7 @@ Thus, **singular values** of the permutation matrix $P$ defined in equation {eq It can be verified that permutation matrices are orthogonal matrices: $$ -P P' = I +P P^\top = I $$ From 0f6432c5a051921038d7d6f4678f284b26cbc200 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:26:02 +0800 Subject: [PATCH 5/9] python update --- lectures/eig_circulant.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index dadf40546..ce00357d2 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -216,7 +216,7 @@ $$ Let's write some Python code to illustrate these ideas. ```{code-cell} ipython3 -def construct_P(N): +def construct_cyclic_shift_matrix(N): P = np.zeros((N, N)) @@ -228,7 +228,7 @@ def construct_P(N): ``` ```{code-cell} ipython3 -P4 = construct_P(4) +P4 = construct_cyclic_shift_matrix(4) P4 ``` @@ -239,7 +239,7 @@ P4 ```{code-cell} ipython3 for i in range(4): - print(f'𝜆{i} = {𝜆[i]:.1f} \nvec{i} = {Q[i, :]}\n') + print(f'𝜆{i} = {𝜆[i]:.1f} \nvec{i} = {Q[:, i]}\n') ``` In graphs below, we shall portray eigenvalues of a shift permutation matrix in the complex plane. @@ -264,7 +264,7 @@ for i, N in enumerate([3, 4, 6, 8]): row_i = i // 2 col_i = i % 2 - P = construct_P(N) + P = construct_cyclic_shift_matrix(N) 𝜆, Q = np.linalg.eig(P) circ = plt.Circle((0, 0), radius=1, edgecolor='b', facecolor='None') @@ -302,7 +302,7 @@ F_{8}=\left[\begin{array}{ccccc} \end{array}\right] $$ -The matrix $F_8$ defines a [Discete Fourier Transform](https://en.wikipedia.org/wiki/Discrete_Fourier_transform). +The matrix $F_8$ defines a [Discrete Fourier Transform](https://en.wikipedia.org/wiki/Discrete_Fourier_transform). To convert it into an orthogonal eigenvector matrix, we can simply normalize it by dividing every entry by $\sqrt{8}$. @@ -347,7 +347,7 @@ Q8 @ np.conjugate(Q8) Let's verify that $k$th column of $Q_{8}$ is an eigenvector of $P_{8}$ with an eigenvalue $w^{k}$. ```{code-cell} ipython3 -P8 = construct_P(8) +P8 = construct_cyclic_shift_matrix(8) ``` ```{code-cell} ipython3 From 54f6609741f001a65631ea963f4f20897fb5355f Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:29:03 +0800 Subject: [PATCH 6/9] update associated permutation section --- lectures/eig_circulant.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index ce00357d2..ce0e0418f 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -79,7 +79,7 @@ first column needs to be specified. Let's write some Python code to generate a circulant matrix. ```{code-cell} ipython3 -def construct_cirlulant(row): +def construct_circulant(row): N = row.size @@ -95,7 +95,7 @@ def construct_cirlulant(row): ```{code-cell} ipython3 # a simple case when N = 3 -construct_cirlulant(np.array([1., 2., 3.])) +construct_circulant(np.array([1., 2., 3.])) ``` ### Some Properties of Circulant Matrices @@ -368,15 +368,11 @@ Next, we execute calculations to verify that the circulant matrix $C$ defined i $$ -C = c_{0} I + c_{1} P + \cdots + c_{n-1} P^{n-1} +C = c_{0} I + c_{1} P + \cdots + c_{N-1} P^{N-1} $$ and that every eigenvector of $P$ is also an eigenvector of $C$. -```{code-cell} ipython3 - -``` - We illustrate this for $N=8$ case. ```{code-cell} ipython3 @@ -388,10 +384,10 @@ c ``` ```{code-cell} ipython3 -C8 = construct_cirlulant(c) +C8 = construct_circulant(c) ``` -Compute $c_{0} I + c_{1} P + \cdots + c_{n-1} P^{n-1}$. +Compute $c_{0} I + c_{1} P + \cdots + c_{N-1} P^{N-1}$. ```{code-cell} ipython3 N = 8 @@ -418,7 +414,7 @@ Now let's compute the difference between two circulant matrices that we have co np.abs(C - C8).max() ``` -The $k$th column of $P_{8}$ associated with eigenvalue $w^{k-1}$ is an eigenvector of $C_{8}$ associated with an eigenvalue $\sum_{h=0}^{7} c_{j} w^{h k}$. +The $j$th column of $Q_{8}$ is an eigenvector of $C_{8}$ associated with eigenvalue $\sum_{k=0}^{7} c_k w^{j k}$. ```{code-cell} ipython3 𝜆_C8 = np.zeros(8, dtype=complex) From 0a299cf97d2c1171820bf499ca29c64c88dab861 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Tue, 7 Jul 2026 22:33:03 +0800 Subject: [PATCH 7/9] update dft section --- lectures/eig_circulant.md | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index ce0e0418f..66fc968e9 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -441,7 +441,7 @@ for j in range(8): The **Discrete Fourier Transform** (DFT) allows us to represent a discrete time sequence as a weighted sum of complex sinusoids. -Consider a sequence of $N$ real number $\{x_j\}_{j=0}^{N-1}$. +Consider a sequence of $N$ real numbers $\{x_j\}_{j=0}^{N-1}$. The **Discrete Fourier Transform** maps $\{x_j\}_{j=0}^{N-1}$ into a sequence of complex numbers $\{X_k\}_{k=0}^{N-1}$ @@ -509,7 +509,7 @@ def plot_magnitude(x=None, X=None): if (X is not None): data.append(X) names.append('X') - xs.append('j') + xs.append('k') num = len(data) for i in range(num): @@ -537,7 +537,7 @@ x_{n} = \sum_{k=0}^{N-1} \frac{1}{N} X_{k} e^{2\pi\left(\frac{kn}{N}\right)i}, \ $$ ```{code-cell} ipython3 -def inverse_transform(X): +def inverse_DFT(X): N = len(X) w = np.e ** (complex(0, 2*np.pi/N)) @@ -551,7 +551,7 @@ def inverse_transform(X): ``` ```{code-cell} ipython3 -inverse_transform(X) +inverse_DFT(X) ``` Another example is @@ -562,7 +562,7 @@ $$ Since $N=20$, we cannot use an integer multiple of $\frac{1}{20}$ to represent a frequency $\frac{11}{40}$. -To handle this, we shall end up using all $N$ of the availble frequencies in the DFT. +To handle this, we shall end up using all $N$ of the available frequencies in the DFT. Since $\frac{11}{40}$ is in between $\frac{10}{40}$ and $\frac{12}{40}$ (each of which is an integer multiple of $\frac{1}{20}$), the complex coefficients in the DFT have their largest magnitudes at $k=5,6,15,16$, not just at a single frequency. @@ -625,7 +625,7 @@ X = DFT(x) X ``` -Now let's evaluate the outcome of postmultiplying the eigenvector matrix $F_{20}$ by the vector $x$, a product that we claim should equal the Fourier tranform of the sequence $\{x_n\}_{n=0}^{N-1}$. +Now let's evaluate the outcome of postmultiplying the eigenvector matrix $F_{20}$ by the vector $x$, a product that we claim should equal the Fourier transform of the sequence $\{x_n\}_{n=0}^{N-1}$. ```{code-cell} ipython3 F20, _ = construct_F(20) @@ -635,13 +635,9 @@ F20, _ = construct_F(20) F20 @ x ``` -Similarly, the inverse DFT can be expressed as a inverse DFT matrix $F^{-1}_{20}$. +Similarly, the inverse DFT can be expressed as an inverse DFT matrix $F^{-1}_{20}$. ```{code-cell} ipython3 F20_inv = np.linalg.inv(F20) F20_inv @ X ``` - -```{code-cell} ipython3 - -``` From 568e6ad27b07a6df4f95f3451939072c4c8689a6 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Thu, 9 Jul 2026 22:24:53 +0800 Subject: [PATCH 8/9] update on motivation --- lectures/eig_circulant.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index 66fc968e9..3b15c9c54 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -17,13 +17,15 @@ kernelspec: This lecture describes circulant matrices and some of their properties. -Building on eigenvalues and eigenvectors from {doc}`linear_algebra`, we use circulant matrices to connect several useful concepts, including +Circulant matrices are useful because multiplying by them is closely connected to convolution, and their eigenvectors can be constructed using the Discrete Fourier Transform. + +We use circulant matrices to connect several useful concepts, including * convolution * Fourier transforms * permutation matrices -These connections appear elsewhere in this series: {doc}`hoist_failure` uses Fourier transforms to compute convolutions, while {doc}`svd_intro` develops related matrix decomposition tools. +For background on eigenvalues and eigenvectors, see {doc}`linear_algebra`; for another use of Fourier transforms and convolution, see {doc}`hoist_failure`. Circulant matrices are also widely used in machine learning, for example, in image processing. From 29d24ba0cb115b8e6d38907c818065e955c7dd78 Mon Sep 17 00:00:00 2001 From: Longye Tian Date: Thu, 9 Jul 2026 22:36:03 +0800 Subject: [PATCH 9/9] update the definition --- lectures/eig_circulant.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/lectures/eig_circulant.md b/lectures/eig_circulant.md index 3b15c9c54..f0d021530 100644 --- a/lectures/eig_circulant.md +++ b/lectures/eig_circulant.md @@ -47,19 +47,6 @@ To construct an $N \times N$ circulant matrix, we need only the first row, say, $$ \begin{bmatrix} c_{0} & c_{1} & c_{2} & c_{3} & c_{4} & \cdots & c_{N-1} \end{bmatrix} .$$ -```{prf:definition} Circulant matrix -:label: def-circulant-matrix - -An $N \times N$ matrix $C$ is **circulant** if there are numbers $c_0, \ldots, c_{N-1}$ such that - -$$ -C_{ij} = c_{(j-i) \bmod N}, -\qquad 0 \leq i,j \leq N-1. -$$ - -Equivalently, each row is obtained from the previous row by shifting entries one step to the right. -``` - After setting entries in the first row, the remaining rows of a circulant matrix are determined as follows: @@ -75,6 +62,21 @@ c_{1} & c_{2} & c_{3} & c_{4} & c_{5} & \cdots & c_{0} \end{array}\right] $$ (eqn:circulant) +This pattern can be formalized as follows. + +```{prf:definition} Circulant matrix +:label: def-circulant-matrix + +An $N \times N$ matrix $C$ is **circulant** if there are numbers $c_0, \ldots, c_{N-1}$ such that + +$$ +C_{ij} = c_{(j-i) \bmod N}, +\qquad 0 \leq i,j \leq N-1. +$$ + +Equivalently, each row is obtained from the previous row by shifting entries one step to the right. +``` + It is also possible to construct a circulant matrix by creating the transpose of the above matrix, in which case only the first column needs to be specified.