Enforcing Symmetries in Neural Networks#
A physical model often should not depend on an arbitrary choice of origin or orientation. Symmetry makes this requirement precise and lets a neural network enforce the corresponding transformation laws by construction.
Symmetry#
A mathematical object has a symmetry when a transformation either leaves it unchanged or changes it according to a specified rule. We are particularly interested in physical equations written in coordinates. Their form and predictions should transform consistently when the coordinate origin or orientation changes.
Groups#
Group theory organizes transformations and their compositions. A group consists of a set \(G\) and a binary operation that maps each ordered pair \((g_1,g_2)\in G\times G\) to an element denoted by \(g_1g_2\):
We usually omit the symbol \(\cdot\) and write \(g_1g_2\). The operation must satisfy four properties:
Closure: For all \(g_1, g_2 \in G\), \(g_1 g_2 \in G\).
Associativity: For all \(g_1, g_2, g_3 \in G\), \((g_1 g_2) g_3 = g_1 (g_2 g_3)\).
Identity: There exists an element \(e \in G\) such that for all \(g \in G\), \(e g = g e = g\).
Inverses: For all \(g \in G\), there exists an element \(g^{-1} \in G\) such that \(g g^{-1} = g^{-1} g = e\).
Examples of groups#
Familiar number systems give simple examples. The integers \(\mathbb{Z}\) and the real numbers \(\mathbb{R}\) each form a group under addition: the identity is \(0\), and the inverse of \(x\) is \(-x\). The nonzero rational numbers and the nonzero real numbers each form a group under multiplication: the identity is \(1\), and the inverse of \(x\) is \(1/x\). In each case the set is closed under the operation, and the operation is associative. We exclude zero from the multiplicative examples because it has no multiplicative inverse.
For physical models, we need groups of transformations. Translations provide the first example.
The translation group#
Consider the Euclidean space \(\mathbb{R}^3\) with elementwise addition:
For each translation vector \(\mathbf{b}\in\mathbb{R}^3\), define the map \(t_{\mathbf{b}}:\mathbb{R}^3\to\mathbb{R}^3\) by
The translation group is
with function composition as its operation. Since \(t_{\mathbf{b}_1}\circ t_{\mathbf{b}_2}=t_{\mathbf{b}_1+\mathbf{b}_2}\), the map \(\mathbf{b}\mapsto t_{\mathbf{b}}\) is an isomorphism (defined below) from \((\mathbb{R}^3,+)\) to \(T(3)\). The same construction defines \(T(n)\) in \(n\) dimensions.
The general linear group#
The general linear group of dimension \(n\), denoted by \(GL(n,\mathbb{R})\), is the set of all invertible \(n \times n\) matrices. Mathematically,
The group operation is matrix multiplication. The identity element is the identity matrix \(I\), and the inverse of \(A\) is its matrix inverse \(A^{-1}\).
The special orthogonal group#
The special orthogonal group of dimension \(n\), denoted by \(SO(n)\), is the set of all \(n \times n\) orthogonal matrices with determinant one. Mathematically,
The group operation is matrix multiplication. The matrices in \(SO(n)\) represent orientation-preserving rotations in \(n\) dimensions. If \(A,B\in SO(n)\), then \(AB\) is orthogonal and \(\det(AB)=1\), so \(SO(n)\) is closed under multiplication.
\(SO(n)\) is a subgroup of \(GL(n,\mathbb{R})\). We write:
This means that \(SO(n)\) is a group in its own right, and it is a subset of \(GL(n,\mathbb{R})\).
Consider two examples in three-dimensional space. A rotation through angle \(\theta\) about the \(z\)-axis is represented by the matrix \(R_1\in SO(3)\):
Similarly, a rotation through angle \(\phi\) about the \(y\)-axis is represented by \(R_2\in SO(3)\):
Their product is
The product satisfies \(\det(R_3)=1\), as required by closure. In general, \(R_2R_1\ne R_1R_2\), so rotations in three dimensions do not commute.
The orthogonal group#
The orthogonal group of dimension \(n\), denoted by \(O(n)\), is the set of all \(n \times n\) orthogonal matrices. Mathematically,
The group operation is matrix multiplication. The matrices in \(O(n)\) are precisely the linear transformations that preserve Euclidean lengths and angles. They include rotations and reflections. The product of two orthogonal matrices is orthogonal, so \(O(n)\) is closed under multiplication.
\(O(n)\) is a subgroup of \(GL(n,\mathbb{R})\) and it contains \(SO(n)\) as a subgroup. We write:
An element of the orthogonal group that is not in the special orthogonal group is the reflection
It satisfies \(R_4R_4^{\mathsf T}=I\) and \(\det(R_4)=-1\).
Group homomorphisms#
A homomorphism is a map from one group to another that preserves multiplication. It may identify several elements of the source group, so it does not generally amount to relabeling the group elements.
Mathematically, let \(G\) and \(H\) be two groups. If we can find a function \(\phi: G \to H\) such that:
for all \(g_1, g_2 \in G\), then \(\phi\) is a homomorphism from \(G\) to \(H\).
If the homomorphism is bijective, i.e., one-to-one and onto, then it is called an isomorphism. When \(G\) and \(H\) are isomorphic, we write \(G \cong H\).
Example: A homomorphism from \(\mathbb{Z}\) to \(\mathbb{Z}_3\)#
For a homomorphism that is not an isomorphism, consider the group of integers modulo \(3\), denoted by \(\mathbb{Z}_3\). Write \([x]_3\) for the congruence class containing all integers with the same remainder as \(x\) after division by \(3\). The three classes are
The group operation is addition modulo \(3\); for example, \([1]_3+[2]_3=[0]_3\).
Now, consider the function
that sends an integer to its congruence class:
This is a homomorphism because
But it is not an isomorphism because it is not one-to-one.
Example: The group of real numbers with addition is isomorphic to the group of positive real numbers with multiplication#
Let \(\mathbb{R}^+\) be the set of positive real numbers. It is a group under multiplication, with identity \(1\) and inverse \(1/x\) for each \(x>0\).
Consider the function:
defined by:
The function is one-to-one and onto, and
It is therefore an isomorphism from \((\mathbb{R},+)\) to \((\mathbb{R}^+,\times)\).
Group of transformations#
Let \(V\) be a set. The set \(\operatorname{Bij}(V)\) of all bijections from \(V\) to itself is a group under function composition. A group of transformations of \(V\) is a subgroup
Equivalently, \(G\) is a collection of bijections that contains the identity map and is closed under composition and inverses. So, an element \(g\) of \(G\) is a function:
that is one-to-one and onto (bijective). Here the group operation is the composition of functions. So, if \(g_1\) and \(g_2\) are in \(G\), then the composition:
defined by:
is also in \(G\).
Function composition is associative:
The identity element is the identity function
and the inverse of each \(g\in G\) is its inverse bijection \(g^{-1}\in G\). Merely requiring a collection of bijections to be closed under composition would not be enough; the identity and inverse conditions are part of the subgroup definition.
Group actions#
A group can transform another set without itself being presented as a collection of transformations. An action of a group \(G\) on a set \(X\) assigns a map \(D_X(g):X\to X\) to every \(g\in G\) such that
Unlike a group of transformations, a general action may assign the same transformation to more than one group element. The action also does not require \(X\) to be a vector space or the maps \(D_X(g)\) to be linear.
Example: Permutations#
The permutation group \(S_N\) consists of all bijections \(\sigma:\{1,\ldots,N\}\to\{1,\ldots,N\}\) under composition. It acts on an ordered collection \(x=(x_1,\ldots,x_N)\) by relabeling its entries:
The inverse in this definition ensures that \(D_X(\sigma_1\sigma_2)=D_X(\sigma_1)\circ D_X(\sigma_2)\). For a collection of identical atoms, a predicted total energy should be invariant to this relabeling, while atom-indexed outputs should be permuted in the same way (Batzner et al., 2022).
Group representations#
Let \(V\) be a finite-dimensional real vector space, and let \(GL(V)\) denote the group of invertible linear maps from \(V\) to itself under composition. A representation of a group \(G\) on \(V\) is a homomorphism
Thus \(D(e)=I\) and
After choosing a basis of \(V\), each linear map \(D(g)\) is a matrix. A representation need not be injective; an injective representation is called faithful.
Example: Invertible linear maps and the general linear group#
Let \(V\) be a real vector space of dimension \(n\) and let \(GL(V)\) be the set of all invertible linear transformations of \(V\), i.e.,
\(GL(V)\) is a group under function composition.
Choosing a basis identifies this group with the matrix group \(GL(n,\mathbb{R})\). Let \(B = \{ \mathbf{e}_1, \ldots, \mathbf{e}_n \}\) be a basis of \(V\). Then, any linear transformation \(f \in GL(V)\) can be represented by a matrix \(A\) such that:
The matrix \(A\) is invertible because \(f\) is invertible.
The map
that sends a linear transformation to its matrix representation
is an isomorphism between \(GL(V)\) and \(GL(n,\mathbb{R})\):
The Euclidean group#
The Euclidean group \(E(n)\) collects the rigid transformations of \(\mathbb{R}^n\). Its elements are the isometries, meaning the maps \(f:\mathbb{R}^n\to\mathbb{R}^n\) that preserve Euclidean distance:
for all \(\mathbf{x},\mathbf{y}\in\mathbb{R}^n\). The group operation is function composition.
Every Euclidean isometry has a unique affine form
where \(A\in O(n)\) and \(\mathbf{b}\in\mathbb{R}^n\). To see why, set \(\mathbf{b}=f(\mathbf{0})\) and define \(q(\mathbf{x})=f(\mathbf{x})-\mathbf{b}\). Distance preservation gives
An inner-product-preserving map that fixes the origin is linear, so \(q(\mathbf{x})=A\mathbf{x}\) with \(A^{\mathsf T}A=I\).
We identify the isometry with the pair \((\mathbf{b},A)\). Its action on a point \(\mathbf{x}\in\mathbb{R}^n\) is
Composing two such actions gives the pair multiplication law
The term \(A_1\mathbf{b}_2\) shows that the orthogonal component acts on translation vectors. This interaction is the defining feature of a semidirect product.
Semidirect products#
Let \(G\) be a group with subgroups \(H\) and \(K\). Assume that \(H\) is normal in \(G\), that the two subgroups intersect only at the identity, and that every element of \(G\) is a product of an element of \(H\) and an element of \(K\):
Normality means that
for every \(g\in G\) and \(h\in H\). The factorization \(g=hk\) is unique. Indeed, if \(hk=h'k'\) with \(h,h'\in H\) and \(k,k'\in K\), then
so \(h=h'\) and \(k=k'\). Consequently, the map
is a bijection.
Normality also ensures that \(khk^{-1}\in H\) for each \(k\in K\) and \(h\in H\). Multiplication in \(G\) therefore gives
This identity determines the multiplication on \(H\times K\):
With this operation, \(H\times K\) is the semidirect product \(H\rtimes K\). The bijection \(\phi\) transports the group structure of \(G\) to \(H\rtimes K\), so it preserves multiplication and gives the isomorphism
Connection to the Euclidean group#
Under the pair notation for \(E(n)\), the translation \(t_{\mathbf{b}}\) is \((\mathbf{b},I)\) and the orthogonal map \(A\) is \((\mathbf{0},A)\). Every pair has the unique factorization
and \(T(n)\cap O(n)\) contains only the identity. Translations also form a normal subgroup: if \(g=(\mathbf{c},A)\in E(n)\), then
The orthogonal part therefore acts on translation vectors by \(\mathbf{b}\mapsto A\mathbf{b}\). The semidirect-product multiplication is exactly the pair law derived above, so
Transformation laws for physical quantities#
We use the active convention: a Euclidean transformation \(g=(\mathbf{b},A)\) moves a position according to
An equivalent passive change of coordinates is represented by the inverse group element. Other physical quantities transform according to their type. Under \(A\in O(n)\), a true scalar \(s\), a pseudoscalar \(p\), and a polar vector \(\mathbf{v}\) transform as
In three dimensions, an axial vector \(\mathbf{w}\) transforms as \(\mathbf{w}\mapsto\det(A)A\mathbf{w}\). Thus polar and axial vectors transform identically under rotations and acquire opposite parity under reflections. Velocity and force are polar vectors, whereas angular momentum is an axial vector.
A rank-two polar Cartesian tensor \(\mathbf{T}\) transforms as \(\mathbf{T}\mapsto A\mathbf{T}A^{\mathsf T}\). More generally, the components of a rank-\(k\) polar Cartesian tensor transform according to
A pseudotensor acquires an additional factor \(\det(A)\). Translations affect positions, while free vectors and tensors depend only on the orthogonal part \(A\). These rules define different linear representations of the same group, whereas the position rule is affine. A model whose input contains several types must apply the appropriate transformation to each component.
Invariance#
Let \(D_X(g)\) denote the action of \(G\) on an input space \(X\). A scalar-valued function \(f:X\to\mathbb{R}\) is invariant if
for every \(x\in X\) and \(g\in G\). For an isolated molecule in the absence of external fields, the potential energy should remain unchanged when all atomic positions undergo the same rigid transformation (Batzner et al., 2022).
For a finite group with \(|G|\) elements, let \(h_\theta:X\to\mathbb{R}\) be any scalar function with parameters \(\theta\). Averaging over the group produces the invariant function
Reindexing the sum by right multiplication shows that \(f_\theta\) is invariant. For a compact continuous group, the corresponding average is
where \(\mu\) is normalized Haar measure, the probability measure on \(G\) that is unchanged by left or right multiplication by a fixed group element.
Equivariance#
Let \(D_X\) and \(D_Y\) be actions of \(G\) on the input space \(X\) and output space \(Y\). A map \(f:X\to Y\) is equivariant if
for every \(x\in X\) and \(g\in G\). The two actions generally differ.
Invariance is the special case of equivariance in which the output action is trivial: \(D_Y(g)y=y\) for every \(g\in G\) and \(y\in Y\).
For a molecule with the position tuple
the Euclidean group acts on the input by
If \(f(\mathbf{x})=(\mathbf{F}_1,\ldots,\mathbf{F}_N)\) returns the atomic forces, the output action is
An equivariant force model therefore rotates or reflects its predicted forces with the molecule and leaves them unchanged under a common translation (Batzner et al., 2022).
Constructing equivariant neural networks#
Group-equivariant convolutional networks extend convolutional weight sharing from translations to larger symmetry groups (Cohen and Welling, 2016). Equivariance can be preserved layer by layer. The following closure rules provide the basic construction.
First, equivariant maps with the same output action are closed under addition. Let \(Y\) be a vector space on which \(G\) acts through the linear representation \(D_Y\). If \(f_1,f_2:X\to Y\) are equivariant, then
Second, equivariant maps are closed under compatible composition. If \(f:X\to Y\) is equivariant for the actions \(D_X,D_Y\) and \(h:Y\to Z\) is equivariant for \(D_Y,D_Z\), then
Third, tensor products combine feature types. Let \(V\) and \(W\) carry the linear representations \(D_V\) and \(D_W\). If \(a:X\to V\) and \(b:X\to W\) are equivariant, define their pointwise tensor product by \((a\otimes b)(x)=a(x)\otimes b(x)\). It satisfies
For polar vectors under \(O(n)\), the dot product is an invariant scalar and the outer product is a rank-two polar tensor. In three dimensions, the cross product is an axial vector: for \(A\in O(3)\),
The determinant factor is required when reflections are included.
Linear layers between feature types must also respect the group action. A linear map \(L:V\to W\) is equivariant precisely when it is an intertwiner:
for every \(g\in G\). An invariant scalar can multiply, or gate, an equivariant tensor without changing its type. Arbitrary componentwise nonlinearities do not generally preserve vector or tensor equivariance, so nonlinear layers must be assembled from type-preserving operations such as invariant gates and tensor products (Geiger and Smidt, 2022).
An equivariant network therefore tags every channel by its representation, uses intertwiners for linear mixing, and combines channels only through operations with known output types. An irreducible representation is a feature type with no nonzero proper subspace preserved by every group element. Euclidean neural networks organize their channels into these types and use spherical harmonics, angular basis functions with known rotation laws, to construct equivariant features (Geiger and Smidt, 2022).
Exercises#
Verify directly that the permutation rule
\[ \bigl(D_X(\sigma)x\bigr)_i=x_{\sigma^{-1}(i)} \]satisfies the identity and composition requirements of a group action. Explain why a total energy should be invariant to this action while atom-indexed forces should be equivariant.
Derive the inverse of \((\mathbf{b},A)\in E(n)\) and show that
\[ (\mathbf{b},A)^{-1} =\left(-A^{\mathsf T}\mathbf{b},A^{\mathsf T}\right). \]Use the result to verify that \(g=(\mathbf{b},A)\) satisfies \(g\,t_{\mathbf{c}}\,g^{-1}=t_{A\mathbf{c}}\).
Prove that the finite-group average \(f_\theta\) is invariant. Then use the transformation laws to classify the dot product, outer product, and three-dimensional cross product of two polar vectors.
From transformation laws to an equivariant model#
The companion notebook implements these constructions with the e3nn library (Geiger and Smidt, 2022). It represents features by irreducible \(O(3)\) types, forms angular features from spherical harmonics of relative positions, uses tensor products to build equivariant layers, and checks the resulting \(E(3)\) transformation laws numerically.