and How to Use them in e3nn to Build Euclidean Neural Networks
Mario Geiger
MRS Fall Meeting Tutorial
This talk is about
intro
polynomials
examples & code
nonlinearities
Polynomial part
Examples and Code
Nonlinearities
intro
polynomials
examples & code
nonlinearities
intro
polynomials
examples & code
nonlinearities
intro
polynomials
examples & code
nonlinearities
"what are the operations"
"how they compose"
intro
polynomials
examples & code
nonlinearities
"what are the operations"
"how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
"what are the operations"
"how they compose"
"vector spaces on which the action of the group is defined"
rotations, parity, (translations)
scalars, vectors, pseudovectors, ...
intro
polynomials
examples & code
nonlinearities
Group G
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
Equivalent notation D(g)x
⚠️ We use this notation in the code
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
Equivalent notation D(g)x
⚠️ We use this notation in the code
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
Equivalent notation D(g)x
⚠️ We use this notation in the code
intro
polynomials
examples & code
nonlinearities
Group G
Representation D(g,x)
"what are the operations" "how they compose"
"vector spaces on which the action of the group is defined"
Equivalent notation D(g)x
⚠️ We use this notation in the code
intro
polynomials
examples & code
nonlinearities
a1a2a3a4a5a6a7a8a9∈R9
intro
polynomials
examples & code
nonlinearities
3 scalars
Representations are like data types
It tells you how to interpret the data...
a1a2a3a4a5a6a7a8a9∈R9
intro
polynomials
examples & code
nonlinearities
3 scalars
Representations are like data types
It tells you how to interpret the data with respect to the G!
a1a2a3a4a5a6a7a8a9∈R9
intro
polynomials
examples & code
nonlinearities
3 scalars
Representations are like data types
It tells you how to interpret the data with respect to the G!
a1a2a3a4a5a6a7a8a9∈R9
Knowing that a1,a2,a3 are scalars tells you that they are not affected by a rotation of your system
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
Representations are like data types
It tells you how to interpret the data with respect to the G!
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
If the system is rotated, the 3 components of the vector change together!
Representations are like data types
It tells you how to interpret the data with respect to the G!
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
The two vectors transforms indepentently
Representations are like data types
It tells you how to interpret the data with respect to the G!
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
Representations are like data types
It tells you how to interpret the data with respect to the G!
system rotated by g
D(g)a1a2a3a4a5a6a7a8a9
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
Representations are like data types
It tells you how to interpret the data with respect to the G!
system rotated by g
a1a2a3a4a5a6a7a8a9
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
Representations are like data types
It tells you how to interpret the data with respect to the G!
system rotated by g
a1a2a3a4a5a6a7a8a9
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
Representations are like data types
It tells you how to interpret the data with respect to the G!
system rotated by g
a1a2a3a4a5a6a7a8a9
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
Representations are like data types
It tells you how to interpret the data with respect to the G!
system rotated by g
a1a2a3a4a5a6a7a8a9
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
system rotated by g
a1a2a3a4a5a6a7a8a9
e3nn notation:
irreps = "3x0e + 2x1o"
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
system rotated by g
a1a2a3a4a5a6a7a8a9
e3nn notation:
from e3nn import o3
irreps = o3.Irreps("3x0e + 2x1o")
intro
polynomials
examples & code
nonlinearities
3 scalars
a vector
a1a2a3a4a5a6a7a8a9∈R9
a vector
system rotated by g
a1a2a3a4a5a6a7a8a9
from e3nn import o3
irreps = o3.Irreps("3x0e + 2x1o")
import torch
alpha, beta, gamma = torch.randn(3)
irreps.D_from_angles(alpha, beta, gamma)
intro
polynomials
examples & code
nonlinearities
a1a2a3a4a5a6a7a8a9
system rotated by g
a1a2a3a4a5a6a7a8a9
array([[ 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ], [ 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ], [ 0. , 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. ], [ 0. , 0. , 0. , 0.77, 0.22, -0.6 , 0. , 0. , 0. ], [ 0. , 0. , 0. , -0.1 , 0.97, 0.23, 0. , 0. , 0. ], [ 0. , 0. , 0. , 0.63, -0.12, 0.76, 0. , 0. , 0. ], [ 0. , 0. , 0. , 0. , 0. , 0. , 0.77, 0.22, -0.6 ], [ 0. , 0. , 0. , 0. , 0. , 0. , -0.1 , 0.97, 0.23], [ 0. , 0. , 0. , 0. , 0. , 0. , 0.63, -0.12, 0.76]], dtype=float32)
from e3nn import o3
irreps = o3.Irreps("3x0e + 2x1o")
import torch
alpha, beta, gamma = torch.randn(3)
irreps.D_from_angles(alpha, beta, gamma)
intro
polynomials
examples & code
nonlinearities
V
V′
intro
polynomials
examples & code
nonlinearities
V
V′
D(g)
D′(g)
V
V′
intro
polynomials
examples & code
nonlinearities
V
V′
D(g)
D′(g)
V
V′
f
intro
polynomials
examples & code
nonlinearities
V
V′
D(g)
D′(g)
V
V′
f
f
intro
polynomials
examples & code
nonlinearities
V
V′
D(g)
D′(g)
V
V′
f
f
f(D(g)x)
intro
polynomials
examples & code
nonlinearities
V
V′
D(g)
D′(g)
V
V′
f
f
f(D(g)x)
D′(g)f(x)
intro
polynomials
examples & code
nonlinearities
V
V′
D(g)
D′(g)
V
V′
f
f
f(D(g)x)
D′(g)f(x)
=
intro
polynomials
examples & code
nonlinearities
Most simple yet powerful functions
intro
polynomials
examples & code
nonlinearities
Most simple yet powerful functions
Be able to create polynomials compatible with rotations is a powerful tool
intro
polynomials
examples & code
nonlinearities
x↦x2+2(x−4)
intro
polynomials
examples & code
nonlinearities
x↦x2+2(x−4)
xyz↦x2+2(y−z)x
intro
polynomials
examples & code
nonlinearities
x↦x2+2(x−4)
xyz↦x2+2(y−z)x
xyz↦[x2+2(y−z)z4+100xyz]
intro
polynomials
examples & code
nonlinearities
P(D(g)x)=D′(g)P(x)
xyz↦[x2+2(y−z)z4+100xyz]
😟
Not equivariant
xyz↦[x2+y2+z2]
😌
Equivariant
intro
polynomials
examples & code
nonlinearities
equivariant 😊
intro
polynomials
examples & code
nonlinearities
two equivariant functions
f:V1→V2
h:V2→V3
h∘f is equivariant!! 😊
h(f(D1(g)x))=h(D2(g)f(x))=D3(g)h(f(x))
intro
polynomials
examples & code
nonlinearities
two equivariant functions
f:V1→V3
h:V2→V3
h+f is equivariant!! 😊
f(D1(g)x)+h(D2(g)x)=D3(g)(f(x)+h(x))
intro
polynomials
examples & code
nonlinearities
x1x2x3
y1y2y3y4y5
transforming with D(g)
transforming with D′(g)
=x1y1x2y1x3y1x1y2x2y2x3y2x1y3x2y3x3y3x1y4x2y4x3y4x1y5x2y5x3y5
intro
polynomials
examples & code
nonlinearities
⊗
=x1y1x2y1x3y1x1y2x2y2x3y2x1y3x2y3x3y3x1y4x2y4x3y4x1y5x2y5x3y5
Tensor Product
transforms with D(g)⊗D′(g) 😊
dim(D⊗D′)=dim(D)dim(D′)
x1x2x3
y1y2y3y4y5
intro
polynomials
examples & code
nonlinearities
vector⊗vector⊗vector⊗…
tensors of shape 3×3×3×…
Cartesian Tensors
The grow in dimension is problematic
Luckily the tensor product is reducible
intro
polynomials
examples & code
nonlinearities
D defined on V
is reducible if
∃W⊂V
such that
D∣W is a representation
intro
polynomials
examples & code
conclusion
intro
polynomials
examples & code
nonlinearities
x1x2y1x2z1x2x1y2y1y2z1y2x1z2y1z2z1z2
D defined on V
is reducible if
∃W⊂V
such that
D∣W is a representation
x1y1z1⊗x2y2z2=
intro
polynomials
examples & code
nonlinearities
x1x2+y1y2+z1z2
c(x1z2+z1x2)c(x1y2+y1x2)2y1y2−x1x2−z1z2c(y1z2+z1y2)c(z1z2−x1x2)
y1z2−z1y2z1x2−x1z2x1y2−y1x2
x1x2y1x2z1x2x1y2y1y2z1y2x1z2y1z2z1z2
3×3=1+3+5
These can be seen as polynomials!
D is reducible if
∃W⊂V
such that
D∣W is a representation
intro
polynomials
examples & code
nonlinearities
For the group of rotations (SO(3))
They are index by L=0,1,2,…
Of dimension 2L+1
L=0 | d=1 | scalar |
L=1 | d=3 | vector |
L=2 | d=5 | no name? |
intro
polynomials
examples & code
nonlinearities
For the group of rotations + parity (O(3))
They are index by L=0,1,2,…
Of dimension 2L+1
L=0 | d=1 | scalar |
L=1 | d=3 | peudo vector |
L=2 | d=5 | no name? |
L=0 | d=1 | pseudo scalar |
L=1 | d=3 | vector |
L=2 | d=5 | no name? |
Even
Odd
intro
polynomials
examples & code
nonlinearities
L1⊗L2=∣L1−L2∣⊕⋯⊕(L1+L2)
generalization of 3×3=1+3+5
intro
polynomials
examples & code
nonlinearities
Composition of equivariant polynomials ✅
Addition of equivariant polynomials ✅
Tensor Product of equivariant polynomials ✅
L1⊗L2=∣L1−L2∣⊕⋯⊕(L1+L2)
p2(p1(x, y), z)
p1(x, y) + p2(w, z)
from e3nn import o3
p = o3.FullTensorProduct("1o", "1o")
p(x, y)
intro
polynomials
examples & code
nonlinearities
Composition of equivariant polynomials ✅
Addition of equivariant polynomials ✅
Tensor Product of equivariant polynomials ✅
L1⊗L2=∣L1−L2∣⊕⋯⊕(L1+L2)
p2(p1(x, y), z)
p1(x, y) + p2(w, z)
from e3nn import o3
p = o3.FullTensorProduct("1o", "1o")
p(x, y)
intro
polynomials
examples & code
nonlinearities
Representation of highest order (l) in x⊗x⊗…
https://e3nn.org/mrs
intro
polynomials
examples & code
nonlinearities
3×3=1+3+5
x1x2y1x2z1x2x1y2y1y2z1y2x1z2y1z2z1z2
x1x2+y1y2+z1z2
y1z2−z1y2z1x2−x1z2x1y2−y1x2
c(x1z2+z1x2)c(x1y2+y1x2)2y1y2−x1x2−z1z2c(y1z2+z1y2)c(z1z2−x1x2)
x1y1z1⊗x2y2z2=
"1o" time "1o"
"1o"
"1o"
"0e"
"1e"
"2e"
https://e3nn.org/mrs
intro
polynomials
examples & code
nonlinearities
first argument
second argument
output
3×3=1+3+5
https://e3nn.org/mrs
intro
polynomials
examples & code
nonlinearities
3 vectors times 3 vectors given 3 pseudovectors
each output is the weight sum of 9 pseudovectors
This operation contains 27 parameters
https://e3nn.org/mrs
intro
polynomials
examples & code
nonlinearities
Consider for instance the inertia tensor
Iij=∫dxρ(x)(∥x∥2δij−xixj)
It is a vector⊗vector symmetric tensor (Iij=Iji)
It can be decomposed into a scalar and a l=2 irrep
https://e3nn.org/mrs
intro
polynomials
examples & code
nonlinearities
intro
polynomials
examples & code
nonlinearities
ϕ(P(x))Q(x)
invariant polynomial
equivariant polynomial
sigmoid like function
https://e3nn.org/mrs
Thank you for listening!
https://e3nn.org/mrs