Equivariant transformer is all you need

Machine learning, deep learning, has been accelerating computational physics, which has been used to simulate systems on a lattice. Equivariance is essential to simulate a physical system because it imposes a strong induction bias for the probability distribution described by a machine learning model. This reduces the risk of erroneous extrapolation that deviates from data symmetries and physical laws.However, imposing symmetry on the model sometimes occur a poor acceptance rate in self-learning Monte-Carlo (SLMC). On the other hand, Attention used in Transformers like GPT realizes a large model capacity. We introduce symmetry equivariant attention to SLMC. To evaluate our architecture, we apply it to our proposed new architecture on a spin-fermion model on a two-dimensional lattice. We find that it overcomes poor acceptance rates for linear models and observe the scaling law of the acceptance rate as in the large language models with Transformers.

Note:

7 pages, 4 figures, contribution for the 40th International Symposium on Lattice Field Theory (Lattice 2023), July 31st - August 4th, 2023, Fermi National Accelerator Laboratory

model: linear
dimension: 2
acceptance
machine learning
lattice
Monte Carlo
numerical calculations
scaling
induction

References(16)

Figures(6)

[1]

Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics

Kyle Cranmer
(
- Wisconsin U., Madison
)
,
Gurtej Kanwar
(
- Bern U.
)
,
Sébastien Racanière
(
- Unlisted, UK
)
,
Danilo J. Rezende
(
- Unlisted, UK
)
,
Phiala E. Shanahan
(
- MIT, Cambridge, CTP and
- IAIFI, Cambridge
)

- Nature Rev.Phys. 5 (2023) 9, 526-535
•
e-Print:
- 2309.01156
•
DOI:
- 10.1038/s42254-023-00616-w

[2]

Yuki Nagai and Akio Tomiya. Gauge covariant neural network for 4 dimensional non-abelian gauge theory. March

[3]

Highly accurate protein structure prediction with alphafold

J. Jumper
,
R. Evans
,
A. Pritzel

- Nature 596 (2021) 583-589

[4]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale

[5]

Philipp Thölke and Gianni De Fabritiis. TorchMD-NET: Equivariant transformers for neural network based molecular potentials. February

[6]

Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger

Jonathan P. Mailoa

- Nature Commun. 13 (2022) 2453

[7]

Masanobu Horie. E(n)-Equivariant Graph Neural Networks Emulating Mesh-Discretized Physics. Doctor of philosophy in engineering, University of Tsukuba, March

[8]

Yuki Nagai and Akio Tomiya. Self-learning monte carlo with equivariant transformer

[9]

Junwei Liu, Yang Qi, Zi Yang Meng, and Liang Fu. Self-learning monte carlo method. Physical

Review B

[10]

Physics-informed machine learning

G.E. Karniadakis
,
I.G. Kevrekidis
,
L. Lu

- Nature Rev.Phys. 3 (2021) 422-440

[11]

Efficient Langevin simulation of coupled classical fields and fermions

- Phys.Rev.B 88 (2013) 23, 235101
•
DOI:
- 10.1103/physrevb.88.235101

[12]

Self-learning Monte Carlo method and cumulative update in fermion systems

- Phys.Rev.B 95 (2017) 24, 241104
•
e-Print:
- 1611.09364
•
DOI:
- 10.1103/PhysRevB.95.241104

[13]

Georgios Stratis, Phillip Weinberg, Tales Imbiriba, Pau Closas, and Adrian E Feiguin. Sample generation for the spin-fermion model using neural networks

- Phys.Rev.B 106 (2022) 205112

[14]

Hidehiko Kohshiro and Yuki Nagai. Effective Ruderman-Kittel-Kasuya-Yosida-like interaction in diluted double-exchange model: Self-learning monte carlo approach

- J.Phys.Soc.Jap. 90 (2021) 034711

[15]

Self-learning Monte Carlo with deep neural networks

- Phys.Rev.B 97 (2018) 20, 205140
•
DOI:
- 10.1103/physrevb.97.205140

[16]

Jared Kaplan, Sam McCandlish, Tom Henighan

Tom B. Brown