literature
Literature
Authors
Jobs
Seminars
Conferences
DataBETA

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Oct 22, 2020
e-Print:
View in:
pdf
reference search124 citations

Citations per year

2021202220232024202501020304050
Abstract: (submitter)
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.
  • Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In ACL
  • Philip
    • R. Bachman
  • Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In
    • ICLR
    • Attention augmented convolutional networks
      • I. Bello
        ,
      • B. Zoph
        ,
      • Q. Le
        ,
      • A. Vaswani
        ,
      • J. Shlens
      • In ICCV
      • Lucas Beyer
        • Olivier J.
      • Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al
        • Tom B. Brown
      • Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and
        • Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV
        • Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels. In ICML,a
        • Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML,b
        • Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and
          • Jingjing Liu. UNITER: UNiversal Image-TExt Representation Learning. In ECCV,c
          • Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv
          • Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between selfattention and convolutional layers. In ICLR
          • Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR
            • J. Deng
              ,
            • W. Dong
              ,
            • R. Socher
              ,
            • L. Li
          • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL
          • Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander
            • Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, Sylvan
              • Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On robustness and transferability of convolutional neural networks. arXiv
              • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR
              • Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR
              • Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv
              • Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In CVPR