Improved Training of Wasserstein GANs

Mar 31, 2017
e-Print:

Citations per year

20172019202120232024051015
Abstract: (submitter)
Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.
  • [1]
    Towards principled methods for training generative adversarial networks
    • M. Arjovsky
      ,
    • L. Bottou
  • [4]
    Began: Boundary equilibrium generative adversarial networks. arXiv preprint
    • D. Berthelot
      ,
    • T. Schumm
      ,
    • L. Metz
  • [5]
    Maximum-likelihood augmented discrete generative adversarial networks. arXiv preprint
    • T. Che
      ,
    • Y. Li
      ,
    • R. Zhang
      ,
    • R.D. Hjelm
      ,
    • W. Li
    et al.
  • [6]
    One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint
    • C. Chelba
      ,
    • T. Mikolov
      ,
    • M. Schuster
      ,
    • Q. Ge
      ,
    • T. Brants
    et al.
  • [7]
    Calibrating energy-based generative adversarial networks. arXiv preprint
    • Z. Dai
      ,
    • A. Almahairi
      ,
    • P. Bachman
      ,
    • E. Hovy
      ,
    • A. Courville
  • [8]
    Adversarially learned inference
    • V. Dumoulin
      ,
    • M.I.D. Belghazi
      ,
    • B. Poole
      ,
    • A. Lamb
      ,
    • M. Arjovsky
    et al.
  • [9]
    Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680
    • I. Goodfellow
      ,
    • J. Pouget-Abadie
      ,
    • M. Mirza
      ,
    • B. Xu
      ,
    • D. Warde-Farley
    et al.
  • [10]
    Boundary-seeking generative adversarial networks. arXiv preprint
    • R.D. Hjelm
      ,
    • A.P. Jacob
      ,
    • T. Che
      ,
    • K. Cho
      ,
    • Y. Bengio
  • [11]
    Stacked generative adversarial networks. arXiv preprint
    • X. Huang
      ,
    • Y. Li
      ,
    • O. Poursaeed
      ,
    • J. Hopcroft
      ,
    • S. Belongie
  • [12]
    Categorical reparameterization with gumbel-softmax. arXiv preprint
    • E. Jang
      ,
    • S. Gu
      ,
    • B. Poole
  • [13]
    Learning multiple layers of features from tiny images
    • A. Krizhevsky
  • [14]
    Adversarial learning for neural dialogue generation. arXiv preprint
    • J. Li
      ,
    • W. Monroe
      ,
    • T. Shi
      ,
    • A. Ritter
      ,
    • D. Jurafsky
  • [15]
    Recurrent topic-transition gan for visual paragraph generation. arXiv preprint
    • X. Liang
      ,
    • Z. Hu
      ,
    • H. Zhang
      ,
    • C. Gan
      ,
    • E.P. Xing
  • [16]
    Approximation and convergence properties of generative adversarial learning. arXiv preprint
    • S. Liu
      ,
    • O. Bousquet
      ,
    • K. Chaudhuri
  • [17]
    The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint
    • C.J. Maddison
      ,
    • A. Mnih
      ,
    • Y.W. Teh
  • [18]
    Least squares generative adversarial networks. arXiv preprint
    • X. Mao
      ,
    • Q. Li
      ,
    • H. Xie
      ,
    • R.Y. Lau
      ,
    • Z. Wang
  • [19]
    Unrolled generative adversarial networks. arXiv preprint
    • L. Metz
      ,
    • B. Poole
      ,
    • D. Pfau
      ,
    • J. Sohl-Dickstein
  • [20]
    Conditional image synthesis with auxiliary classifier gans. arXiv preprint
    • A. Odena
      ,
    • C. Olah
      ,
    • J. Shlens
  • [21]
    Improved generator objectives for gans. arXiv preprint
    • B. Poole
      ,
    • A.A. Alemi
      ,
    • J. Sohl-Dickstein
      ,
    • A. Angelova
  • [23]
    Improved techniques for training gans. In Advances in Neural Information Processing Systems, pages 2226-2234
    • T. Salimans
      ,
    • I. Goodfellow
      ,
    • W. Zaremba
      ,
    • V. Cheung
      ,
    • A. Radford
    et al.
  • [24]
    A. van
    • den Oord
      ,
    • N. Kalchbrenner
      ,
    • L. Espeholt
      ,
    • A. O. Vinyals
  • [25]
    Optimal transport: old and new, volume 338
    • C. Villani