The Mutual Information In The Vicinity of Capacity-Achieving Input Distributions

Apr 27, 2023
17 pages
e-Print:

Citations per year

0 Citations
Abstract: (arXiv)
The mutual information is bounded from above by a decreasing affine function of the square of the distance between the input distribution and the set of all capacity-achieving input distributions ΠA\Pi_{\mathcal{A}}, on small enough neighborhoods of ΠA\Pi_{\mathcal{A}}, using an identity due to Topsøe and the Pinsker's inequality, assuming that the input set of the channel is finite and the constraint set A\mathcal{A} is polyhedral, i.e., can be described by (possibly multiple but) finitely many linear constraints. Counterexamples demonstrating nonexistence of such a quadratic bound are provided for the case of infinitely many linear constraints and the case of infinite input sets. Using Taylor's theorem with the remainder term, rather than the Pinsker's inequality and invoking Moreau's decomposition theorem the exact characterization of the slowest decrease of the mutual information with the distance to ΠA\Pi_{\mathcal{A}} is determined on small neighborhoods of ΠA\Pi_{\mathcal{A}}. Corresponding results for classical-quantum channels are established under separable output Hilbert space assumption for the quadratic bound and under finite-dimensional output Hilbert space assumption for the exact characterization. Implications of these observations for the channel coding problem and applications of the proof techniques to related problems are discussed.
Note:
  • 17 pages, presented at ISIT 2023, submitted to IEEE Transactions on Information Theory on June 13, 2023, revised on September 6, 2024
  • [1]

    A mathematical theory of communication

    • C.E. Shannon
  • [2]
    Information theory and reliable communication. New York, NY: John
    • R.G. Gallager
  • [3]
    Information theory: coding theorems for discrete memoryless systems. Cambridge, UK: Cambridge University Press
    • I. Csiszár
      ,
    • J. Körner
  • [4]

    Asymptotische abschätzungen in Shannons Informationstheorie

  • [5]

    Information spectrum approach to second-order coding rate in channel coding

    • M. Hayashi
      • IEEE Trans.Info.Theor. 55 (2009) 4947-4966
  • [6]

    Channel coding rate in the finite blocklength regime

    • Y. Polyanskiy
      ,
    • H.V. Poor
      ,
    • S. Verd
      • IEEE Trans.Info.Theor. 56 (2010) 2307-2359
  • [7]

    Channel coding: non-asymptotic fundamental limits

    • Y. Polyanskiy
  • [8]

    A tight upper bound for the third-order asymptotics for most discrete memoryless channels

    • M. Tomamichel
      ,
    • V.Y.F. Tan
      • IEEE Trans.Info.Theor. 59 (2013) 7041-7051
  • [9]
    Asymptotic Estimates in Information Theory with NonVanishing Error Probabilities. now Publishers Inc
    • V.Y.F. Tan
  • [10]

    Third-order analysis of channel coding in the small-to-moderate deviations regime

    • R.C. Yavas
      ,
    • V. Kostina
      ,
    • M. Effros
      • IEEE Trans.Info.Theor. 70 (2024) 6139-6170
  • [11]

    The log-volume of optimal codes for memoryless channels, asymptotically within a few nats

    • P. Moulin
      • IEEE Trans.Info.Theor. 63 (2017) 2278-2313
  • [12]

    Fixed-length lossy compression in the finite blocklength regime

    • V. Kostina
      ,
    • S. Verd
      • IEEE Trans.Info.Theor. 58 (2012) 3309-3338
  • [13]

    Lossy joint source-channel coding in the finite blocklength regime

      • IEEE Trans.Info.Theor. 59 (2013) 2545-2575
  • [14]

    Second-order rate region of constant-composition codes for the multiple-access channel

    • J. Scarlett
      ,
    • A. Martinez
      ,
    • A. Guillén i. Fàbregas
      • IEEE Trans.Info.Theor. 61 (2015) 157-172
  • [15]

    On the dispersions of the gel’fand-pinsker channel and dirty paper coding

    • J. Scarlett
      • IEEE Trans.Info.Theor. 61 (2015) 4569-4586
  • [16]

    Second-order asymptotics for the gaussian mac with degraded message sets

    • J. Scarlett
      ,
    • V.Y.F. Tan
      • IEEE Trans.Info.Theor. 61 (2015) 6700-6718
  • [17]

    On the quadratic decaying property of the information rate function

    • M.X. Cao
      ,
    • M. Tomamichel
  • [18]

    Comments on

    • [18]

      : On the quadratic decaying property of the information rate function

        • IEEE Trans.Info.Theor. 69 (2023) 5528-5531
    • [19]

      The mutual information in the vicinity of capacity-achieving input distributions

      • H.-C. Cheng
    • [20]
      Fundamentals of Convex Analysis, 1st ed., ser. Grundlehren Text Editions. Heidelberg:
      • J.-B. Hiriart-Urruty
        ,
      • C. Lemaréchal
    • [21]

      Information-type measures of difference of probability distributions and indirect observations

      • I. Csiszár
    • [22]

      Methods for quantifying rates of convergence for random walks on groups

      • F.E. Su
    • [23]

      On choosing and bounding probability metrics

      • A.L. Gibbs
        ,
      • F.E. Su
    • [24]
      Theory of statistical inference and information. Dordrecht:
      • I. Vajda