Benchmarking Quantum Computers: Towards a Standard Performance Evaluation Approach

Jul 15, 2024
24 pages
e-Print:

Citations per year

20232024202501
Abstract: (arXiv)
The technological development of increasingly larger quantum processors on different quantum platforms raises the problem of how to fairly compare their performance, known as quantum benchmarking of quantum processors. This is a challenge that computer scientists have already faced when comparing classical processors, leading to the development of various mathematical tools to address it, but also to the identification of the limits of this problem. In this work, we briefly review the most important aspects of both classical processor benchmarks and the metrics comprising them, providing precise definitions and analyzing the quality attributes that they should exhibit. Subsequently, we analyze the intrinsic properties that characterize the paradigm of quantum computing and hinder the naive transfer of strategies from classical benchmarking. However, we can still leverage some of the lessons learned such as the quality attributes of a \textit{good} benchmark. Additionally, we review some of the most important metrics and benchmarks for quantum processors proposed in the literature, assessing what quality attributes they fulfill. Finally, we propose general guidelines for quantum benchmarking. These guidelines aim to pave the way for establishing a roadmap towards standardizing the performance evaluation of quantum devices, ultimately leading to the creation of an organization akin to the Standard Performance Evaluation Corporation (SPEC).
Note:
  • 24 pages, 4 figures
  • [1]

    Computer benchmarking: Paths and pitfalls: The most popular way of rating computer performance can confuse as well as inform; avoid misunderstanding by asking just what the benchmark is measuring

    • J. Dongarra
      ,
    • J.L. Martin
      ,
    • J. Worlton
      • IEEE Spectrum 24 (1987) 38-43
  • [2]

    How not to lie with statistics: the correct way to summarize benchmark results

    • P.J. Fleming
      ,
    • J.J. Wallace
  • [3]

    Notes on Calculating Computer Performance

  • [4]

    Characterizing Computer Performance with a Single Number

    • J. Smith
  • [5]

    How to Build a Benchmark

    • J. Kistowski
      ,
    • J.A. Arnold
  • [6]

    Performance evaluation and benchmarking with realistic applications

  • [7]

    Measuring Computer Performance

    • D.J. Lilja
  • [8]

    The art of computer systems performance analysis

    • R. Jain
  • [9]

    Systems benchmarking

    • S. Kounev
      ,
    • K.-D. Lange
      ,
    • J. von Kistowski
  • [10]

    Performance Evaluation of Computer and Communication Systems

    • J. Le Boudec
  • [11]

    Workload modeling for computer systems performance evaluation

    • D.G. Feitelson
  • [12]

    Using Benchmarking to Advance Research: A Challenge to Software Engineering

    • S.E. Sim
      ,
    • S. Easterbrook
      ,
    • R.C. Holt
  • [13]

    Problems of Monetary Management: The U.K. Experience

    • C. Goodhart
    • [14]

      Challenges in computer architecture evaluation

      • K. Skadron
        ,
      • M. Martonosi
        ,
      • D.I. August
        • Computer 36 (2003) 30-36
    • [15]

      Not All Benchmarks Are Created Equal

      • R. Blume-Kohout
        ,
      • E. Nielsen
        ,
      • T.Proctor
    • [16]

      Metrics & Benchmarks for Digital Quantum Computing

      • R. Blume-Kohout
    • [17]

      Performance Standards for Quantum Computing

      • R. Blume-Kohout
    • [18]

      Quantum Computing Metrics and Benchmarking Frameworks: Not Yet

      • R. Blume-Kohout
        ,
      • K. Young
    • [19]

      Quantum Computing Summit

      • L. Gomes
    • [20]

      Standard for Quantum Computing Performance Metrics & Performance Benchmarking

      • [20]
        standards association
      • [22]

        Metrics and Benchmarks for Quantum Processors: State of Play

        • R. Blume-Kohout
          ,
        • K. Young
      • [24]
        We focus in this article on the study of benchmarking related to physical devices or processors, even though interesting debates are held by the community in other benchmarking fields such as algorithms benchmarking [151, 152]