nonhermitian

Benchpress

2 min read

Benchpress

Collaborators

Abdullah Ash Saki, Sebastian Brandhofer, Luciano Bello, Shelly Garion, Matthew Treinish , and Ali Javadi

Summary

Benchpress is a framework for evaluating the performance of quantum computing software in the areas of quantum circuit construction, manipulation, and optimization (compilation). We created a test suite of over 1000 tests, both from ourselves and other open-source tools, and ran them over 7 different software frameworks to determine which has the best overall performance in terms of output circuit quality (number of 2Q gates and depth) and run time (wall-clock time).

In short, Qiskit was the winner in terms of both quality and run time by a fair amount, followed by Tket from Quantinuum. However, since our initial work, qpanda3 from OriginQC has become the closest competitor to Qiskit.

Motivation

Several groups (including ourselves) were claiming that their SDK was the most performant choice. However, no one really bothered to do a meaningful head-to-head comparison. Such benchmarks are complicated by the fact that there are disparate feature sets amongst SDKs, and some tests require re-coding in each language. We later found that many SDKs also fail tests, sometimes in ways that are difficult to capture, e.g. a test that seemingly runs forever, or tests that eat all available memory. With Benchpress we tackled these issues head on, and created a testing framework that allows for (almost) straightforward testing of quantum software.

Possible extensions

Currently, Benchpress does not test SDKs that use functions to represent quantum circuits. Examples include cudaq from Nvidia and PennyLane from Xanadu. Unlike SDKs that represent quantum circuits as objects, it is not straightforward to extract the needed data from these workflows.

Graph View

Backlinks