Benchmarking a quantum circuit is more than recording its runtime or counting qubits. A useful benchmark connects the circuit you wrote to the circuit actually executed, measures the quality of its output, and records enough context to reproduce the comparison. This checklist explains how to compare circuit versions, simulators, and hardware backends using depth, gate counts, execution details, sampling uncertainty, fidelity proxies, and reproducibility checks.
Overview
A quantum circuit benchmark should answer a specific question. Are you trying to reduce compilation overhead? Compare two hardware targets? Determine whether an optimization pass preserves the algorithm's output? Estimate whether hardware noise overwhelms the signal? The right metrics depend on that goal.
Start by separating three layers of performance:
- Program-level cost: the logical qubit count, circuit depth, number of operations, and number of two-qubit gates in the circuit you authored.
- Execution-level cost: the transpiled or compiled circuit, mapped qubits, inserted routing operations, queue or submission time, device execution time, and shot count.
- Result quality: agreement with an exact solution, noiseless reference, ideal probability distribution, or another accepted baseline.
Do not treat one number as a universal score. A circuit with fewer logical gates can become larger after mapping to a device's connectivity graph. A fast simulator can produce an exact result while saying little about hardware noise. A hardware run can have a short execution window but require many repeated submissions to achieve a stable estimate.
For every benchmark, record the circuit identifier, algorithm or workload, logical qubit count, input data, parameter values, measurement layout, compiler settings, target backend, shot count, software versions, and timestamp. If any of these change between runs, describe the comparison as a new experiment rather than a direct repeat.
Checklist by scenario
Comparing two circuit versions
- Run both versions on the same input, with the same parameter values and measurement definition.
- Count logical operations by type, giving special attention to two-qubit gates because they often contribute substantially to depth, routing overhead, and noise exposure.
- Record logical depth and width before compilation.
- Compile both versions for the same target basis and connectivity, using the same optimization configuration where possible.
- Compare transpiled depth, total operation count, two-qubit operation count, and any added swaps or other routing operations.
- Check result quality against the same reference. A smaller circuit is not an improvement if its output changes beyond the tolerance required by the application.
Keep a small experiment table with one row per version. Useful columns include circuit revision, compiler configuration, pre-compilation depth, post-compilation depth, post-compilation two-qubit count, shots, runtime fields, quality metric, and notes.
Comparing a simulator with quantum hardware
- Use a noiseless simulator first to verify that the circuit and measurement interpretation are correct.
- Run a noisy simulation only if its noise model and assumptions are documented. Treat it as a model-based comparison, not a substitute for device data.
- Submit the identical compiled workload to hardware when the platform permits that level of control.
- Use the same shot count, or report clearly when shot counts differ.
- Compare distributions rather than only the most frequent bitstring. Useful measures include total variation distance, Hellinger distance, cross-entropy-style scores, or application-specific loss, provided the definition is stated.
- Repeat the hardware run when possible and report variation across repetitions, not just the best result.
The simulator may expose exact state probabilities, while hardware returns samples. To make the comparison fair, sample the ideal simulator with the same number of shots or compare hardware samples with an explicitly defined distribution metric.
Comparing hardware backends
- Confirm that each backend can support the circuit's qubit count, gate set, measurement pattern, and connectivity requirements.
- Compile separately for each target. Never compare one backend's native circuit with another backend's uncompiled logical circuit.
- Record mapped qubits and the resulting depth and two-qubit count for each target.
- Keep workload, shots, parameter values, and result-processing code constant.
- Record execution-related fields separately from queue, reservation, or submission overhead when the platform exposes them.
- Compare quality and variability alongside time. The fastest backend is not necessarily the most useful if it produces a less reliable result.
What to double-check
Metric definitions
Depth is the length of the longest sequence of operations under the chosen scheduling or layering convention. It can differ before and after compilation. Gate count should identify which operations are included; a total count that combines one-qubit gates, two-qubit gates, barriers, measurements, and administrative instructions may obscure the real cost. Two-qubit count should be reported explicitly and, where useful, separated into algorithmic gates and compiler-added routing operations.
Execution time is ambiguous. Distinguish local circuit-construction time, compilation time, submission overhead, queue delay, device execution time, result retrieval time, and total wall-clock time. Use the fields provided by the platform, but document their meaning because reporting APIs and terminology can change.
Sampling error
Measured probabilities are estimates from a finite number of shots. For a bitstring with observed frequency p from N shots, a rough standard error is proportional to √(p(1−p)/N). Rare outcomes can therefore have substantial relative uncertainty even when the total shot count appears large. Report shots and, for important outcomes, confidence intervals or repeated-run variation.
Fidelity proxies
When the exact target state is unavailable, use a clearly labeled proxy: success probability, expectation-value error, overlap with a trusted simulator, distribution distance, or application loss. These are not interchangeable. A circuit can preserve an expectation value while changing the full distribution, or match the most likely bitstring while performing poorly on less common outcomes.
Reproducibility
Save the source circuit, compiled circuit or compilation summary, backend identifier, selected qubits if applicable, compiler options, random seeds when supported, shot count, parameter values, raw counts, and result-processing script. Also record whether calibration, noise, or backend configuration information was available at the time of execution. Hardware conditions may vary, so reproducibility means preserving context, not promising identical samples.
Common mistakes
- Comparing uncompiled circuits: logical depth does not reveal routing overhead on a constrained device.
- Using different workloads: changing parameters, initial states, measurements, or post-processing can invalidate a performance comparison.
- Reporting only the best run: this hides run-to-run variation and can make a noisy result appear more stable than it is.
- Mixing timing categories: queue delay and device execution time answer different operational questions.
- Ignoring readout effects: measured output errors can arise from state preparation, gates, measurement, or classical post-processing.
- Overinterpreting simulator results: an ideal simulator tests circuit logic, not physical hardware performance.
- Changing optimization settings silently: compiler level, layout choice, routing method, and seed can materially affect the compiled circuit.
- Calling a proxy fidelity: use the precise metric name unless the quantity is actually a validated fidelity estimate.
For debugging, establish a baseline ladder: a small hand-checkable circuit, an ideal simulation, a sampled noiseless simulation, a noisy simulation if justified, and finally hardware. This makes it easier to identify whether a discrepancy comes from circuit construction, compilation, sampling, or the device.
When to revisit
Revisit a benchmark whenever the circuit, compiler, SDK, backend, qubit mapping, shot policy, or result-processing code changes. A platform update may alter available gates, transpilation behavior, timing fields, backend constraints, or measurement reporting. Treat those changes as reasons to rerun the benchmark rather than assuming old numbers remain comparable.
Before a planning or review cycle, refresh the benchmark in a controlled way:
- Freeze the workload and define the question the benchmark must answer.
- Recompile every candidate for the current target configuration.
- Run the ideal reference and hardware or noisy experiments with documented shots.
- Collect raw results and calculate the same quality metrics for every candidate.
- Report medians or distributions when repeated runs are available, alongside the full experimental context.
- Archive the benchmark record so a future result can be compared with the correct version, not merely with a copied headline number.
A practical benchmark is a repeatable experiment, not a single score. If you consistently track compiled circuit structure, execution categories, sampling uncertainty, output quality, and reproducibility metadata, you can make meaningful comparisons as circuits and quantum platforms evolve.