A quantum machine-learning model is not useful merely because it runs on a quantum circuit. To evaluate it fairly, compare it with strong classical baselines, measure more than accuracy, and report enough detail for another developer to reproduce the result. This guide provides a practical quantum ML evaluation checklist, a repeatable experiment template, and worked examples for comparing models across simulators and hardware.
Overview
The central question in quantum machine-learning evaluation is not “Did the circuit train?” It is “Did the quantum component produce a meaningful improvement under the same data, compute, and reporting conditions?” That improvement might be predictive performance, calibration, parameter efficiency, robustness to noise, or a useful tradeoff between quality and training cost.
A credible comparison normally includes four model groups:
- A simple baseline: for example, a majority-class predictor, linear or logistic regression, or a small decision tree.
- A strong classical baseline: a carefully tuned model appropriate for the dataset, such as a regularized kernel method, gradient-boosted model, or multilayer neural network.
- A quantum model: such as a variational quantum classifier, quantum kernel method, or hybrid model with a parameterized circuit.
- A controlled ablation: a model that removes or changes the quantum component, encoding method, ansatz, measurement strategy, or feature map.
This structure helps separate a genuine modeling effect from a data-preparation advantage, an unusually favorable split, or an implementation detail. For background on encoding choices, see quantum data encoding methods compared. If your experiment includes a variational circuit, the end-to-end VQE workflow offers a useful pattern for moving from simulation to hardware.
How to estimate model performance
Start by defining the quantity you will estimate before running the experiment. For a classification task, a useful primary estimate is the mean test score across independent random seeds:
Mean score = (score1 + score2 + ... + scorek) / k
Also calculate the spread across runs. A model with a slightly higher mean accuracy but much greater variability may be less dependable than a simpler model with stable results. Report the standard deviation or an interval based on the observed runs, and state exactly how it was calculated.
For an end-to-end estimate of experimental burden, separate the main cost components:
- Training evaluations: number of optimizer steps multiplied by the number of circuit evaluations per step.
- Measurement workload: circuit evaluations multiplied by the number of shots, observables, or measurement settings.
- Classical workload: preprocessing, optimization, gradient computation, validation, and post-processing time.
- Inference workload: circuit executions and shots required to classify one new example or a batch.
A simple workload estimate is:
Total shots = training steps × evaluations per step × shots per evaluation
This is not a universal price estimate. Provider pricing, queue time, simulator configuration, batching, and hardware access vary. Treat it as a transparent accounting method rather than a financial claim. Record wall-clock time and execution counts alongside model quality.
For circuit-specific measurements, track qubit count, logical gate count, two-qubit gate count, circuit depth, number of trainable parameters, shots, and whether gradients were computed analytically, by parameter shift, by finite difference, or through another method. The guide to benchmarking a quantum circuit provides a complementary checklist for these measurements.
Inputs and assumptions
Dataset design
Keep the dataset split fixed across models. Use training, validation, and test sets when hyperparameter tuning is required. The test set should be opened only after model choices are finalized. For small datasets, use repeated stratified cross-validation or another justified resampling method, but do not mix validation results with final test results.
Document class balance, missing-value handling, scaling, feature selection, dimensionality reduction, and data encoding. Fit every learned preprocessing step on the training data only. Leakage in normalization, feature selection, or resampling can make a quantum model—or its classical counterpart—appear stronger than it is.
Baseline controls
A quantum ML benchmark is weak if the classical comparison is deliberately under-tuned. Use the same input features and, where appropriate, the same reduced feature dimension. Record the baseline's parameter count, training procedure, stopping rule, hardware, and runtime. If the quantum model receives a special feature transformation, include a classical model using that transformation as a control.
Metrics
Choose metrics based on the decision the model supports:
- Balanced classification: accuracy, balanced accuracy, macro F1, and a confusion matrix.
- Imbalanced classification: precision, recall, F1, area under a suitable precision-recall curve, and class-specific errors.
- Probabilistic predictions: log loss and calibration. A model can have good accuracy but poorly calibrated confidence.
- Regression: mean absolute error, root mean squared error, and, when relevant, error by target range.
- Operational comparison: training time, inference time, shots, circuit depth, parameter count, and failure rate.
Do not select metrics after seeing which model wins. Define the primary metric, secondary metrics, and acceptance threshold in advance.
Noise and execution conditions
Run at least one idealized simulation condition and one condition that reflects the intended deployment path. For example, compare exact expectation values with finite-shot simulation, then add a noise model or hardware execution when appropriate. Keep the noise assumptions visible: noise model, shot count, transpilation or compilation settings, error mitigation, and device configuration.
Noise can affect both the mean result and its variance. If mitigation is used, report the unmitigated result as well. The quantum error mitigation workflow can help organize this comparison.
Worked examples
Example 1: A small binary classifier
Assume a binary dataset has already been split into fixed training, validation, and test partitions. Four models use the same selected features: a majority baseline, logistic regression, a tuned classical model, and a variational quantum classifier. Run each model with five predetermined seeds.
The evaluation table should include one row per model and columns for mean test accuracy, standard deviation, macro F1, log loss, calibration error, training time, parameter count, circuit depth where applicable, and total shots. These are illustrative fields, not expected results.
Suppose the quantum model has a marginally higher mean accuracy than logistic regression but lower macro F1, worse calibration, and substantially greater shot usage. The correct conclusion is not that the quantum model “wins.” A careful conclusion would state that the observed accuracy difference did not translate into a broad advantage under the selected metrics and resource budget. You could then test whether the result changes with a different encoding, circuit depth, or noise condition—but each change should be treated as a new experiment.
Example 2: Estimating run workload
Assume a training configuration uses 80 optimizer steps, 12 circuit evaluations per step, and 1,024 shots per evaluation. The estimated training workload is:
80 × 12 × 1,024 = 983,040 shots
If five seeds and three noise conditions are run, the planned workload becomes:
983,040 × 5 × 3 = 14,745,600 shots
This estimate excludes validation and inference unless those are added separately. It also assumes every evaluation uses the same shot count. If adaptive shots, early stopping, batching, or failed jobs are involved, record actual totals in addition to the planned estimate.
Example 3: Testing a claimed quantum advantage
For a claim that a quantum model is better than a classical model, define the claim narrowly. It might be “higher balanced accuracy at equal feature dimension and a fixed shot budget,” or “similar predictive quality with fewer trainable parameters.” Then test that claim directly. Do not compare accuracy from one random split with runtime from another, and do not compare an ideal simulator result with hardware output without labeling the execution conditions.
When to recalculate
Recalculate the evaluation whenever an input that can change the conclusion changes. This includes the dataset or split, preprocessing pipeline, feature encoding, ansatz, optimizer, shot count, simulator backend, noise model, hardware target, compiler or transpiler settings, error-mitigation method, SDK version, or classical baseline.
Revisit the experiment after major framework releases because APIs, default compilation behavior, gradient implementations, and backend capabilities can change. Store the environment specification, dependency versions, circuit-generation code, configuration files, seed list, raw measurement data, and result-processing scripts. A version migration checklist such as the quantum SDK compatibility guide can help identify changes that require a rerun.
Before publishing or sharing a result, use this final checklist:
- Freeze the dataset split and preprocessing rules.
- Run a simple and a strong classical baseline.
- Predefine primary and secondary metrics.
- Repeat every model across multiple recorded seeds.
- Report mean, spread, and practical resource use.
- Separate ideal simulation, noisy simulation, and hardware results.
- Publish enough configuration detail to reproduce the circuit and training loop.
- State limitations instead of treating a small score difference as proof of advantage.
Quantum machine-learning evaluation becomes more useful when it is treated as an accounting problem as well as a modeling problem: account for data, metrics, randomness, circuit resources, execution conditions, and classical alternatives. That discipline makes results easier to interpret today and easier to revisit when the underlying tools or benchmarks change.