Q-Psi vs Qiskit and TKET: What the MQT Benchmark Actually Showed
da1ul7e3kjvs73877tag1. The Need for Honest Cross-Compiler Benchmarking
In quantum software literature, new compilers frequently claim universal superiority over established tools like Qiskit or TKET by selectively publishing favorable benchmarks or running unoptimized competitor baselines.
In our MQT Benchmark campaign, Q-Psi compared its compiler against Qiskit O3 and TKET on IBM Heron r2 (ibm_marrakesh, 36,864 shots) across 6 standard MQT algorithms.
2. The Headline vs. The Reality
Across all 6 physical algorithms, Qiskit O3 held a small aggregate advantage (mean delta \(-0.0067\), 95% CI: \([-0.0123, -0.0010]\)). Q-Psi tied or closely matched Qiskit on simple circuits but was disfavored on deep circuits like qft_n6.
A naive aggregate calculation suggests Q-Psi "beat" TKET with a \(+0.0398\) delta. However, sensitivity analysis reveals that this aggregate positive delta was caused by a single workload: vqe_su2_n8, where TKET's heavy-hex routing suffered a severe layout failure (\(+0.2696\) delta for Q-Psi).
vqe_su2_n8 shifts the aggregate delta to -0.0062 ([-0.0120, -0.0006]).3. Why Claims of General Superiority Are Prohibited
Claiming "Q-Psi outperforms TKET" without disclosing the single-workload sensitivity would be scientifically misleading. Q-Psi classifies this result as MIXED COMPETITIVENESS and explicitly disclaims universal compiler superiority.