Benchmarks¶
This page is generated by docs/scripts/benchmarks.py from the JSON
output of scripts/bench_exact.py — every number below was measured,
on the machine stamped here, by a run anyone can repeat.
Measured on: Linux x86_64, 4 cores, Python 3.11.15, treecf 0.3.1, xgboost 3.2.0, lightgbm 4.6.0. The exact engine is sequential; wall times are per solve.
Exact versus genetic¶
Same model, same target, same seeds; the genetic backend's distance gap
is how far its plan sits from the proved minimum of the default objective
(σ-normalized distance, sparsity_weight=0). It is not a quality gap: the
heuristic's plans change fewer features, which that objective does not
reward — see what optimal means.
Per-solve budgets: 5 s wall, 2,000,000 nodes. classic is the
default exact engine (search="classic"); refine is the opt-in
coarse-to-fine engine (search="refine"), timed on the same solves.
| Scenario | genetic median | classic warm | classic cold | refine warm | refine cold | distance gap median | proof mix (classic) | proof mix (refine) |
|---|---|---|---|---|---|---|---|---|
| small 30t/d4/8f [HEADLINE] | 0.005 s | 0.246 s | 0.239 s | 0.009 s | 0.004 s | 14.33% | optimal: 10 | optimal: 10 |
| medium 60t/d5/12f | 0.008 s | 5.295 s | 5.288 s | 0.315 s | 0.293 s | 0.00% | heuristic: 10 | optimal: 10 |
| large 300t/d6/50f | 0.129 s | 10.923 s | 10.953 s | 11.389 s | 11.232 s | 0.00% | heuristic: 5 | heuristic: 5 |
Time to certificate, by model scale¶
Each cell asks for a certified answer — proved optimal, optimal within gap, or certified infeasible — with a 60 s wall budget and an effectively unlimited node budget.
| Scenario | classic: median | classic ≤ 10 s | classic ≤ 60 s | refine: median | refine ≤ 10 s | refine ≤ 60 s |
|---|---|---|---|---|---|---|
| 50t/d3/8f | 0.129 s | 100% | 100% | 0.010 s | 100% | 100% |
| 50t/d3/12f | 0.096 s | 100% | 100% | 0.011 s | 100% | 100% |
| 50t/d3/20f | — | 0% | 0% | 18.785 s | 0% | 100% |
| 50t/d5/8f | 17.336 s | 0% | 100% | 0.032 s | 100% | 100% |
| 50t/d5/12f | — | 0% | 0% | 0.230 s | 100% | 100% |
| 50t/d5/20f | — | 0% | 0% | — | 0% | 0% |
| 100t/d3/8f | 2.785 s | 100% | 100% | 0.038 s | 100% | 100% |
| 100t/d3/12f | 5.257 s | 100% | 100% | 0.051 s | 100% | 100% |
| 100t/d3/20f | — | 0% | 0% | — | 0% | 0% |
| 100t/d5/8f | — | 0% | 0% | 0.134 s | 100% | 100% |
| 100t/d5/12f | — | 0% | 0% | 6.450 s | 100% | 100% |
| 100t/d5/20f | — | 0% | 0% | — | 0% | 0% |
| 200t/d3/8f | 28.837 s | 0% | 100% | 0.081 s | 100% | 100% |
| 200t/d3/12f | — | 0% | 0% | 0.713 s | 100% | 100% |
| 200t/d3/20f | — | 0% | 0% | — | 0% | 0% |
| 200t/d5/8f | — | 0% | 0% | 0.330 s | 100% | 100% |
| 200t/d5/12f | — | 0% | 0% | 20.632 s | 0% | 100% |
| 200t/d5/20f | — | 0% | 0% | — | 0% | 0% |
A dash in a median column: no seed certified within the 60 s budget on this machine; a dash across a mode's columns: that engine was not measured.
Native categorical models¶
LightGBM models with four numeric levers plus native categorical features of cardinality 3, 8, and 15 — the search runs over category blocks, so cardinality is not the cost driver.
| Scenario | classic: median | classic ≤ 10 s | classic ≤ 60 s | refine: median | refine ≤ 10 s | refine ≤ 60 s |
|---|---|---|---|---|---|---|
| 50t/d3/4num+3cat | 0.006 s | 100% | 100% | 0.006 s | 100% | 100% |
| 100t/d5/4num+3cat | 0.013 s | 100% | 100% | 0.013 s | 100% | 100% |
| 200t/d5/4num+3cat | 0.021 s | 100% | 100% | 0.018 s | 100% | 100% |
A dash in a median column: no seed certified within the 60 s budget on this machine; a dash across a mode's columns: that engine was not measured.
Positioning¶
How treecf's approach relates to the other families of counterfactual methods for tree ensembles:
| Family | Representatives | Optimality | Constraints | Verified output | Extra dependency |
|---|---|---|---|---|---|
| Tree-native search (this library) | treecf | proof, certified infeasibility, certified regions | declarative, compiled for every backend | float-verified against the parsed model, always | none — numpy plus the bundled core |
| Model-agnostic perturbation | DiCE (genetic/random/kdtree) | none | post-hoc penalty terms | not re-verified by default | none |
| Nearest-neighbor greedy | NICE | none | none | validity checked, cost not minimized | none |
| MILP over the ensemble | OCEAN-style formulations | proof, given a solver | linear, in the MILP | exact within the encoding | commercial or heavyweight MILP solver |
| Counterfactual maps / gradient methods | alibi CounterfactualProto | none | none | gradient-based; struggles on non-differentiable ensembles | TensorFlow |
Measured head-to-head numbers against the pip-installable competitors
(per-instance wall time, validity, plan cost) are in
Backends — against other CF libraries;
they come from scripts/bench_vs_competitors.py under one shared protocol.
Reproduce: uv run python scripts/bench_exact.py --full --matrix --categorical --search both --json results.json, then re-run this generator.