qMRLab cross-version benchmark

9 qMRI models fitted against identical, checksum-pinned data across 17 targets from 4 software families (qmrlab, qmrust, quit, sct). Reference target: qmrlab v3.0.0.

Did every fit this benchmark declares actually run? The grid below is one target per row, one model per column, and one fit per cell.

15 × 9 targets × models93 produced maps10 failed32 not applicable

Every cell that is not a tick prints its reason in What did not run, under the grid, rather than in a tooltip — read that list first.

The most common misreading is the muted not applicable cells. A target declares the models it supports in its target.yml, and omission is the only way this repo declares a gap, so those cells are a scope decision and not a fit that broke.

The tiles that follow count the whole run; the grid under them is version history only, which is why it holds fewer targets. The note between them names what that leaves out.

17
targets
4
software families
9
models
100
successful fits
10
failed fits

Version history: qMRLab releases and qmrust. quit v3.4, sct 7.3 are on the Cross-software tab, and so is anything of theirs that failed.

Every declared target × model in this view. A status colour never carries meaning alone — each cell has an icon, and hovering gives the reason a cell failed.
b1_afib1_daminversion_recoverymono_t2mt_ratiomt_satqmt_spgrqmt_spgr_ramanivfa_t1
qmrlab V1······✓✓·
qmrlab v2.0.0·✓✓··✓✓✕✓
qmrlab v2.0.1·✓✓··✓✓✕✓
qmrlab v2.0.2·✓✓··✓✓✕✓
qmrlab v2.0.3·✓✓··✓✓✓✓
qmrlab v2.0.5·✓✓··✓✓✓✓
qmrlab v2.1.0·✕✓··✕✓✓✓
qmrlab v2.2.0·✕✓··✕✓✓✓
qmrlab v2.3.0·✓✓·✓✕✓✓✓
qmrlab v2.4.0·✓✓✓✓✕✓✓✓
qmrlab v2.4.1·✓✓✓✓✕✓✓✓
qmrlab v2.5.0✓✓✓✓✓✓✓✓✓
qmrlab v3.0.0✓✓✓✓✓✓✓✓✓
qmrlab master✓✓✓✓✓✓✓✓✓
qmrust main✓✓✓✓✓✓✓✓✓
ok failed missing not applicable

What did not run