BBSure validation run: 200 simulated sets

Decision tree + profile scheme on 200 simulated SO SAT data sets: 100 CMB + noise realizations (seeds 8000–8099; r log-uniform in [10−3, 0.03], 10 with r = 0), each analysed with PySM d9s4 (constant spectral indices: the true model has no moments) and with one PySM sky drawn among d9/d10/d12 × s4/s5/s7. nside 128, no E modes in the signal, hit-weighted noise (pessimistic goal), covariance from Gaussian simulations with foregrounds fitted on d1s1. Code: BBSure branch r0_pruning (commit 0bd6188) with BBPower Bsure. r in units of 10−3; pull = (r − rtrue)/σ, σ the standard deviation of the posterior.

Decision tree and profile scheme

flowchart of the decision tree and of the profile scheme
The flowchart run on every set. Resr0: each round adds the most significant family with r = 0 and prunes only its new spectra with r = 0 (M1,0); then M1,r (pruned with r free) and M1,0 are compared with M0,r; families discarded in a round are not retested. Borderline choices are repeated with up to 3 PolyChord runs and flagged as undecided. Click to zoom.

Main results (profile scheme)

Summary: tree → profile scheme

Bias = mean pull ± its error over the sets; width = standard deviation of the pulls (1 if σ is right). Coverage with the 68% central interval of each set (expected 68%). Detections and upper limits as concluded by each scheme.

Goodness of fit of the final model

Number of sets per PTE bin; tree: χ² of the tree model, profile scheme: χ² at r = r*. With ~450 degrees of freedom the mean χ²/dof over 100 sets should be 1.000 ± 0.007.

Paired difference (rrand − rd9s4)/σrand

Same CMB + noise in each pair, only the foregrounds change; profile scheme, with the tree and the number of pairs below.

Plots

estimated r against true r
Estimated r (median ± σ) against the true r; left: Bayesian tree, right: profile scheme; top: d9s4, bottom: random skies. Blue: detection, orange: upper limit. Dashed: weighted linear fit rest = a + b rtrue.
pull histograms, d9s4
Pull histograms, d9s4: all sets, upper limits, detections. Filled: profile scheme; black outline: tree; solid curve: N(0,1).
pull histograms, random skies
Pull histograms, random skies, same layout.
r set by set, d9s4
r set by set, d9s4, profile scheme: median and 68% interval (detections), 95% upper limit (orange bar); the star is the true r. Red circle and band: wrong conclusion (true r above the 95% upper limit, or detection with true r = 0).
r set by set, random skies, with d9s4 of the same set
r set by set, random skies, profile scheme. Black star: true r; green diamond: d9s4 result of the same set (same CMB + noise), so the distance from the diamond shows the effect of the foregrounds alone. Red circle and band: wrong conclusion. Sky under each set.
same plots for the Bayesian tree
r set by set, d9s4, tree
d9s4, tree.
r set by set, random skies, tree
Random skies, tree; star: true r, diamond: d9s4 of the same set.

Mean pull by sky

The PTE flag here is set at 0.01. At 0.05 it fired in about a quarter of the sets, also in d9s4 where Mnone is the true model (PTE < 0.05 in 23%, expected 5%): the χ² is systematically high, a normalisation (covariance) issue rather than a sign of bad foreground modelling. At 0.01 it fires in 4% of the d9s4 and 7% of the rand sets.

All sets

PTEfinal model PTE < 0.01 ≈mommoment adopted or rejected with |ln B| < 2σ ≈cutpruning undecided ≈besttwo best models of the profile scheme within 2σ M₁,₀ vs M₀,r: M₁,₀ ✓preferred M₁,₀ ≈✓preferred, |ln B| < 2σ M₁,₀ ≈✗rejected, |ln B| < 2σ detyellow cells: detection with true r < 3×10⁻³ UL ✗ / det ✗ red cells: wrong conclusion: true r above the 95% upper limit, or detection with true r = 0