05 · Data Quality and Confidence Factor
Use: Every published outcome and SROI ratio carries a confidence factor between 0% and 100%. This file defines how it is calculated, when we suppress a number entirely, and how we communicate uncertainty in plain English.
The confidence factor formula
confidence_factor =
0.30 × sample_size_score
+ 0.25 × response_rate_score
+ 0.20 × instrument_validity_score
+ 0.15 × attribution_score
+ 0.10 × data_freshness_score
Result rounded down to the nearest 1%. Maximum publishable: 95% (we don't claim 100%; impact work always has residual uncertainty).
Component definitions
| Component | 1.0 | 0.5 | 0 |
|---|---|---|---|
| sample_size_score | n ≥ 30 in the metric | n ≥ 10 | n < 10 (and ≥5; below 5 is suppressed) |
| response_rate_score | ≥75% of cohort responded to both pre and post | ≥50% | <50% |
| instrument_validity_score | Validated instrument used unchanged | Validated instrument adapted (length / wording) with rationale logged | Bespoke instrument, no validation work |
| attribution_score | Control cohort or strong comparator | Pre/post only, clean | Self-report only, no pre-baseline |
| data_freshness_score | Within 30 days of measurement | Within 90 days | Older than 90 days |
Suppression rules — when we don't publish a number at all
A metric is suppressed (replaced with "n suppressed for anonymity" or "insufficient data — see methodology") when ANY of:
n < 5for the metric.- The combination of demographic filters used to slice the data would re-identify an individual (e.g. "one menopause-experiencing manager in Engineering" — even if n ≥ 5 in the parent cohort).
- Response rate is below 25% — the responders are too self-selected to draw any signal.
- Confidence factor would compute below 30% — published number would be more misleading than useful.
Suppressed metrics are listed by name in every report with the reason, so the reader knows we measured them rather than chose not to.
Communicating uncertainty in plain English
Every published number is paired with a confidence band:
| Confidence factor | Plain-English label |
|---|---|
| 80–95% | "Strong evidence." |
| 60–79% | "Good evidence with normal caveats." |
| 40–59% | "Early signal, treat as indicative." |
| 30–39% | "Direction of travel only; small sample." |
| <30% | Suppressed (do not publish). |
Example caption:
73% of managers said they had held a difficult conversation they would previously have avoided. Strong evidence (confidence 82%, n=41 of 48 cohort members, pre/post pulse, instrument adapted from Sprint 1 / 01).
What "data quality" means at intake
Before a metric goes anywhere near a report, it passes the intake check:
- Source row(s) exist in the data model (Sprint 5 / 05) with valid foreign keys.
- Aggregate respects the
n ≥ 5view (Sprint 5 / 05). - Pre and post timestamps are within the engagement window.
- Free-text responses have been coded by ≥2 reviewers if used as a theme.
- Outliers (>3 SD) are flagged; kept by default, removed only with reason logged.
- Missing-data rate is logged.
If any box is unchecked, the metric goes back to the Account Manager and Data Lead before publication.
Independent re-calculation
Once per quarter, the Head of Impact picks one client's headline numbers at random and asks the Data Lead to re-derive them from raw data without looking at the prior calculation. Discrepancies >2% trigger an investigation; >5% trigger a correction under file 10.
What we do not do with confidence factor
- Hide it. It appears next to every monetised number, every time.
- Apply it after-the-fact to make a weak number look stronger ("rounded up for presentation").
- Use it as an excuse to publish anything below the suppression threshold.
- Average confidence factors across unrelated metrics into a single "report confidence" — that hides where the weakness actually lives.