Decision quality methodology
What we measure—and what the numbers cannot prove.
Decize follows a decision from the confidence recorded when it was made to its eventual outcome. This page defines the measures built from that record, the rules that keep hindsight out, and the limits a responsible reader should keep beside every score.
Measurement contract
The record comes before the score.
A number is useful only if the confidence behind it cannot be rewritten while an outcome stands and the denominator is visible. Decize applies three rules before interpretation begins.
01
Record the expectation
The confidence a person confirms cannot change while the recorded outcome stands. Clearing a wrong outcome reopens it.
02
Wait for a real outcome
Unresolved rows do not count as misses, and elapsed time alone does not turn an open question into an answer.
03
Show the denominator
Rates remain blank below the display floor and forecast summaries carry their sample size and early-sample status.
Metric definitions
One formula, one denominator, one honest reading.
These are the definitions used by the product today. Product thresholds are identified as product thresholds; they are not dressed up as universal statistical laws.
- Outcome closure
Calculation
decisions with a recorded outcome ÷ those plus the decisions still due oneWhen it appears
As a count from the first decision due an outcome. As a rate once at least 5 are due one; the dashboard and its PDF also wait until the team has recorded 50 outcomes.
Responsible reading
Whether the team is completing the learning loop. It does not say whether the decisions were good.
- On-time execution
Calculation
deadlines whose due day has ended and were finished (marked done or an outcome recorded) by the end of that day ÷ all such deadlines; imported decisions are not gradedWhen it appears
Not shown at present: the dashboard, its PDF and the emails leave it out.
Responsible reading
Whether commitments were completed when promised. A timely decision can still produce a poor outcome.
- Owner coverage
Calculation
decisions with an owner ÷ all decisions in the selected reporting rangeWhen it appears
In the weekly and monthly emails, over the last 30 days, once at least 5 decisions were logged in them. In the dashboard's PDF, as a count, and as a rate once the team has recorded 50 outcomes and the range contains at least 5 decisions.
Responsible reading
Whether accountability is explicit. It measures presence of an owner, not that person’s performance.
- Brier score
Calculation
mean of (confirmed confidence ÷ 4 − result)², where result is 1 if it went well and 0 otherwiseWhen it appears
On Business, once the team has recorded 50 outcomes and at least 5 decisions are scored; labeled an early sample below 20.
Responsible reading
Lower is better. Zero means every confirmed confidence matched the result perfectly; the score alone does not explain why.
- Skill versus base rate
Calculation
1 − (Brier score ÷ Brier score from always predicting the sample base rate)When it appears
With the Brier score; left blank when every result is identical and the baseline is zero.
Responsible reading
Positive is better than the sample’s base-rate forecast, zero is parity, and negative is worse.
- Reliability bins
Calculation
mean predicted probability compared with observed frequency inside four probability bandsWhen it appears
On Business, once the team has recorded 50 outcomes; each band is withheld until it contains at least 5 scored decisions.
Responsible reading
Points near the diagonal are better calibrated. Empty bands mean insufficient data, not perfect calibration.
The five-record minimum protects against brittle percentages and very small slices. It is deliberately conservative product behavior, not a claim that n=5 makes an estimate precise. Brier results from fewer than 20 scored decisions are separately labeled early sample.
Integrity rules
A standing outcome cannot rewrite the confidence.
These controls make the measures auditable inside a workspace and harder to flatter after the fact.
- 1
Confidence is locked while the outcome stands
Once a person records how a decision turned out, its confidence cannot be set or confirmed while that outcome stands. Clearing a wrong outcome reopens it.
- 2
Decisions without a recorded outcome do not score
A decision is scored only when a person has confirmed its confidence and recorded how it turned out. One still waiting for its outcome is excluded rather than treated as wrong.
- 3
AI readings do not become human judgment
Model-inferred confidence is excluded from the confidence-versus-outcome view and the Brier score until a person confirms it, and an outcome Decize reads from an email reply is only a suggestion until the person taps it and presses Save. Value alignment is labeled an AI estimate, and AI-drafted titles and summaries are saved as drafted.
- 4
Small teams see counts first
Until a team has recorded 50 outcomes, the dashboard and its PDF count how decisions went instead of showing rates, what works or calibration. Below that, a single decision swings a rate too far to read.
- 5
Small groups stay blank
Past that, rates and probability bands use a five-record privacy and display floor. Five is not presented as statistical certainty; scores from fewer than 20 scored decisions remain explicitly early-sample results.
- 6
Organization learning is the unit
Decize reports shared workflow and outcome measures. It does not publish leaderboards or individual decision-quality scores.
Interpretation limits
Evidence about a workflow, not a verdict on a team.
Decize measures recorded behavior. The results are descriptive, not causal: a higher closure rate does not prove that the product caused better decisions, and a lower Brier score does not identify which process produced it.
Small samples remain uncertain even after the display floor. The mix and difficulty of decisions can change across periods, teams, and organizations, so comparisons need context. Decize does not convert these measures into employee rankings or professional advice.
The outcome-closure chart displays up to 12 mature weekly cohorts, leaves out any week with fewer than 5 decisions due an outcome, and waits for 10 such weeks before presenting a trend. That ten-week rule is a Decize interpretation guardrail, not a universal threshold for statistical significance.
Primary sources
The ideas behind the product, linked at the source.
External references explain the established methods and terminology. They do not endorse Decize, validate its implementation, or replace the product limits above.
- 1Verification of Forecasts Expressed in Terms of Probability
Glenn W. Brier · Monthly Weather Review · 1950
The original paper behind the mean-squared probability score Decize reports.
- 2Sampling Uncertainty and Confidence Intervals for the Brier Score
Bradley, Schwartz and Hashino · Weather and Forecasting · 2008
Why a score calculated from a small set of forecasts should be treated as an estimate, not a verdict.
- 32015 Letter to Shareholders
Amazon · published 2016
The primary source for the one-way-door and two-way-door decision terminology used in Decize.
- 4Process or Product Monitoring and Control
NIST/SEMATECH e-Handbook of Statistical Methods
Background for reading measurements over time without treating a single point as a trend.
Inspect the product
Read the definitions beside the evidence.
The documentation explains what ships, the public ledger shows real decisions from building Decize, and the contact page reaches the team responsible for both.