Answer test · run November 22, 2026

Check our accuracy yourself.

Decize answers “what is our rule?” from your team’s own rulebook. This page publishes the test we run before every change to that answer: 3 sample handbooks, 33 questions, the right answer for each, and what the live run on production scored on November 22, 2026.

The results

Five numbers, all from one run.

Run on November 22, 2026 against production (commit 7401d644). The test’s pass line: every question answered and zero wrong answers. It passed.

  • 24 of 24

    Questions with a rule got the current rule

    Every question that has a rule in the handbook was answered with the rule in force today.

  • 0

    Wrong or old rules cited

    No answer cited a replaced rule or a wrong one.

  • 9 of 9

    Said “no rule yet” instead of guessing

    Where the handbook holds no rule, Decize said so every time.

  • 48 of 48

    Imported rules quote the handbook word for word

    Every rule Decize extracted from the handbooks carries a quote that is really in the text.

  • 3 of 3

    Disagreeing rules flagged

    Every labelled pair of in-force rules that contradict each other was flagged.

Check it yourself

Every rule, every question, every right answer.

33 questions over 33 rules; 9 of the questions have no rule behind them on purpose. The run’s per-question answers were not saved. The pass line needs every question answered and none wrong, so each question below passed.

Larkspur Metalworks

11 rules · 11 questions · 3 with no rule

The rules

  • Overtime over 4 hours needs the foreman's approval

    Overtime · Owner: Ruben Castillo · Decided Feb 12, 2024 · Replaced by “Overtime over 6 hours needs Marta's approval”

  • Overtime over 6 hours needs Marta's approval

    Overtime · Owner: Marta Okafor · Decided Mar 3, 2025 · In force

  • Scrap above 3 percent stops the line

    Quality · Owner: Dev Patel · Decided Sep 5, 2023 · Replaced by “Scrap above 5 percent stops the line”

  • Scrap above 5 percent stops the line

    Quality · Owner: Dev Patel · Decided Aug 18, 2025 · In force

  • First-article inspection for every new part number

    Quality · Owner: Dev Patel · Decided Apr 2, 2024 · In force

  • Eye protection on the shop floor for everyone

    Safety · Owner: Ruben Castillo · Decided Jun 12, 2023 · In force

  • A machine is locked out before a guard comes off

    Safety · Owner: Ruben Castillo · Decided Jun 12, 2023 · In force

  • Visitors are escorted on the shop floor

    Visitors · Owner: Ruben Castillo · Decided May 20, 2024 · In force

  • Cleared customers may walk the floor alone

    Visitors · Owner: Marta Okafor · Decided Jan 13, 2025 · In force

  • Tooling over $1,500 needs two written quotes

    Purchasing · Owner: Marta Okafor · Decided Sep 9, 2024 · In force

  • A week or more off needs 3 weeks' notice

    Time off · Owner: Joan Whitfield · Decided Nov 1, 2023 · In force

The questions

  1. How many overtime hours can I work in a week before I need approval, and who approves it?

    Expected: Overtime over 6 hours needs Marta's approval

    Passed, Nov 22, 2026

  2. What scrap rate on a job stops the line?

    Expected: Scrap above 5 percent stops the line

    Passed, Nov 22, 2026

  3. Do I need safety glasses when I am only walking through the shop?

    Expected: Eye protection on the shop floor for everyone

    Passed, Nov 22, 2026

  4. What has to happen before I take a guard off a machine?

    Expected: A machine is locked out before a guard comes off

    Passed, Nov 22, 2026

  5. Do we inspect the first parts of a brand new job?

    Expected: First-article inspection for every new part number

    Passed, Nov 22, 2026

  6. When do I need to get quotes before ordering tooling?

    Expected: Tooling over $1,500 needs two written quotes

    Passed, Nov 22, 2026

  7. How much notice do I give for a week off?

    Expected: A week or more off needs 3 weeks' notice

    Passed, Nov 22, 2026

  8. Can a customer walk through the shop floor without an escort?

    Expected: The two rules disagree: Visitors are escorted on the shop floor / Cleared customers may walk the floor alone. The right answer cites one and flags the other.

    Passed, Nov 22, 2026

  9. Do we pay a shift differential for the night shift?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

  10. What is the dress code for the front office?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

  11. Are personal phones allowed on the shop floor?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

Cobalt Lane Software

12 rules · 11 questions · 3 with no rule

The rules

  • Production deploys must finish by 2 pm

    Releases · Owner: Imani Brooks · Decided Mar 18, 2024 · Replaced by “Production deploys can run until 4 pm”

  • Production deploys can run until 4 pm

    Releases · Owner: Imani Brooks · Decided Jul 14, 2025 · In force

  • Cut releases from main

    Releases · Owner: Imani Brooks · Decided Mar 18, 2024 · In force

  • Require changelog entry per release

    Releases · Owner: Imani Brooks · Decided Mar 18, 2024 · In force

  • Every pull request needs one approving review

    Code review · Owner: Imani Brooks · Decided Jan 22, 2024 · In force

  • Billing pull requests need two approving reviews

    Code review · Owner: Imani Brooks · Decided Jan 22, 2024 · In force

  • Docs-only changes merge without review

    Docs · Owner: Priyanka Shah · Decided May 12, 2025 · In force

  • On-call rotates every week

    On-call · Owner: Priyanka Shah · Decided Oct 2, 2023 · Replaced by “On-call rotates every two weeks”

  • On-call rotates every two weeks

    On-call · Owner: Priyanka Shah · Decided Feb 10, 2025 · In force

  • Post a status update within 15 minutes of a severity 1

    Incidents · Owner: Priyanka Shah · Decided Jun 3, 2024 · In force

  • Keep secrets out of the repository

    Security · Owner: Tomas Ruiz · Decided Feb 12, 2024 · In force

  • Software under $200 a month needs no approval

    Spending · Owner: Lena Fischer · Decided Nov 4, 2024 · In force

The questions

  1. What is the latest time of day we can run a production deploy?

    Expected: Production deploys can run until 4 pm

    Passed, Nov 22, 2026

  2. How often does on-call rotate, and when is handover?

    Expected: On-call rotates every two weeks

    Passed, Nov 22, 2026

  3. How fast do we have to post a status update when there is a severity 1 incident?

    Expected: Post a status update within 15 minutes of a severity 1

    Passed, Nov 22, 2026

  4. How many reviewers does a change to the billing code need?

    Expected: Billing pull requests need two approving reviews

    Passed, Nov 22, 2026

  5. Is one approving review enough for a pull request that touches billing code?

    Expected: Billing pull requests need two approving reviews

    Passed, Nov 22, 2026

  6. Where do API keys and other secrets go?

    Expected: Keep secrets out of the repository

    Passed, Nov 22, 2026

  7. Can I buy a 150 dollar a month tool for the team without asking anyone?

    Expected: Software under $200 a month needs no approval

    Passed, Nov 22, 2026

  8. Does a docs-only change need a review before it merges?

    Expected: The two rules disagree: Every pull request needs one approving review / Docs-only changes merge without review. The right answer cites one and flags the other.

    Passed, Nov 22, 2026

  9. Are engineers allowed to use AI coding assistants on customer code?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

  10. How much equipment budget does a new hire get?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

  11. Do we work a four-day week?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

Pennant & Rowe Advisory

10 rules · 11 questions · 3 with no rule

The rules

  • Invoices go out on the 25th

    Billing · Owner: Gwen Aldous · Decided Aug 14, 2023 · Replaced by “Invoices go out on the last business day of the month”

  • Invoices go out on the last business day of the month

    Billing · Owner: Gwen Aldous · Decided Apr 7, 2025 · In force

  • Invoices unpaid after 45 days go to a partner

    Billing · Owner: Gwen Aldous · Decided Jan 15, 2024 · In force

  • Conflict-of-interest check before every new engagement

    Engagements · Owner: Naomi Okoye · Decided Mar 11, 2024 · In force

  • Client files stay on the shared drive

    Confidentiality · Owner: Naomi Okoye · Decided Dec 4, 2023 · In force

  • Client travel over $400 needs partner approval

    Expenses · Owner: Tobias Wren · Decided Sep 23, 2024 · In force

  • Timesheets are due at noon on Monday

    Time and attendance · Owner: Tobias Wren · Decided Feb 5, 2024 · In force

  • Long leave needs two weeks' notice

    Time and attendance · Owner: Tobias Wren · Decided Jul 1, 2024 · In force

  • Office on Tuesdays and Thursdays

    Working pattern · Owner: Tobias Wren · Decided Feb 26, 2024 · In force

  • Fully remote, no office days

    Working pattern · Owner: Gwen Aldous · Decided Mar 24, 2025 · In force

The questions

  1. On what date do invoices go out each month?

    Expected: Invoices go out on the last business day of the month

    Passed, Nov 22, 2026

  2. When does a client invoice that has not been paid get escalated?

    Expected: Invoices unpaid after 45 days go to a partner

    Passed, Nov 22, 2026

  3. Do we check for conflicts before taking on a new client?

    Expected: Conflict-of-interest check before every new engagement

    Passed, Nov 22, 2026

  4. Can I email a client's spreadsheet to my personal address to work on it at home?

    Expected: Client files stay on the shared drive

    Passed, Nov 22, 2026

  5. Does client travel need sign-off before I book it?

    Expected: Client travel over $400 needs partner approval

    Passed, Nov 22, 2026

  6. When are timesheets due?

    Expected: Timesheets are due at noon on Monday

    Passed, Nov 22, 2026

  7. How far ahead do I have to ask for a three-week holiday?

    Expected: Long leave needs two weeks' notice

    Passed, Nov 22, 2026

  8. Do I have to be in the office on Thursdays?

    Expected: The two rules disagree: Office on Tuesdays and Thursdays / Fully remote, no office days. The right answer cites one and flags the other.

    Passed, Nov 22, 2026

  9. How big is the referral bonus for bringing in a new client?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

  10. What is the firm's policy on putting client data into AI tools?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

  11. What is the dress code for client meetings?

    Expected: No rule — the right answer is “no rule yet”

    Passed, Nov 22, 2026

Method and limits

The real thing, with nothing saved.

How it is run

The test calls the same production AI paths customers use: the one that answers a question from the rulebook, the one that reads a handbook and pulls out its rules, and the one that checks rules for disagreement. It writes nothing to any database. Each handbook is sent through the import, each question is asked, and each pair of in-force rules in the same area is checked. The scoring rules are listed on the right; the scoring code itself lives in our private repository.

What counts as wrong

  • An answer that cites a replaced rule, or any rule other than the one in force.
  • An answer that cites no rule that can be checked.
  • An answer that gives a rule when the handbook holds none.
  • A flat answer when two in-force rules disagree: it must cite one and flag the other.
  • An imported rule is “grounded” only if its quote appears in the handbook word for word, whitespace and capital letters aside.

Limits

  • The handbooks were written by the Decize team. They are fictional and small, 10 to 12 rules each.
  • This is not customer data.
  • Results can differ on real, messier handbooks.
  • The questions were written by us too. That is why we publish them: so you can judge whether they are fair.

The run is dated November 22, 2026. A later change to how Decize answers is tested again before it ships. See also the methodology.

Try it

Ask the sample metal shop yourself.

Put your own question to the Larkspur Metalworks handbook and compare the answer with the right one above. See also our public decision ledger and how we handle security.