QA TEMPLATE / SAMPLE
Test Specification
Test specification | Sources, scope, risk and execution criteria
What this document answers
Define Order Book update rules, test scope and traceable oracles. This set focuses on stale-sequence updates overwriting current state, based on this site's synthetic Fail Demo.
Blank template
[Required] applies to every case. [Conditional] is required when applicable; explain when it is not. Copy the text into your case documents and replace the brackets. This static page has no editing or saving function.
[Required] Spec ID / version: [ ] Document status: [draft / awaiting review / approved] [Required] Product / target version / owner: [ ] [Required] Objective, scope, exclusions: [ ] [Required] Requirements and oracles: REQ ID | source / version | rule | confirmation status / person / date [Required] Risks: RISK ID | failure mode | impact | likelihood | detectability | rationale [Required] Trace: REQ ↔ RISK ↔ Scenario ↔ Case / version ↔ Run ↔ Defect [Required] Questions: Q ID | question | owner | affected IDs | answer source / date | new criteria version [Required] Execution gates: version, role, data, oracle, isolation, budget, stop conditions and evidence [Required] Authority: permitted accounts / operations, AI-readable data, sanitization, retention location / period, cleanup scope [Required] Review: design and result reviewers, methods, sources and limitations [Required] Defects: severity definitions, decision authority, lifecycle, retest and regression selection [Required] Exit: critical-path evidence; unresolved S1 / S2 disposition; explained traceability gaps [Required] Revision: reason | old / new version | affected items | reassessment and retest plan [Conditional] AI tests: model / prompt / data versions, K attempts, allowed variation, calibration and review [Conditional] Data contract: DC ID / version / approval status; source, period, as-of rules, units, formula, missing / zero values, rounding and tolerance Reference execution results in separate Runs. Do not prefill actual results here.
Filled example: Order Book update order
DEMO-SPEC-MDQ · v1.2
Objective: verify sequential updates, stale-sequence rejection and snapshot loading after reconnect. Scope is isolated synthetic market-data state. Exclude real exchange protocols, orders, executions, assets and load testing.
Environment label: DEMO-BOOK / build-demo-1; no real endpoint. Document status: teaching draft. Case IDs and the 104→103 failure come from app/run-records/fail-demo/page.tsx. The supplemental data contract is for teaching and has no product-owner approval.
| Requirement / oracle | Synthetic rule | Source / status |
|---|---|---|
| DEMO-REQ-NEXT | For current=104, incoming=105, apply the update and retain the new sequence. | TC-MDQ-016 sequential-update scenario; concrete values are teaching assumptions. |
| DEMO-REQ-STALE | For current=104, incoming=103, reject the old event and preserve both sequence and the entire Order Book. | TC-MDQ-017; 104 / 103 are retained from this site's Fail Demo. |
| DEMO-REQ-RECONNECT | After reconnect, replace old state with the specified latest complete snapshot. | TC-MDQ-018; snapshot 106, prices and quantities are teaching supplements. |
Shared data contract
DEMO-DC-MDQ · v1.0
Dataset DEMO-BOOK-DATA-v1 contains only the self-contained synthetic values below. Pair DEMO-BASE/QUOTE: price units QUOTE/BASE, quantity units BASE. These simplified rules do not represent any exchange's sequence ranges or WebSocket protocol.
| Clause | Definition |
|---|---|
| C1 Snapshots and events | Baseline SNAP-104: sequence=104, bids=[[100,2]], asks=[[101,3]]. E-105: sequence=105, updates bid quantity at price 100 to 4. E-103: sequence=103, attempts to update the same bid quantity to 9. Reconnect SNAP-106: sequence=106, bids=[[100,5]], asks=[[101,6]]. Each case starts with its own SNAP-104 and must not contaminate later cases. |
| C2 Oracles and tolerance | After a sequential update, expect sequence=105, bid 100×4, ask 101×3. After a stale event, expect sequence=104, bid 100×2, ask 101×3; the entire state must stay unchanged. After reconnect, expect the complete SNAP-106. Sequences are integers. Compare prices and quantities as exact decimals, with absolute tolerance 0 and no rounding. Zero-quantity deletion, missing values, equal sequences and gaps are undefined and need confirmation; missing is not zero. |
| C3 Timing and observation | Compare before / after state at the event-completion checkpoint, or after full snapshot application for reconnect. This synchronous teaching scenario has no product-latency SLA and does not inherit the original 30-second wait. Record source as-of time, capture time and timezone separately. Sequence establishes order; wall-clock time does not replace it. Missing completion signals or observable state mean Blocked; elapsed seconds alone cannot establish Pass / Fail. |
| C4 Evidence and stopping | Retain event IDs, inputs, complete before / after state and checkpoints. Valid preconditions with a C2 violation mean Fail; inability to prepare isolated snapshots or observe state means Blocked. Preserve confirmed failures alongside other blockers. Stop the affected case after stale data overwrites state and retain snapshots; run other cases only on independent isolated data. |
Risks and traceability
| Requirement / risk | Scenario / case | Result entry point |
|---|---|---|
| DEMO-REQ-STALE / RISK-OB-SEQ: stale market data overwrites current state and may affect downstream pricing decisions. Likelihood and detectability: Unknown. | DEMO-SC-BOOK → TC-MDQ-017 v1.2 | DEMO-RUN-FAIL-001 → DEF-MDQ-004 (all synthetic) |
| DEMO-REQ-NEXT / RISK-OB-SEQ | DEMO-SC-BOOK → TC-MDQ-016 v1.2 | Case results for this batch |
| DEMO-REQ-RECONNECT / RISK-OB-SEQ | DEMO-SC-BOOK → TC-MDQ-018 v1.2 | Case results for this batch |
Questions and version impact
| Question / accountability | Affected items | Resume condition |
|---|---|---|
| DEMO-Q-SEQ: what are the actual feed rules for sequences, reconnect and completion signals? The product / feed-integration owner must confirm. | Three requirements, DC C1–C3, three cases and their Runs; real rules Unknown. | Obtain the official specification and version, revise the oracle, identify differences, reassess affected cases and append Runs. Old Runs retain their original versions. |
| DEMO-Q-DATA: which inputs may AI read, how long is evidence retained, and who may clean it up? The data owner must confirm. | External AI extraction, retention and cleanup. This example calls neither AI nor real services. | Confirm allowed data categories, sanitization, model / service, storage location and retention. Do not transfer or clean up data before confirmation. |
For review disagreements, retain both conclusions and original evidence for the designated owner to recheck. Do not promote affected decisions to Ready. For AI misjudgment, preserve inputs, model / prompt and evaluation versions; relabel manually, assess the affected scope, then reevaluate on a held-out set.
Execution, review and exit
Before execution
Check version, isolated fixture, event input, completion signal, oracle, data access, operation limits and evidence per case. Do not operate the product without required prerequisites; mark selected cases Blocked.
Review
Design review compares sources with cases. Result review inspects raw inputs / outputs before comparing conclusions. Switching roles within the same model is not independent review. These are synthetic filled examples with no human approval or product execution.
Three exit conditions
- Critical paths complete all required steps and checkpoints with valid evidence for the current version / environment.
- No unresolved S1. S2 is fixed with required retest / regression, or explicitly deferred by an authorized owner with reason, accountability and review date.
- No unexplained gaps in traceability.
Finishing testing, meeting exit criteria and authorizing release are separate. See the Quality Report.