Add snapshot workflow implementation plan
This commit is contained in:
Dwindi Ramadhana
174
Snapshot_Driven_Workflow_PLAN.md
Normal file
174
Snapshot_Driven_Workflow_PLAN.md
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
# Snapshot-Driven Live Question Workflow Plan
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Rework tryout import so the first imported snapshot creates the live exam automatically, while later imports become non-destructive snapshot candidates inside the existing tryout workspace. Admins then intentionally replace live questions slot-by-slot after reviewing differences. Imported JSON snapshots remain full historical records, but later snapshot accordions hide unchanged rows by default to keep review focused and prevent accidental no-op replacements.
|
||||||
|
|
||||||
|
## Key Changes
|
||||||
|
|
||||||
|
- Main `/admin/tryouts` import creates a new tryout/exam and auto-promotes all valid imported questions from the first snapshot into live `items`.
|
||||||
|
- Existing tryout workspace gets an **Import New Snapshot** action. Importing there creates a full queued snapshot record, but does not change the live exam.
|
||||||
|
- Replace the current single “Imported Snapshot Questions” section with collapsible snapshot sections:
|
||||||
|
- Snapshot 1: baseline source rows plus live/retired status.
|
||||||
|
- Snapshot 2+: review queue showing changed/new/removed/invalid rows by default.
|
||||||
|
- Add a **Show unchanged questions** toggle per snapshot; default hidden.
|
||||||
|
- Snapshot comparison for UI review must compare each snapshot row against the current live slot, not only against the immediately previous snapshot.
|
||||||
|
- If a newer snapshot supersedes an older pending candidate for the same slot, mark the older candidate as `Superseded by Snapshot N`.
|
||||||
|
- Snapshot comparison must use a canonical content hash so "same content" and "changed content" are deterministic across backend, UI, and tests.
|
||||||
|
- Rename the normal state away from “Promoted”. Use lifecycle language:
|
||||||
|
- `Live`: currently served for that slot.
|
||||||
|
- `Pending Snapshot`: imported but not live.
|
||||||
|
- `Changed`: differs from current live slot.
|
||||||
|
- `No Change`: same as current live slot; hidden by default.
|
||||||
|
- `Retired`: previously live, no longer served.
|
||||||
|
- `Stale Variant`: derived from replaced/retired content, excluded from exam delivery.
|
||||||
|
- Keep retired originals and stale variants as operational history, not moved between backend tables. Frontend may visually group retired originals and variants under their snapshot/slot accordion.
|
||||||
|
- Removed slots from later snapshots are warnings by default. They do not change live delivery unless an admin explicitly retires the live slot.
|
||||||
|
- Add **Simulate Exam Flow** to show what the runtime would serve by slot using current live items and selection mode.
|
||||||
|
|
||||||
|
## Backend/API Changes
|
||||||
|
|
||||||
|
- Store every imported JSON snapshot as a full immutable import record.
|
||||||
|
- Add clear state ownership:
|
||||||
|
- `Item` owns original question lifecycle delivery state: `live` or `retired`.
|
||||||
|
- Variants keep their existing review state (`draft`, `approved`, `rejected`) and gain/keep an explicit delivery exclusion state such as `stale`.
|
||||||
|
- Snapshot row review statuses (`Changed`, `No Change`, `New Slot`, `Removed`, `Invalid`, `Superseded`) are computed from snapshot row + current live slot unless persistence is required for audit/history.
|
||||||
|
- Add canonical content hashing for snapshot/live comparison:
|
||||||
|
- Hash normalized question stem, options, correct answer, explanation, media references, scoring metadata, and any fields that affect what students see or how the item is scored.
|
||||||
|
- Ignore import timestamps, snapshot IDs, database row IDs, admin notes, and formatting-only whitespace differences.
|
||||||
|
- Option order should count as content unless the runtime already randomizes options independently.
|
||||||
|
- Add slot identity rules:
|
||||||
|
- Slot number is the primary comparison key.
|
||||||
|
- Slot number must be present for a row to become live.
|
||||||
|
- Slot number must be unique inside one snapshot.
|
||||||
|
- Reordered JSON rows should not matter.
|
||||||
|
- Duplicate slot numbers should block snapshot commit or force an explicit admin resolution flow before commit.
|
||||||
|
- Extend import preview for existing tryout imports to compare full incoming snapshot rows against the current live item per slot:
|
||||||
|
- same content: `No Change`.
|
||||||
|
- changed content: replacement candidate.
|
||||||
|
- new slot: add candidate.
|
||||||
|
- removed slot: latest snapshot no longer contains a previously live slot.
|
||||||
|
- invalid row: missing options/answer or cannot become live.
|
||||||
|
- Add a tryout-scoped import endpoint for “new snapshot inside existing tryout”; it must reject JSON for a different source tryout unless admin confirms title/id/count warnings in the request.
|
||||||
|
- Add a slot replacement endpoint that accepts selected snapshot question IDs and explicit confirmations:
|
||||||
|
- replacing live original for changed slot.
|
||||||
|
- retiring existing variants as stale.
|
||||||
|
- accepting count/title/source mismatch if relevant.
|
||||||
|
- Replacement requests must include stale-preview guards:
|
||||||
|
- `expected_live_item_id`.
|
||||||
|
- `expected_live_content_hash`.
|
||||||
|
- Backend must run replacement in a transaction.
|
||||||
|
- If the current live slot changed after preview, reject and ask the admin to refresh/re-preview.
|
||||||
|
- Backend must reject selected replacements whose snapshot content is the same as the current live slot.
|
||||||
|
- On first snapshot only, auto-create live original `Item` rows for all valid imported questions.
|
||||||
|
- Invalid first-snapshot rows are still stored in the immutable snapshot record, never become live items, and should create an import health warning in the tryout workspace.
|
||||||
|
- On later snapshot replacement:
|
||||||
|
- Always create a new live original revision and mark the previous live original `retired`, even if the current live item has no answers.
|
||||||
|
- Mark variants under the retired original as `stale`.
|
||||||
|
- Exclude `retired` originals and `stale` variants from exam selection.
|
||||||
|
- Add an explicit retire-live-slot endpoint/action for removed slots. Importing a snapshot that omits a slot must never auto-retire the current live slot.
|
||||||
|
- Support restoring an older snapshot slot:
|
||||||
|
- Restored original becomes `Live`.
|
||||||
|
- Current live original becomes `Retired`.
|
||||||
|
- Variant review status remains unchanged while stale/unstale delivery state changes with the parent original.
|
||||||
|
- Previously approved variants under the restored original become servable again.
|
||||||
|
- Draft/rejected variants keep their review status.
|
||||||
|
- Define superseded rules:
|
||||||
|
- Only pending candidates for the same slot can be superseded.
|
||||||
|
- A newer snapshot candidate for the same slot supersedes older pending candidates.
|
||||||
|
- If a newer snapshot matches current live, older pending candidates for that slot become superseded by the newer no-change snapshot.
|
||||||
|
- Already-live or retired items are never marked superseded.
|
||||||
|
- Add migration/backfill for existing tryouts:
|
||||||
|
- Backfill existing imported/promoted questions into a Snapshot 1-style baseline where possible.
|
||||||
|
- Map current user-facing “promoted” live questions to `live`.
|
||||||
|
- Map replaced or older originals to `retired` where the relationship is known.
|
||||||
|
- Verify runtime filters exclude pending, retired, draft, rejected, and stale records after migration.
|
||||||
|
- Preview/replacement API should return a stable shape that the frontend can render without re-deriving lifecycle rules, for example:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
{
|
||||||
|
snapshotId?: string;
|
||||||
|
slotNumber: number;
|
||||||
|
status: "changed" | "new_slot" | "removed" | "invalid" | "no_change" | "superseded";
|
||||||
|
currentLiveItemId?: string;
|
||||||
|
snapshotQuestionId?: string;
|
||||||
|
currentLiveContentHash?: string;
|
||||||
|
snapshotContentHash?: string;
|
||||||
|
warnings: string[];
|
||||||
|
canReplace: boolean;
|
||||||
|
canRetireLiveSlot: boolean;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Primary backend areas: `tryout_json_import.py`, `admin.py`, CAT/session selection filters.
|
||||||
|
|
||||||
|
## Frontend Changes
|
||||||
|
|
||||||
|
- In `/admin/tryouts/{id}/questions`, render snapshots as collapsible groups with slot rows.
|
||||||
|
- First snapshot should look like the live exam baseline, not a manual promotion queue.
|
||||||
|
- Later snapshots should hide `No Change` rows by default and show only actionable review rows.
|
||||||
|
- Add **Show unchanged questions** toggle inside each later snapshot accordion.
|
||||||
|
- Later snapshot row statuses:
|
||||||
|
- `Changed`
|
||||||
|
- `New Slot`
|
||||||
|
- `Removed From Latest Snapshot`
|
||||||
|
- `Invalid`
|
||||||
|
- `No Change`
|
||||||
|
- `Superseded`
|
||||||
|
- Removed-slot rows should show as warnings with an explicit `Retire Live Slot` action, not as automatic changes.
|
||||||
|
- Invalid rows should show the reason they cannot become live, including missing slot number, duplicate slot number, missing options, missing answer, or unsupported content.
|
||||||
|
- Import inside tryout opens a modal:
|
||||||
|
- select JSON.
|
||||||
|
- show preview before committing snapshot.
|
||||||
|
- require checkbox confirmations only for title/source/count mismatch.
|
||||||
|
- Import preview should block commit or require an explicit resolution path for duplicate slot numbers.
|
||||||
|
- Replacement modal requires explicit confirmations only when the action changes live exam behavior:
|
||||||
|
- “I understand this replaces the live question for slot X.”
|
||||||
|
- “I understand existing variants for this slot will become stale.”
|
||||||
|
- Replacement modal should submit the expected live item ID and content hash from the preview so stale-preview conflicts can be detected.
|
||||||
|
- Replace “Open Live Question” with clearer actions:
|
||||||
|
- `Open Live Item`
|
||||||
|
- `Preview Slot`
|
||||||
|
- `Restore This Version` for retired originals.
|
||||||
|
- Add **Simulate Exam Flow** view showing current live delivery order and any skipped/missing slots.
|
||||||
|
- Simulate Exam Flow should also surface import health warnings such as invalid first-snapshot rows, missing live slots, and removed-slot warnings that have not been acted on.
|
||||||
|
|
||||||
|
Primary frontend area: `QuestionManagement.tsx`.
|
||||||
|
|
||||||
|
## Test Plan
|
||||||
|
|
||||||
|
- First import creates a tryout, full snapshot record, and live original items for all valid slots.
|
||||||
|
- First import with invalid rows stores invalid rows in the snapshot, creates live items only for valid rows, and shows import health warnings.
|
||||||
|
- Later import stores the full JSON snapshot but shows unchanged rows hidden by default.
|
||||||
|
- `Show unchanged questions` reveals `No Change` rows.
|
||||||
|
- Snapshot 3 comparison still marks slot #1 as changed if current live is Snapshot 1 content and Snapshot 3 matches Snapshot 2 content.
|
||||||
|
- Backend rejects replacing a live slot with identical snapshot content.
|
||||||
|
- Backend rejects replacement when `expected_live_item_id` or `expected_live_content_hash` no longer matches current live state.
|
||||||
|
- Newer snapshot marks older pending candidate for same changed slot as superseded.
|
||||||
|
- Newer no-change snapshot marks older pending candidates for the same slot as superseded.
|
||||||
|
- Snapshot 2 with changed slot #1 leaves current live item untouched until replacement.
|
||||||
|
- Replacing changed slot #1 retires prior original, marks its variants stale, and makes new snapshot item live.
|
||||||
|
- Restoring Snapshot 1 slot #1 makes that version live again and restores eligible variants.
|
||||||
|
- Restore reactivates only approved variants under the restored original; draft/rejected variants remain non-servable.
|
||||||
|
- Removed slot in a later snapshot does not auto-retire the live slot.
|
||||||
|
- Explicit retire-live-slot action removes that slot from runtime delivery and appears in simulation.
|
||||||
|
- Missing slot number makes a snapshot row invalid.
|
||||||
|
- Duplicate slot numbers block snapshot commit or require explicit admin resolution.
|
||||||
|
- Reordered JSON rows do not produce false changes.
|
||||||
|
- Canonical content hash ignores formatting-only whitespace differences.
|
||||||
|
- Option order changes are detected as content changes unless runtime option randomization makes order irrelevant.
|
||||||
|
- Same file re-import behavior is deterministic and does not create conflicting live state.
|
||||||
|
- Migration/backfill maps existing promoted/live questions and excludes non-live records from runtime delivery.
|
||||||
|
- Fixed exam simulation shows only current live servable items in slot order.
|
||||||
|
- Runtime session next-item endpoint never serves pending, retired, rejected, draft, or stale items.
|
||||||
|
|
||||||
|
## Assumptions
|
||||||
|
|
||||||
|
- First snapshot is trusted as the initial canonical exam and should auto-promote valid questions.
|
||||||
|
- Later snapshots are destructive only when admin promotes/replaces selected slots, not when imported.
|
||||||
|
- Imported JSON snapshots should remain complete historical records.
|
||||||
|
- Snapshot review UI should default to actionable differences only.
|
||||||
|
- Slot number is the primary comparison key for snapshot-to-live review.
|
||||||
|
- Variants belong to the live original version they were generated from.
|
||||||
|
- Historical answers/calibration must remain attached to the exact item version students saw.
|
||||||
|
- “Promoted” should disappear as a normal user-facing state after this workflow is implemented.
|
||||||
Reference in New Issue
Block a user