SCORM conformance testing¶
The offline suites cannot answer whether a real LMS accepts this package or agrees with how it reports learner state. That needs a live run, and it is the last unverified thing about the SCORM path.
Two halves, and only the first can be automated.
1. Package validation — automated¶
Tests~/tools/validate-package.py checks a generated .zip against what an importer inspects before it will accept the package: manifest well-formedness, namespace declarations, identifier uniqueness, identifierref resolution, scormtype/scormType casing, URI-safe hrefs, declared-versus-present file parity, that no schemaLocation names a file the package omits, the launch file, and the injected page helper.
CI runs this on every push, against packages the EditMode suite generates through the shipped packaging code (ScormBuildSmokeTests), and uploads them as the scorm-packages artifact. That is deliberate: the figure quoted here used to come from a local Builds/ folder that is not in the repository, so nobody could reproduce it from a clone — and with no arguments the script printed "0/0 checks passed" and exited 0, which in CI is indistinguishable from a clean run. It now exits non-zero when it validates nothing.
This covers the checks an importer applies before it will accept a package, which is where most import failures come from. It is not schema validation and not the ADL Test Suite: it says nothing about runtime behaviour, and it does not validate the manifest against the official control documents.
Schema validation — checked by hand, 2026-10-01¶
Done once, directly, rather than left as an assumption. Both manifests were taken from packages ScormBuildSmokeTests generated through the shipped packaging code, and validated against the official control documents:
| Manifest | Validated against | Result |
|---|---|---|
| SCORM 1.2 | adlcp_rootv1p2.xsd, which imports imscp_rootv1p1p2.xsd | valid |
| SCORM 2004 | imscp_v1p1.xsd | valid |
Load the IMS content-packaging schema alone and the 1.2 manifest reports one error — adlcp:scormtype unresolvable against a strict wildcard. That is the ADL extension schema being absent from the validator, not a defect in the manifest; entering through adlcp_rootv1p2.xsd, which imports the IMS schema, resolves it and the manifest validates. Worth knowing before repeating this, because the failure looks like a real one.
Why this is not automated. Running it in CI means either shipping the control documents or fetching them, and neither is clean. The ADL files are US Government work, but the IMS ones are 1EdTech copyright whose notice grants excerpts for requests for proposals and refers product use to a separate licence — not a redistribution grant a commercial package should assume. Fetching instead does not work either: imsglobal.org serves its schema, but the ADL namespace URLs return 500 or nothing, so the step would fail for reasons unconnected to the code, which is the kind of check people learn to ignore.
The value lost is small. The manifest is generated from a fixed template with a handful of substitutions, so the failures schema validation would catch are the ones ScormManifestTests and the validator above already cover. This entry exists so the result is on record rather than repeated.
2. Runtime conformance — manual¶
Needs a SCORM Cloud account (the free tier is enough) and a browser. Upload each package, launch it, drive the validation panel, then read the registration's Debug Log — that is where the actual LMSSetValue / SetValue traffic and any error codes appear.
Test both packages. SCORM 1.2 and 2004 use different API objects, different element names and different data models; a pass on one says nothing about the other.
Use a platform other than the one any previous validation ran on, so the result is evidence about the product rather than a repeat of a known-good pairing.
Editor coverage¶
The packages under test can come from either verified editor. Both have been built and packaged end to end:
| Editor | Suite | WebGL build | Package | Validator |
|---|---|---|---|---|
| 2021.3.45f2 | 320/320 | yes | yes | 23/23 |
| 6000.3.5f2 | 320/320 | yes | yes | 23/23 per package |
Preconditions¶
- Both CI jobs green on the commit under test.
- Both packages built through the packer, with Decompression Fallback on.
- Both passing
validate-package.pyat 100% with no warnings.
What to look for¶
Ordered by how much it would cost to get wrong. Every step maps to a control that exists in the shipped validation panel — several of them had to be added, because the checklist previously asked for things the panel could not do.
| # | Do this | Expect | Guards |
|---|---|---|---|
| 1 | Launch the SCO | Session initializes; log shows LMSInitialize/Initialize returning true | API discovery, version match |
| 2 | Read Launch Context | Entry, Credit and Mode reported. PassingScore is null | The in-box packer emits no mastery score, so this is a documented limitation — confirm it rather than assume it |
| 3 | Write Score, Read Score | Values round-trip. On 2004, cmi.score.scaled is written alongside raw/min/max | M-5 |
| 4 | Set Passed, then Set Completed | The pass survives. 2004: success_status stays passed. 1.2: lesson_status stays passed | The pass→completion rule, on both standards. 1.2 used to overwrite it with completed, following the finish sequence the FAQ recommends |
| 5 | Set Failed, Read Status | Fail recorded; completion follows the configured failedImpliesCompleted policy | Status axes |
| 6 | Write Objective Score | Score elements only. No success_status, no completion_status, no objective success | C-1 — a score of 20 was once recorded as a pass |
| 7 | Write Objective Success, then Write Objective Completion, then Read Objective | Each writes its own element. On 1.2, completion warns and writes nothing | C-1, and the 1.2 objective-completion limitation |
| 8 | Write Interaction (correct) | type is choice on both standards; result is correct; no 405 on the type write | multiple_choice was written on 1.2 — a token in neither standard |
| 9 | Write Interaction (incorrect) | result is incorrect on 2004 and wrong on 1.2. No 406. The instructor report shows the answer judged wrong, not unjudged | The most important step here. Every incorrect answer used to lose its result on a conformant 2004 LMS while correct answers recorded fine |
| 10 | Write Interaction (correct) again, then Read Interaction | One record per id, updated in place — not two appended | Interaction dedupe |
| 11 | Write Out-Of-Range Score (150 of 0–100) | Refused, and the console names the right meaning for this standard | M-1 — the error tables differ per standard |
| 12 | Write Suspend Data, Commit | Accepted; no 405; the commit returns true and appears in the log | H-1, and commit is the persistence moment |
| 13 | Exit = Suspend, Finish; relaunch; Read Launch Context, Read Suspend Data | cmi.entry reads resume; bookmark and suspend data restored intact | #19, suspend/resume |
| 14 | After that resume: Write Interaction (incorrect), then Read Interaction | The write succeeds and the index is correct | On platforms that journal interactions per session, the 1.2 local cache is not reconciled against cmi.interactions._count — a failure here is a known issue to fix, not a surprise |
| 15 | Exit = TimeOut, Finish | cmi.exit accepts time-out on both standards. No 406 | The 2004 runtime wrote timeout; this was the one open question in the SCORM path and is now resolved from the specification. Confirm it |
| 16 | Exit = Logout, and Exit = Normal (separate registrations) | Both accepted. Normal writes "", which ends the attempt | Exit vocabulary |
| 17 | Finish, then press any write button | Writes refused locally and never reach the LMS. No 133/301 in the log | H-9 |
| 18 | Set some state, then close the tab without pressing Finish | The log shows a Commit at unload; progress retained on relaunch | H-2, and the page helper is the only thing that does this |
| 19 | Complete normally, Finish | Terminate/LMSFinish returns true; the registration shows complete with the expected status and score | End to end |
Out of scope¶
Multi-SCO packages, sequencing and navigation, attempt management, and the xAPI State and Profile APIs. All four are documented as not covered. The xAPI path is validated separately with XApiValidationRunner against a real LRS.
Recording the result¶
Fill this in. A conformance run that nobody wrote down has to be done again.
| Field | SCORM 1.2 | SCORM 2004 |
|---|---|---|
| LMS platform | SCORM Cloud (Rustici Engine 22.38.484) | SCORM Cloud (Rustici Engine 22.38.484) |
| Standard as imported | SCORM 1.2 | SCORM 2004 3rd Ed. |
| Manifest parser | "Congratulations, your manifest looks great!" | "Congratulations, your manifest looks great!" |
| Package | scorm12.zip, 4.7 MB, 18 entries | scorm2004.zip, 4.7 MB, 18 entries |
| Built with | Unity 6000.3.5f2, Gzip + decompression fallback | Unity 6000.3.5f2, Gzip + decompression fallback |
validate-package.py | 23/23 | 23/23 |
| Launch configuration | content frame; API found via the parent chain | content frame; API_1484_11 found via the parent chain |
| Date | 2026-10-01 | 2026-10-01 |
Outcome: passed, with two defects found and fixed during the run. Both are recorded below, because a conformance run that only records its passes is not evidence of much.
Results¶
| # | SCORM 1.2 | SCORM 2004 |
|---|---|---|
| 1 Launch and initialize | pass | pass |
| 2 Launch context | pass — Entry AbInitio, Credit Credit, Mode Normal, PassingScore (not reported) | pass — entry read directly |
| 3 Score | pass — read back 85, LMS reported 85.00% | not run |
| 4 Pass then completed | pass — status stayed Passed; LMS recorded Success passed, Completion complete | pass (guarded by the same rule) |
| 5 Set failed | not run | not run |
| 6 Objective score only | pass | pass — score.raw 85, success_status unknown |
| 7 Objective success and completion | pass — completion refused with the documented message | pass — completion_status completed |
| 8 Interaction, correct | pass — type=choice, result=correct | pass — stored as choice / correct |
| 9 Interaction, incorrect | pass — result=wrong | pass — stored as choice / incorrect, no 406 |
| 10 Re-answer | pass | pass — three writes of one id, _count stayed 2 |
| 11 Out-of-range score | pass — refused locally, LMS score stayed 85.00% | not run |
| 12 Suspend data and commit | pass | pass — Commit -> True |
| 13 Suspend, relaunch, resume | not run | pass — cmi.entry = resume, suspend_data byte-identical, interactions and objective intact |
| 14 Post-resume interaction write | not run | pass — updated in place, no duplicate at index 2 |
15 Exit time-out | not run | pass — accepted, no 406 |
| 16 Exit logout and normal | not run | pass — both accepted |
| 17 Writes after finish | not run | not run |
| 18 Commit on tab close | not run | not run |
| 19 Finish | pass — registration Satisfied true, Completed true, Attempts 1 | pass — Finish -> True |
Read directly from the LMS data model at the end of the SCORM 2004 run, every read returning error 0:
cmi.objectives._count 1
cmi.objectives.0.score.raw 85
cmi.objectives.0.success_status unknown <- a score write asserts nothing
cmi.objectives.0.completion_status completed
cmi.interactions._count 2
cmi.interactions.0.id question_2
cmi.interactions.0.type choice
cmi.interactions.0.result incorrect <- the 2004 spelling, accepted
cmi.interactions.1.id question_1
cmi.interactions.1.type choice
cmi.interactions.1.result correct
cmi.entry resume
cmi.suspend_data {"checkpoint":"ValidationCheckpoint","score":85}
What the run found¶
The diagnostics proxy was emptying LastError. The injected page helper built its proxy with Object.create(api) and gave it own properties only for the calls it wraps, so everything else ran with this bound to the proxy. SCORM Cloud's adapter keeps its error state in properties on this, so a single GetValue through the proxy gave it own copies of ErrorNumber, ErrorString and ErrorDiagnostic which then shadowed the adapter's for the rest of the session. Measured: the adapter holding 351 while the proxy answered 0.
ILmsRuntime.LastError and DescribeLastError() are documented public API that the README and FAQ both tell customers to check, and they returned "no error" for the whole session on this platform, in the default configuration. Every message the package produces that decodes that value said "OK" too. No offline test could have caught it: the JavaScript harness's adapter kept state in a closure, which is reachable whatever this is. The harness now has two adapters that keep state the way real ones do.
Re-answering a question failed on SCORM 2004. correct_responses is a collection and SCORM Cloud refuses a write to an index that already holds a pattern, with 351. Since recording an interaction updates the existing record in place, every re-answer hit it. The pattern is now written only when the record is created, which is also the only time it can be meaningful.
Both were found because this branch started checking the result of every write. Before that, both failed silently.
Not covered by this run¶
Steps 17 and 18 on either standard, and 13 to 16 on SCORM 1.2. Step 18 in particular — commit on tab close — is the one guarantee with no automated equivalent, and it remains unverified against a live LMS.
The SCORM 1.2 interaction-cache reconcile (the local mirror is not checked against cmi.interactions._count on resume) stays deferred: step 14 passed on SCORM 2004, where the index comes from the LMS, and the 1.2 path was not exercised after a resume. A failed element write is now reported as a warning either way, so the failure mode is loud rather than silent.
3. Data-model enforcement — measured¶
A second, narrower run on 2026-10-01, probing the data model directly rather than through a course. The point was to establish what a strict SCORM 2004 LMS actually accepts, because three of this package's design decisions rest on the answer and none of them is settled by reading the specification alone.
Platform: SCORM Cloud, SCORM 2004 3rd Edition. Method: API_1484_11 called from the player window, each write followed by GetLastError(), GetErrorString() and GetDiagnostic().
| Write | Result | Error |
|---|---|---|
cmi.completion_status = completed | accepted | 0 |
cmi.completion_status = passed | rejected | 406 Data Model Element Type Mismatch |
cmi.completion_status = failed | rejected | 406 Data Model Element Type Mismatch |
cmi.success_status = passed | accepted | 0 |
cmi.score.scaled = 0.85 | accepted | 0 |
cmi.interactions.0.id, .type, .result = incorrect | accepted | 0 |
cmi.interactions.1.result = wrong | rejected | 406 Data Model Element Type Mismatch |
The LMS's own diagnostic on both rejected status writes, verbatim: "The completion_status data model element must be a proper vocabulary element."
The first and fourth rows are the controls. Writes to the same element, and to the neighbouring one, succeeded in the same session, so the rejections are the vocabulary being enforced rather than a broken session.
What this establishes¶
Completion and success are not interchangeable on 2004. passed and failed belong to cmi.success_status, and cmi.completion_status refuses them. An implementation that folds the two 2004 axes back into SCORM 1.2's single lesson_status cannot report a pass on 2004 at all — the write is refused at the point of the call, before any question of persistence arises. This is why IScormDataModel carries CompletionStatus and SuccessStatus as separate values rather than projecting one onto the other, and why a SCORM 1.2 Completed write is suppressed once success is on record instead of being routed through a shared axis.
The interaction vocabulary genuinely differs by standard. wrong is valid 1.2 and is refused by 2004, which wants incorrect. ScormStatusMap.InteractionResult translates for precisely this reason. Without it, a course written against 1.2 vocabulary loses every interaction result on 2004 — and loses it quietly, because the rejection is visible only to a caller that checks.
A refused write is invisible unless the API surfaces it. Every rejection above returned false and set error 406, with a diagnostic naming the problem. An API whose setters return a bare boolean and expose no error accessor leaves the caller unable to tell "refused" from "refused for a reason you could fix", and the diagnostic reaches only the browser console — which nobody is watching on a deployed course. ILmsRuntime.LastError and DescribeLastError() exist for this, and section 2 records what happened when they were themselves broken.
Caveat¶
These are validation results at the point of the call, which is all the three conclusions above rest on. The probe drove the API directly without a SCO loading, and the sandbox registration summary read unknown both before and after, so nothing here is evidence about persistence or rollup. Section 2 covers that, through a real course.