Automated quality checks

Every rendered course is put through five layers of checks before anyone is asked to look at it. The purpose is not to replace your reviewer. It is to make sure their attention goes on the content rather than on finding a black frame in lesson nine.

More than forty distinct checks run across the five layers, and many of them run once per lesson, so a twenty-lesson course is measured several hundred times. The list below is the whole set by name, not a selection. The result of every one is recorded, so a failure tells you which check, on which lesson, and with what measurement.

Some of these end in a pass or a fail. Others end in a warning, which is a measurement handed to your reviewer rather than a verdict the system acts on. Where a check warns rather than fails, it says so below.

Layer 1: Technical

Will this play, everywhere, without anyone apologising for it?

• 1080p at 30 frames per second, H.264 video and AAC audio.
• Audio and video the same length, to a tenth of a second.
• Decodes cleanly from end to end.
• Loudness inside a normal speech band, so one lesson is not twice as loud as the next.
• No silence longer than 2.5 seconds.
• No black frames.
• Each lesson’s loudness matched against the rest of the course, not just against the standard, so a single quiet lesson is caught.
• The finished course is at least as long as the course promises. This one measures the rendered narration and reports the real words-per-minute rate it implies, which is how a wrong speech-rate assumption gets caught. Warning.
• Lesson lengths do not spread too far apart. Warning.

Layer 2: Fidelity

Does the video say what the approved script said?

• The narration is transcribed back from the rendered audio and compared with the script you approved. Drift is caught here rather than by a learner. This one is scored only in the languages whose transcription floor has been measured, which is not the same thing as a rule about script, and it warns rather than passing silently everywhere else. See producing in another language.
• Every image the lesson planned for was actually generated.
• Images came from the lane that was asked for, rather than quietly substituted from another. Warning.
• Every library asset the plan named is accounted for.
• No text is baked into generated artwork. Text inside an image cannot be corrected, translated or read by a screen reader, so it is rejected rather than tolerated. The reader is Latin-only, so on a non-Latin course this warns and names what it could not read.

Layer 3: Motion and design

Is anything actually happening on screen?

• The screen is never frozen for more than 12 seconds.
• Frames carry content rather than empty space. A slide that is technically animated but visually empty fails.
• Slide changes are timed to the narration rather than to a fixed clock.
• Generated images are at least 1600 pixels wide. Anything narrower raises a warning on the report rather than sending the course back, because a slightly small illustration is a judgement call and your reviewer is better placed to make it.

Layer 4: Voice and pedagogy

Does it read like a person wrote it, and does it teach?

This is the largest layer by some distance. It is grouped here for reading, but every line is a separate recorded check.

How it is written

• Machine-writing tells are rejected: the stock phrases, and the em-dash habit that gives generated prose away at a glance. Applied to the scripts, the lesson openings, the workbook and the words on the slides. This one is English-specific and does not run on other languages, which it says on the report rather than passing them silently.
• No markdown symbols left in text that gets spoken aloud. An asterisk read out loud is unmistakable. This one runs in every language.
• A plain-language check on sampled narration, because material written for a specialist reader does not become a course simply by being read aloud.
• One agreed spelling per idea, held across every lesson, slide and question.
• The narration does not simply read the slide back to the learner. Warning.
• Slide text stays inside its word caps.

The assessment

• Quiz shape: four options each, correct answers spread across at least three of the four positions rather than clustered, and no question asked twice.
• A question count sized to the course length, at 3 minutes of course per question, with a floor of 8 questions and a ceiling of 40. That ceiling binds sooner than most buyers expect: a two-hour course earns 40 and gets 40, and everything longer than two hours gets 40 as well. A 300-hour programme earns 6,000 and still receives 40.
• Whether the ceiling flattened this course’s quiz is measured and reported as its own warning, with the earned figure beside the given one, so a capped course is never reported as a correctly sized one. If you are commissioning something long, read that line: it is the difference between a final quiz and an assessment, and a long programme needs an assessment structure we will design with you rather than a longer list.
• Every quiz question answerable from the lesson it belongs to.
• Objectives are stated and assessable, each with a real condition and criterion.
• The quiz assesses the objectives, question by objective.
• The number of objectives follows the amount of material rather than a fixed target.

Whether it teaches

• Each lesson shows Merrill’s marks: a problem, activation, demonstration, application, integration.
• Every lesson is grounded in the source material you supplied.
• Facts and numbers in the script are verified.
• Worked-example arithmetic is checked.
• Lessons are substantive rather than padded out to length. Warning.
• How much new vocabulary a lesson introduces at once. Warning.
• Each lesson opens from the one before it rather than starting cold. Warning.
• What kinds of lesson the course is made of, so it is not twenty of the same shape.
• Pictures are spread evenly across the course, and how many slides carry one at all.
• Images requested, served and lost, counted end to end.
• The visual variety budget after any downgrades.
• No wording copied from your source material.
• The syllabus is not simply your source document’s own table of contents reordered. Warning.
• The plan a person approved, compared against what was actually written.

Layer 5: Package completeness

Is the whole course actually there?

• Every lesson in the confirmed course has a rendered video.
• The course opening is present.
• No recorded artefact is missing or empty.
• The course cover is generated at 750 by 422 pixels.
• The student workbook is present and substantial.
• The listing fields are inside their length limits.

What happens when a gate fails

A hard failure does not reach you. The course goes back rather than forward.

Where the fault is in the render, such as a frozen frame, a bad encode or narration that drifted, the course is re-rendered.

Where the fault is in the text, re-rendering cannot fix it, and the course goes back to the document for a human to edit. A quiz question nobody could answer from its lesson, a workbook that is too thin, a listing field that is too long: all of these need a person, and the system says so rather than looping the same document past the same gate forever.

Checks marked as warnings above produce a measurement rather than a failure, because the right answer depends on context a machine does not have. Warnings are shown to your reviewer, not acted on automatically.

What they do not check

Stated plainly, because a list of checks invites the assumption that everything else is covered:

• Whether the content is correct. The checks verify that the video says what the approved script said. Whether the script is true about your organisation is your review, at the document stage.
• Whether you were entitled to use the material. The copied-wording check finds reproduced sentences. It cannot know what rights you hold.
• Lip sync, where an avatar presents a lesson. It is on the human checklist deliberately.
• Whether the course is any good. No automated gate can tell you that.