Lecture video: beta acceptance tests

A new lecture video reached Bunny and played with English captions. Admin model saves, resets and stale-write rejection also passed.

SP-679 | Live tests: 16 and 17 November 2026 | Report: 17 November 2026

Status as of 17 November 2026

Requested beta follow-up checks passed. Invalid model IDs, persisted transcript metadata and saved-model behavior across a setting change and retry are now verified through the live API. Provider failure cases passed local mocked checks, as requested. This report does not claim exhaustive acceptance or production verification.

These screenshots and this generated video come from the real beta LMS, using a synthetic three-scene lesson in an isolated QA course. This is not a local mock demonstration.

PR #622, merged | Successful staging admin deployment | Jira SP-679

Why these checks matter

The admin and runner were blocked before reaching backend authentication. After the authorized Cloudflare configuration change and beta routing updates, the admin could manage model choices and the runner could complete a fresh generation. Delivery, captions and playback were checked separately.

Upload

Synthetic XLSX and three-slide PPTX.

→

Review

English detected. Instructor confirmed US English.

→

Generate

Six speech beats and a new Bunny delivery.

→

Play

22.87 seconds, captions and seeking checked.

The new generated video

This is the 480p stream exported from the newly delivered Bunny video. It contains the generated narration. It is output evidence, not a recording of the complete UI test session.

Live player proof

New beta video paused on the Explain the idea slide with matching English captions.
Actual Chrome capture from beta. Seeking to the middle of the new video displays the matching English caption over the slide.

Acceptance checks and results

Live observations, local automated checks and unverified cases are listed separately.

CheckResultEvidence and scope
Admin accessLive passConfig loads through the Cloudflare beta route.
Catalog default and choicesLive passOpus 5.5 is the default. The editor offers the two approved exact IDs.
Global save and resetLive passSaved Opus 5, saw the global override, then reset to Opus 5.5.
Tenant override and resetLive passSaved a tenant override. Reset preview showed the inherited model. The tenant returned to Default / Opus 5.5.
Stale editorLive passA second tab saved a newer revision. The older tab was rejected with HTTP 409.
Anonymous and invalid sessionsLive passBoth direct staging-admin catalog probes returned HTTP 401.
Upload, profile and language reviewLive passThree-scene XLSX/PPTX accepted. English detected. The job waited for instructor confirmation before audio.
New Bunny uploadLive passNew job delivered successfully. The resumed Cloud Run execution completed in 93 seconds.
New captionsLive passEnglish WebVTT has six cues matching the synthetic script. The instructor list reports captions attached.
Playback and seekingLive passChrome played to the end of the 22.87-second video. Seeking changed the slide and displayed the matching English caption.
Reloaded delivery stateLive passDelivered and captions-attached state remained after a full page reload.
Failed script profilingLive passA one-scene negative fixture failed the minimum-scene self-check after four bounded attempts. The UI showed failure and retry; the course slot was released.
Transcript database persistenceLive passAuthenticated beta API returned transcript_ready=true, en-US, lecture_script and six segments. The same metadata and saved timestamp remained after a full page reload.
Provider denial, retirement and outageAutomated passFive existing runner tests passed again. Seven additional local mocked checks passed: 401 credentials, 403 access denial, 404 missing model, 429 rate limit, 529 outage, connection timeout and Retry-After beyond the deadline. Live provider fault injection is outside the agreed scope.
Backend model contractsAutomated pass25 backend unit tests passed. These are local tests, not live beta fault injection.
Invalid model IDs through live APILive passSix authenticated tenant writes returned HTTP 400 video_model_invalid: alias, typo, wrong case, trailing whitespace, unknown ID and null. The inherited setting remained unchanged.
Saved model through edits/retriesLive API passJob A retained default Opus 5.5 after the tenant changed to Opus 5 and on retry. Newly confirmed job B received tenant Opus 5. A controlled worker read the backend snapshots and stopped before generation. Both QA jobs are terminal and the tenant override was removed.
Production generationNot runNo production render or production admin backend change was made.

Saved model lifecycle on beta

These checks used the real beta confirm, configuration, retry and internal worker APIs. A controlled test worker claimed the dedicated QA jobs before the automatically dispatched runner. It read the saved model and then reported an intentional test failure before provider calls, audio, rendering or Bunny upload. This is an API lifecycle test, not a second full video rendering test.

ActionObserved saved model
Confirm job A with no overrideclaude-opus-5-5, source default
Change tenant setting to Opus 5; read job A againclaude-opus-5-5, source default
Retry job Aclaude-opus-5-5, source default
Confirm new job Bclaude-opus-5, source tenant
CleanupBoth QA jobs stopped; tenant override removed; default Opus 5.5 restored.
Staging admin model row after cleanup, showing Default and claude-opus-5-5.
Actual staging-admin capture after cleanup. The model row shows Default and claude-opus-5-5.

Provider failure test scope

As requested, credential denial, model access denial, rate limits, outages and timeouts are checked with local mocked responses. Credential and access failures make one request. Persistent rate limits, outages and connection timeouts stop after three requests. A Retry-After delay beyond the remaining deadline is rejected after one request. No shared beta provider credentials were changed.

Known limits and follow-up

Evidence records

Successful job: TozDf_OA-KwLbo_wRoWpg. Video: c5622021-803e-4a6d-ba43-903bfb3a9e4a. Final execution: lecture-video-runner-beta-4sdr6.

The exported MP4 fully decoded without errors. It is 1,661,875 bytes, 854 x 480, H.264/AAC. The English VTT is 568 bytes with six cues.

Download generated video | Download English captions | Structured results | Invalid-ID responses | Transcript persistence | Saved-model snapshots | Lifecycle cleanup | Provider unit checks

How to use the feature

  1. In staging admin, open Config and search for lecture. The model editor offers the approved choices and shows the source of the effective value.
  2. For a tenant override, open Tenant overrides from the model row. Reset previews the inherited value before removing the override.
  3. In the QA course, upload a script and slides. Confirm the narration language before generation continues.
  4. Wait for delivery and caption attachment, then check playback. Caption attachment can finish after the delivered status first appears.