Automated Testing

instruqt lab test plays a lab for you. It starts a real session and works through the lab the way a user would: it opens every page, checks and solves every task, and answers every quiz from the answers in the lab’s own configuration. It stops at the first thing that misbehaves and exits non-zero, so you can run it by hand before sharing a lab or in CI on every change.

Unlike instruqt lab validate, which reads your HCL on disk, this runs the lab on the platform. It tests the version the platform has, not the files in your working directory.

Prerequisites

• The Instruqt CLI installed
• Authentication: instruqt auth login, or INSTRUQT_TOKEN for CI
• A lab on the platform, synced from your repository

The lab command group does not appear in root instruqt --help, but the subcommands work when you invoke them as instruqt lab ….

Run a test

# A lab in your configured team
instruqt lab test my-first-lab

# A lab in another team
instruqt lab test acmecorp/my-first-lab

With a bare lab slug, the team is the one you chose at instruqt auth login. Pass <team>/<lab> to test a lab in any team you have access to.

Interrupting the CLI with Ctrl-C stops the CLI, not the run. The run finishes on the platform and stops its own session, and the command prints the session and run IDs so you can follow it with instruqt lab logs.

What a run does

A run starts a session of your lab, waits for the sandbox to come up, and loads the lab. Then it works through the content from the first page to the last, doing what a user would do:

• Opens each page, and confirms the running lab has the tasks and quizzes your configuration says belong there.
• Checks each task before solving it. The check is expected to fail here.
• Solves each task, which runs its solve scripts and unlocks whatever comes next.
• Answers each quiz using the answers in your configuration.

When it reaches the end — or stops early because something misbehaved — it shuts the session down.

Why a task’s check has to fail first

A check that passes on a task nobody has touched cannot tell a finished task from an unstarted one, so every user would watch that task complete itself. Running the check first is how the test catches it, and a check that passes there fails the run with the check passed on an untouched task.

Reading the report

The run reports each step as it happens. In a terminal, the step in progress shows a spinner, which is replaced by its result when the step finishes.

instruqt lab test acmecorp/http-basics

==> Testing lab: acmecorp/http-basics
    Starting session                          OK  session hfwc8re245hc
    Waiting for sandbox                       OK  2m3s
    Loading lab                               OK
  Making HTTP Requests and Reading Headers with curl
    Page 1 of 4: Send a GET Request
      Visiting page                           OK
      Checking task "fetch-homepage"          OK
      Solving task "fetch-homepage"           OK
    Page 2 of 4: Inspect Response Headers
      Visiting page                           OK
      Checking task "save-headers"            OK
      Solving task "save-headers"             OK
      Answering quiz "header-knowledge-check" OK
==> Lab test passed
    4 pages, 4 tasks, 2 quizzes in 2m13s

• Chapters and pages are the titles a user sees, and tasks and quizzes are named as you named them in your configuration.
• The session ID appears next to the step that created it. Use it with instruqt lab logs --session while the run is going or after it ends.
• A step that takes longer than a few seconds shows how long it took.
• When output is not a terminal — a pipe, a CI log — each line is printed once, when the step finishes, with no spinner and no redrawing.

Testing a branch

By default the run tests the lab’s default branch. Point it at another branch or a tag with --ref:

instruqt lab test my-first-lab --ref my-feature-branch

When a test fails

The failing step is the last line of the report, with the reason underneath it. The command exits 1.

instruqt lab test acmecorp/http-basics

==> Testing lab: acmecorp/http-basics
    Starting session                          OK  session warhjw01eg4g
    Waiting for sandbox                       OK  1m53s
    Loading lab                               OK
  Your First curl Request
    Page 1 of 4: Send a GET Request
      Visiting page                           OK
      Checking task "fetch-homepage"          OK
      Solving task "fetch-homepage"           FAIL
    its solve scripts failed: cannot reach the target server
    [ERROR] Solving task "fetch-homepage" on Send a GET Request: its solve scripts failed: cannot reach the target server
    Logs: instruqt lab logs --session warhjw01eg4g
    Run:  labtest-0eb8931f-b4c5-473e-8d82-7395f0712f5a

The message under the failing step is what the lab itself reported, so start there. For anything a script printed, follow the session ID into lab logs.

Running in CI

Set INSTRUQT_TOKEN in the environment and run the command; the exit code is the result.

instruqt lab test acmecorp/http-basics --ref "$GIT_BRANCH"

• Exit 0 when the run passed, 1 when it failed or could not start.
• Output has no spinner or escape sequences when stdout is not a terminal.
• A run takes as long as the lab does — mostly the sandbox coming up — so allow several minutes.

One run per lab at a time

Starting a second test of the same lab replaces the first one’s session, and the first run then fails with session is no longer observable. Let a run finish before starting another of the same lab.

What it does not test

• That a check can pass. The check runs only before the task is solved, and solving marks the task done without running the check again. A check that can never pass is not caught.
• That a wrong answer is rejected. Quizzes are only ever answered correctly.
• Cleanup scripts, and anything that only happens when a user does something unexpected.
• Your local files. The run tests what the platform has for the branch or tag you named. Push your changes first.

Related

• Testing & Validation — format, validate, and read lab logs
• Interactive Tasks — check and solve scripts
• Task reference — task and condition configuration
• Quiz reference — quiz and question configuration