Module 02 · Running a campaign · 15 min
Run, and read the results
The verdicts a run can get, why a skipped or flaky test is never a pass, and a real pipeline report read by the engine.
Review
From the previous lesson: EX-07 cannot be tested in CAMP-03. What happens to it in the campaign's reports?
The course's answer
It stays in every report, listed as not testable with its reason (no load environment). Dropping it would make the reports look complete when one requirement was never measured.
Six verdicts, not two
A run is not just "passed" or "failed". Each verdict says something different, and mixing them up is how a report ends up claiming more than was tested.
| Verdict | Means | Counts as |
|---|---|---|
| Passed | Every step matched its expected result | Executed, a success |
| Failed | At least one step differed from its expected result | Executed, a failure |
| Blocked | The test could not run to the end: missing data, environment down, a prerequisite missing | Not executed |
| Not applicable | The case does not apply to this release | Executed, set aside |
| Not testable | The campaign has no means to test it | Left out, and listed |
| Settled | It failed, and a person with authority accepted the deviation, in writing | Executed, never a success |
Three rules go with them:
- A manual test is passed by a person, and the run records who. Nobody marks a screen test passed on someone else's behalf.
- A case run on several datasets gets the worst result. Passed on two profiles and failed on the third is failed.
- A flaky test counts as failing until it is fixed. Rerunning it until it passes hides a defect, in the product or in the test.
A report from the pipeline
Automated tests do not get their verdict from a person: the test runner writes a report, usually in the JUnit XML format. The trap is to read only its last line.
Below is the report of ClaimTriage's pipeline for build 2.3.0-142. The engine reads it with the script the Testing Agent uses, which summarises it by level. Look at the raw report if you like, then predict: which level was not tested at all?
The ClaimTriage pipeline's test report
JUnit XML, build 2.3.0-142, as the runner wrote it. The engine reads it with the Testing Agent's script.
Show the raw report
…
Make your prediction first.
What the summary says
Seventeen tests, thirteen passed: a dashboard would show 76% green, or "1 failure". The summary by level says something else:
- Component: failed. One real assertion failure, the critical travel claim routed to
queue-travel, the bug from module 01. And one test skipped, with "flaky, to look at later" as its only reason. - Integration: incomplete. Two tests crashed with
ConnectionRefusedErrorbefore checking anything: the claims-system stub was not running. - System: not tested. Zero tests. Not "passed": not tested. No single sign-on, no permission enforced by the platform, no end-to-end path was checked by this pipeline.
A level with no test is reported as not tested, never as passing. A skipped or errored test is never counted as passed.
Each problem counts as something different
The next lesson takes the failed run further: what turns a failure into a defect someone can fix.