Skip to content
Moamen Elmasry
FREN

Free course

Test a product that contains generative AI

Learn to test a product that contains generative AI, from the basics of testing to the evaluation package you hand to the person who decides. By practising: the tests and evaluations run in your browser.

  • Free
  • Modules 01 and 02 open, the rest with your free space
  • 9 modules and a capstone
  • About 14 hours
  • A verifiable certificate
Start module 01
The course in 1 minute, no sound: the path module by module, what happens in a lesson, and the certificate.
What the video shows, shot by shot
  1. Testing a generative AI product can be learned. By practising.
  2. A complete course, free: 10 modules, 43 lessons, 29 skills, about 14 hours.
  3. Part 1: test any product. 01 Testing basics, what a test proves and at which level. 02 Running a campaign, say where a campaign stands. Both open.
  4. Part 2: evaluate generative AI. 03 First evaluation, why an average can lie. 04 Foundations, which gate a behaviour belongs behind. 05 The two gates, what each gate proves.
  5. Then measure, prove, govern. 06 What to measure, 07 Designing the evidence, 08 Running it, 09 Governance.
  6. Capstone: your own evaluation package. Six artefacts and a handover for the decision owner.
  7. Every module ends with a milestone: the module 03 milestone, each skill with its evidence.
  8. In every lesson: predict, run, write, compare. The predicted Fail, the verdict decided by a veto, a written answer then the course's, then "Skill unlocked".
  9. Everything runs in your browser. Nothing is sent.
  10. At the end: a verifiable certificate, for LinkedIn (example in demo mode).
  11. Two open modules: moamen.fr/course.
Who it is for
People who test a product or sign off a generative AI feature before it ships: product owners, business owners, testers, project leads. No programming is required.
What you leave with
A complete evaluation package on a capability of your choice, ready to hand to the person who decides on the release.
What it does not cover
Performance, security and accessibility testing, which each need their own methods; writing automated tests; and validating classic machine learning models.

How you learn

10 runs, 13 graded exercises, 56 questions where you write your answer before you see one, and 14 texts on your own case. Five gestures come back in every lesson.

  1. Before each run, you commit to a verdict: Pass, Fail or Not Ready. A prediction you commit to, even a wrong one, makes the result stick.

    An evaluation in module 03: its editable acceptance rules, then the question "which verdict do you expect?" with Fail selected, and the Run the evaluation button.
    Before the run: Fail is chosen, the run button waits.

What you will be able to do

Your path · 0 of 29 skills

The modules

Open a module to see its lessons and the skills it unlocks.

Part 1 · Testing a product

For any product, with or without AI. Optional if you already know the basics of testing.

  1. 01Testing basicsWhat a test proves, where its expected result comes from, and at which level to prove it.64 min

    Lessons and skills

    • You will be able to take an expected result from a specification, never from what the code does today.
    • You will be able to write an acceptance criterion someone can test, and leave "to confirm" what nobody has decided, naming who decides.
    • You will be able to derive the cases of a criterion: nominal path, equivalence classes, boundary values and errors.
    • You will be able to place each test at the lowest level that can prove it.
  2. 02Running a campaignFrame it, run it, turn failures into defects, and say where it stands without claiming more than was tested.62 min

    Lessons and skills

    • You will be able to read a test report without counting a skipped, errored or flaky test as passed, and give each run the right verdict.
    • You will be able to qualify a failure, write a defect a developer can reproduce, and close it only on a new passing test.
    • You will be able to compute the figures of a campaign and write a status that starts with what is not done.
    • You will be able to report an exit criterion that is not met, with its evidence, and name who may accept the gap.

Part 2 · Evaluating generative AI

What changes when several answers can be right, and how to prove the product is reliable enough anyway.

Already know the basics of testing? Start here, with module 03.

  1. 03First evaluationFree accountRun a real evaluation and watch it fail at 96%, before any theory.25 min

    Lessons and skills

    • You will be able to run an evaluation and read its verdict from the gates, not from the average.
    • You will be able to show why a threshold alone lets a fabricated claim through.
    • You will be able to tell Not Ready from Fail, and say why typing a value to get unstuck is worse than either.
  2. 04FoundationsFree accountWhy AI changes testing, and how to classify what you are looking at.45 min

    Lessons and skills

    • You will be able to classify a behaviour with two questions: can the right output be stated in advance, and was it specified or annotated?.
    • You will be able to split a hybrid capability into its behaviours and state the acceptance rule: DoD and DoT and UAT.
    • You will be able to refuse to put a zero-tolerance behaviour behind a threshold, even 0.9999.
  3. 05The two gatesFree accountWhat the Definition of Done and the Definition of Trustworthy prove, and what they refuse to conclude.83 min

    Lessons and skills

    • You will be able to build and run a DoD gate with no threshold in it, and say why that is the finished answer.
    • You will be able to score a DoT design against its ten components and conclude Not Ready without filling a single gap.
    • You will be able to write the message that holds Not Ready and still gives the team the shortest way to a run.
  4. 06What to measureFree accountIs the number measuring the thing? And when may an AI do the scoring?46 min

    Lessons and skills

    • You will be able to say what a metric cannot see before anyone quotes it.
    • You will be able to compute a judge's calibration, overall and within a subgroup.
    • You will be able to accept a judge as a metric while refusing it behind a veto, and explain the difference.
  5. 07Designing the evidenceFree accountThe reference set, data protection, thresholds, slices and vetoes.117 min

    Lessons and skills

    • You will be able to sort observed failures into degree, subgroup and kind: threshold, slice, veto.
    • You will be able to write an acceptance block where every value you do not own stays a placeholder, and every veto is executable.
    • You will be able to work out the minimum size of a slice from the failure rate the owner calls unacceptable.
    • You will be able to specify a golden set without writing a single reference answer yourself.
  6. 08Running itFree accountFrom a protocol to a run whose evidence survives it.58 min

    Lessons and skills

    • You will be able to turn a brief into a manifest and predict its blockers before running it.
    • You will be able to update the manifest once the owner decides, and explain why a passing average and slice are not enough.
  7. 09GovernanceFree accountWho decides, and how to hold the line when you are asked to move fast.46 min

    Lessons and skills

    • You will be able to name who owns each decision in a release week.
    • You will be able to hold the boundary with a senior person and offer the shortest legitimate path.
  8. CapstoneYour own evaluation packageFree accountEvaluate a capability of your choosing, and assess it against the course rubric.288 min

    Lessons and skills

    • You will be able to produce the six artefacts of an evaluation for a real capability: classification, DoT design, dataset specification, runnable manifest, honest readiness and decisions register.

Free to read. The course content may not be reused without its author's permission.