Automated test execution
These three skills take care of the automated part of the flow: generating the executable suite in the project's framework, running it, and automatically repairing what fails for minor reasons —whether based on a real failed run or on a PR diff that broke selectors.
lippia-neo-generate-test-suite
When to use it
When a test plan exists and you need to generate the executable suite in the configured
framework (it reads qa-stack.yaml to choose the adapter: Playwright, Lippia, or Cypress;
Pytest is in stub state and aborts). It supports FE, BE, or hybrid suites.
It also covers alternative modes without a full plan: TDD mode (a single acceptance criterion)
and "scenario expansion" mode (only the missing cases). Key rule: if any executed scenario
failed (status: "fail"), generating its regression test is mandatory and unconditional
regardless of the active mode — the only exception is if a file with the bug-id tag of that
same bug already exists. Before generating each case, it checks whether an existing test already covers the same flow
(precondition, action, and result) so as not to duplicate it. It doesn't write reports: that's done by
lippia-neo-write-generation-report.
{
"config": {
"test_framework": "playwright",
"app_type": "web",
"tests_dir": "tests/",
"test_plans_dir": "test-plans/",
"auth": "basic_auth",
"language": "javascript"
}
}
| Input | Type | Description |
|---|---|---|
config.test_framework, config.app_type, config.tests_dir, config.test_plans_dir, config.auth, config.language |
Required | All resolved from qa-stack.yaml. |
config.env_file |
Optional | Path to the environment's credentials/tokens; if it's null, the templates leave a TODO instead of the real environment variable. |
config.lippia_config_path |
Optional | Only relevant if test_framework is Lippia. |
input.manual_execution_report_path |
Optional | Handoff artifact (ui_selectors, executed_scenarios, findings, summary) produced by any skill that reproduces or explores the app. |
input.tdd_single_ac |
Optional | TDD mode: string with the AC-ID; it generates only that test. |
input.existing_tests |
Optional | Output of lippia-neo-find-existing-tests; if there are matches from the same module, the new files are added to that same folder. |
input.missing_cases |
Optional | Output of lippia-neo-review-test-coverage; if present (even if empty), it enables "scenario expansion" mode. |
Output
| Field | Description |
|---|---|
status |
success / partial (some case skipped) / aborted (adapter in stub state or plan not found). |
generated_files |
path, test_case_id, suite_type (fe/be) of each generated file. |
skipped_scenarios |
Skipped cases, with test_case_id and reason (for example, already covered by an existing test or filtered out by TDD mode). |
warnings |
Pending TODOs or other degradations. |
Chains with
Consumes existing_tests, missing_cases, and manual_execution_report_path from the previous skills; its
output feeds lippia-neo-run-and-heal and then lippia-neo-write-generation-report/
lippia-neo-write-test-report.
lippia-neo-run-and-heal
When to use it
To run the test suite (in any configured framework) and auto-repair the ones that
fail until they all pass or the maximum number of iterations is reached (it reads qa-stack.yaml to
choose the adapter: Playwright, Lippia, or Cypress; Pytest is in stub state and aborts). It never
exceeds max_iterations (cap of 3); before escalating, it completes all
pending TODOs and replaces hardcoded credentials with environment variables — neither
of these is escalated as an excuse; they're always resolved.
{
"config": {
"test_framework": "playwright",
"tests_dir": "tests/",
"reports_dir": "test-results/reports/",
"commands": {
"run_tests": "npx playwright test"
}
}
}
| Input | Type | Description |
|---|---|---|
config.test_framework, config.tests_dir, config.reports_dir, config.commands.run_tests |
Required | |
config.max_iterations |
Optional | Defaults to 3; use 1 for a post-deploy smoke test without healing. |
config.plateau_detection |
Optional | Defaults to true; pass false when max_iterations=1. |
config.run_label |
Optional | Label for the run in the report (post-fix, regresion-modulo, smoke). |
config.include_dirs / config.exclude_dirs |
Optional | Subfolders to include/exclude (Playwright and Cypress, mutually exclusive). |
config.target_tests |
Optional | Specific files to run instead of the full suite (strings or objects with path; e.g. existing_tests from lippia-neo-find-existing-tests or repaired_tests from lippia-neo-repair-selectors-by-diff); takes priority over include/exclude_dirs. Playwright and Cypress — in Lippia it's ignored (it doesn't support scoping by folder or file) and this is noted in escalation_notes. |
Output
| Field | Description |
|---|---|
status |
green / plateau_abort / max_iter_abort. |
iterations_used, final_pass_count, final_fail_count |
|
healed_tests |
file, failure_category (selector_broken/timeout/assertion_mismatch/api_error/environment/unknown), fix_applied, fix_successful. |
escalation_required, escalation_notes |
If human intervention is needed: tests that keep failing, attempted fixes, and a concrete suggestion. |
report_path |
JSON of the run, written to reports_dir. |
Chains with
Runs after lippia-neo-generate-test-suite (or directly on existing tests); when
escalation_required is true, that result is reported to a human via
lippia-neo-write-test-report. It's also step 3 of the ui-selector-repair plan and the only
step of ui-selector-regression, scoped with target_tests to the tests that
lippia-neo-repair-selectors-by-diff repaired.
lippia-neo-repair-selectors-by-diff
When to use it
Right after lippia-neo-find-ui-tests-by-code-diff, in a selector repair flow:
it statically repairs the selectors of the tests returned
in matched_tests, rewriting them based on what the diff itself changed (testing attributes,
aria-label, role, id, route, class, or renamed/removed text). If the diff adds a testing
attribute (data-testid, data-test, data-qa, data-pw) to an element that the test
was looking for by class, text, or named role, it migrates the selector to that attribute ("promotion").
It doesn't run the suite or open a browser — the PR isn't deployed to any environment yet, so
the diff is the only source of truth. It never "guesses" a replacement: if the diff doesn't establish an unambiguous
old→new relationship, the case stays in unresolved for human review.
{
"input": {
"pr_diff_path": "tmp/pr-1234.diff",
"matched_tests": [
{
"path": "tests/checkout.spec.ts",
"suite_type": "fe",
"matched_on": [
{ "term": "apply-coupon", "matched_location": "structured_body", "match_strength": "partial", "match_mode": "exact" }
]
}
]
},
"config": {
"test_framework": "playwright"
}
}
| Input | Type | Description |
|---|---|---|
input.pr_diff_path |
Required | The same diff that lippia-neo-find-ui-tests-by-code-diff consumed. |
input.matched_tests |
Required | The array returned by that skill, never rebuilt by hand; all the terms of each file are evaluated. If it's empty, it returns none without opening anything. |
config.test_framework |
Required | Determines what a selector looks like in the test code (playwright/lippia/pytest; with any other framework, including cypress, it aborts with a warning). |
Output
| Field | Description |
|---|---|
status |
repaired (everything resolved) / partial (there are replacements and also unresolved) / blocked (only unresolved) / none (nothing mechanically repairable). |
repaired_tests |
Per file, changes with selector_type, old_value, new_value, occurrences, and old_selector_type only when there was a promotion (the selector type changed). |
unresolved |
path, matched_on (the term, as a string), reason. |
warnings |
Scope-only matches with nothing to repair, applied selector promotions, traceability of received/evaluated/repaired/unresolved terms. |
Chains with
Step 2 of the ui-selector-repair plan, gated by has_matches from
lippia-neo-find-ui-tests-by-code-diff; unresolved is the signal to escalate to a human. Its
repaired_tests feeds config.target_tests of lippia-neo-run-and-heal (step 3 of the same
plan and, after the deploy, the ui-selector-regression plan).