Skip to content

Automated test execution

These three skills take care of the automated part of the flow: generating the executable suite in the project's framework, running it, and automatically repairing what fails for minor reasons —whether based on a real failed run or on a PR diff that broke selectors.


lippia-neo-generate-test-suite

When to use it

When a test plan exists and you need to generate the executable suite in the configured framework (it reads qa-stack.yaml to choose the adapter: Playwright, Lippia, or Cypress; Pytest is in stub state and aborts). It supports FE, BE, or hybrid suites. It also covers alternative modes without a full plan: TDD mode (a single acceptance criterion) and "scenario expansion" mode (only the missing cases). Key rule: if any executed scenario failed (status: "fail"), generating its regression test is mandatory and unconditional regardless of the active mode — the only exception is if a file with the bug-id tag of that same bug already exists. Before generating each case, it checks whether an existing test already covers the same flow (precondition, action, and result) so as not to duplicate it. It doesn't write reports: that's done by lippia-neo-write-generation-report.

Invocation example
{
  "config": {
    "test_framework": "playwright",
    "app_type": "web",
    "tests_dir": "tests/",
    "test_plans_dir": "test-plans/",
    "auth": "basic_auth",
    "language": "javascript"
  }
}
Input Type Description
config.test_framework, config.app_type, config.tests_dir, config.test_plans_dir, config.auth, config.language Required All resolved from qa-stack.yaml.
config.env_file Optional Path to the environment's credentials/tokens; if it's null, the templates leave a TODO instead of the real environment variable.
config.lippia_config_path Optional Only relevant if test_framework is Lippia.
input.manual_execution_report_path Optional Handoff artifact (ui_selectors, executed_scenarios, findings, summary) produced by any skill that reproduces or explores the app.
input.tdd_single_ac Optional TDD mode: string with the AC-ID; it generates only that test.
input.existing_tests Optional Output of lippia-neo-find-existing-tests; if there are matches from the same module, the new files are added to that same folder.
input.missing_cases Optional Output of lippia-neo-review-test-coverage; if present (even if empty), it enables "scenario expansion" mode.

Output

Field Description
status success / partial (some case skipped) / aborted (adapter in stub state or plan not found).
generated_files path, test_case_id, suite_type (fe/be) of each generated file.
skipped_scenarios Skipped cases, with test_case_id and reason (for example, already covered by an existing test or filtered out by TDD mode).
warnings Pending TODOs or other degradations.
Chains with

Consumes existing_tests, missing_cases, and manual_execution_report_path from the previous skills; its output feeds lippia-neo-run-and-heal and then lippia-neo-write-generation-report/ lippia-neo-write-test-report.


lippia-neo-run-and-heal

When to use it

To run the test suite (in any configured framework) and auto-repair the ones that fail until they all pass or the maximum number of iterations is reached (it reads qa-stack.yaml to choose the adapter: Playwright, Lippia, or Cypress; Pytest is in stub state and aborts). It never exceeds max_iterations (cap of 3); before escalating, it completes all pending TODOs and replaces hardcoded credentials with environment variables — neither of these is escalated as an excuse; they're always resolved.

Invocation example
{
  "config": {
    "test_framework": "playwright",
    "tests_dir": "tests/",
    "reports_dir": "test-results/reports/",
    "commands": {
      "run_tests": "npx playwright test"
    }
  }
}
Input Type Description
config.test_framework, config.tests_dir, config.reports_dir, config.commands.run_tests Required
config.max_iterations Optional Defaults to 3; use 1 for a post-deploy smoke test without healing.
config.plateau_detection Optional Defaults to true; pass false when max_iterations=1.
config.run_label Optional Label for the run in the report (post-fix, regresion-modulo, smoke).
config.include_dirs / config.exclude_dirs Optional Subfolders to include/exclude (Playwright and Cypress, mutually exclusive).
config.target_tests Optional Specific files to run instead of the full suite (strings or objects with path; e.g. existing_tests from lippia-neo-find-existing-tests or repaired_tests from lippia-neo-repair-selectors-by-diff); takes priority over include/exclude_dirs. Playwright and Cypress — in Lippia it's ignored (it doesn't support scoping by folder or file) and this is noted in escalation_notes.

Output

Field Description
status green / plateau_abort / max_iter_abort.
iterations_used, final_pass_count, final_fail_count
healed_tests file, failure_category (selector_broken/timeout/assertion_mismatch/api_error/environment/unknown), fix_applied, fix_successful.
escalation_required, escalation_notes If human intervention is needed: tests that keep failing, attempted fixes, and a concrete suggestion.
report_path JSON of the run, written to reports_dir.
Chains with

Runs after lippia-neo-generate-test-suite (or directly on existing tests); when escalation_required is true, that result is reported to a human via lippia-neo-write-test-report. It's also step 3 of the ui-selector-repair plan and the only step of ui-selector-regression, scoped with target_tests to the tests that lippia-neo-repair-selectors-by-diff repaired.


lippia-neo-repair-selectors-by-diff

When to use it

Right after lippia-neo-find-ui-tests-by-code-diff, in a selector repair flow: it statically repairs the selectors of the tests returned in matched_tests, rewriting them based on what the diff itself changed (testing attributes, aria-label, role, id, route, class, or renamed/removed text). If the diff adds a testing attribute (data-testid, data-test, data-qa, data-pw) to an element that the test was looking for by class, text, or named role, it migrates the selector to that attribute ("promotion"). It doesn't run the suite or open a browser — the PR isn't deployed to any environment yet, so the diff is the only source of truth. It never "guesses" a replacement: if the diff doesn't establish an unambiguous old→new relationship, the case stays in unresolved for human review.

Invocation example
{
  "input": {
    "pr_diff_path": "tmp/pr-1234.diff",
    "matched_tests": [
      {
        "path": "tests/checkout.spec.ts",
        "suite_type": "fe",
        "matched_on": [
          { "term": "apply-coupon", "matched_location": "structured_body", "match_strength": "partial", "match_mode": "exact" }
        ]
      }
    ]
  },
  "config": {
    "test_framework": "playwright"
  }
}
Input Type Description
input.pr_diff_path Required The same diff that lippia-neo-find-ui-tests-by-code-diff consumed.
input.matched_tests Required The array returned by that skill, never rebuilt by hand; all the terms of each file are evaluated. If it's empty, it returns none without opening anything.
config.test_framework Required Determines what a selector looks like in the test code (playwright/lippia/pytest; with any other framework, including cypress, it aborts with a warning).

Output

Field Description
status repaired (everything resolved) / partial (there are replacements and also unresolved) / blocked (only unresolved) / none (nothing mechanically repairable).
repaired_tests Per file, changes with selector_type, old_value, new_value, occurrences, and old_selector_type only when there was a promotion (the selector type changed).
unresolved path, matched_on (the term, as a string), reason.
warnings Scope-only matches with nothing to repair, applied selector promotions, traceability of received/evaluated/repaired/unresolved terms.
Chains with

Step 2 of the ui-selector-repair plan, gated by has_matches from lippia-neo-find-ui-tests-by-code-diff; unresolved is the signal to escalate to a human. Its repaired_tests feeds config.target_tests of lippia-neo-run-and-heal (step 3 of the same plan and, after the deploy, the ui-selector-regression plan).