Accessibility and performance budgets

AccessibilityCopy Anchor Link

The @a11y tagCopy Anchor Link

Tag a UI or hybrid scenario @a11y and, when the scenario ends, SDODS audits the page it finished on with axe-core through @axe-core/playwright. The audit covers WCAG 2.0, 2.1 and 2.2 level A and AA rules, and nothing else.

@ui @inventory @a11y
Feature: Inventory accessibility

  @regression
  Scenario: The inventory page has no serious accessibility violations
    Given I am on the login page
    When I login with "standard_user" and "{{standardPassword}}"
    Then I should be on the inventory page
    And there should be 6 products listed

The scenario fails when axe reports any violation at or above a11y.failOn. Either way, the full axe result is attached as sdods/a11y-scenario.

a11y:
  failOn: serious # minor | moderate | serious | critical; the default is serious
  include: [] # limit the audit to these selectors
  exclude: ['#third-party-chat'] # leave these selectors out
  scope: final # final | every-page; the default is final

An environment file can override single keys, for example a11y: { failOn: critical } while a team works through its backlog.

Every page a scenario visitsCopy Anchor Link

By default the tag audits the page the scenario ends on. A flow that goes login → inventory → cart audits only the cart. Set scope: every-page to audit every distinct URL the scenario reaches:

a11y:
  scope: every-page

• Each URL is audited after the step that reached it. The audit waits until the page has settled at the same point the @perf tag reads a page: the load event, the first contentful paint, then a short settle window. Nothing else drives the page between two steps, so an audit never races the next click.
• Each URL is audited once. A later step on the same URL does not audit it again. The page the scenario ends on is audited as it stands at the end, as with final.
• Violations fail the scenario once, at its end. The error lists the violations under each URL that has them. The sdods/a11y-scenario attachment carries a pages list with each URL, its status and its own axe result, and the a11y process gate counts one audit per URL.
• A URL the scenario left inside a single step (a redirect, or one step that clicks through two pages) is listed as not-audited, so the gap is visible. It is not a failure.

What the tag does not do:

• With scope: final, it audits only the final page. To check a page in the middle of a flow, set scope: every-page, or add an explicit step after navigating to it.
• It skips a scenario that already failed. The page that scenario stopped on proves nothing, and a list of violations on top of the error would only bury it. The attachment records the skip.
• It does not audit a URL twice. If a whole-page audit step already ran on a URL, the tag leaves that URL to the step's threshold, even when the step ran after the tag's own audit.
• It fails when nothing can be audited. A scenario that never navigated fails. So does an include selector that matches nothing, or an audit in which axe examined no rule at all.

Automated rules cover only part of WCAG, so a passing audit is a floor, not a certificate.

Accessibility stepsCopy Anchor Link

The steps give you finer control than the tag:

Each axe step attaches its result as sdods/a11y/<step>.

PerformanceCopy Anchor Link

BudgetsCopy Anchor Link

Budgets are set in configuration, never in the feature file. An environment overrides single keys.

perf:
  budgets: { pageLoadMs: 4000, lcpMs: 3000, apiP95Ms: 1500 }

perf:
  budgets: { pageLoadMs: 5000 } # looser on staging

The @perf tagCopy Anchor Link

While a UI or hybrid @perf scenario runs, SDODS records the vitals of every document the main frame loads. Each document reports once it has loaded, painted and let largest contentful paint settle. When the scenario ends, SDODS compares every recording with every configured page budget, and the p95 of the scenario's API calls with apiP95Ms. API calls are timed in every layer, so an @api @perf scenario is judged on apiP95Ms alone.

@ui @cart @perf
Feature: Cart performance

  @regression @skip:webkit
  Scenario: The login and cart pages load within their budgets
    Given I am on the login page
    When I login with "standard_user" and "{{standardPassword}}"
    And I add "{{defaultProduct}}" to the cart
    And I navigate to the "cart" page
    Then the cart should list "{{defaultProduct}}"

The measurements, budgets and any breaches are attached as sdods/perf-scenario. The scenario fails in these cases:

• A budget is breached. The message names the vital, the page, the value and the limit.
• A budgeted vital was never measured. For example, WebKit does not implement largest contentful paint, so with lcpMs configured the demo scenario carries @skip:webkit. A missing value is never treated as 0.
• The environment has no budgets at all. This fails with CONFIG_INVALID.
• No configured budget applies to what the scenario measured. For example, a UI scenario with only apiP95Ms configured and no API calls proves nothing.
• Every API call was replayed from a HAR. A replayed call reports the recording's timing, not this environment's.

The tag does not judge a scenario that already failed. A document the scenario left before it could report (a navigation away before it painted) is listed as unreported in the attachment, not failed.

Performance stepsCopy Anchor Link

For a measurement at a specific point in the flow, or a limit that belongs to one scenario, use the steps:

In processesCopy Anchor Link

A process can require both results:

processes:
  - name: release-gate
    tags: '@smoke or @regression'
    gates: { minPassRate: 100, maxFlaky: 0, perfBudgets: true, a11y: true }

sdods run --process release-gate evaluates the gates after the run and prints one row per gate. If any gate fails, the command exits with code 1 and error GATE_FAILED. It does this even when every scenario passed.

• a11y: true fails on any blocking violation in an @a11y audit. It also fails when no @a11y audit ran.
• perfBudgets: true fails on any breach in an @perf scenario. It also fails when no @perf scenario was judged.

Selecting nothing the gate can check is not a pass. The verdict is written to gates.json in the run directory, and the server records it on the run.

Next stepsCopy Anchor Link

• Processes and testing types
• Insights and flaky management