Skip to main content
OrchestKit v7.5.2 — 89 skills, 31 agents, 99 hooks · Claude Code 2.1.74+
OrchestKit
Skills

Testing E2e

End-to-end testing patterns with Playwright — page objects, AI agent testing, visual regression, accessibility testing with axe-core, and CI integration. Use when writing E2E tests, setting up Playwright, implementing visual regression, or testing accessibility.

Reference medium

Primary Agent: test-generator

E2E Testing Patterns

End-to-end testing with Playwright 1.58+, visual regression, accessibility, and AI agent workflows.

Quick Reference

CategoryRulesImpactWhen to Use
Playwright Corerules/e2e-playwright.mdHIGHSemantic locators, auto-wait, flaky detection
Page Objectsrules/e2e-page-objects.mdHIGHEncapsulate page interactions, visual regression
AI Agentsrules/e2e-ai-agents.mdHIGHPlanner/Generator/Healer, init-agents
A11y Playwrightrules/a11y-playwright.mdMEDIUMFull-page axe-core scanning with WCAG 2.2 AA
A11y CI/CDrules/a11y-testing.mdMEDIUMCI gates, jest-axe unit tests, PR blocking
End-to-End Typesrules/validation-end-to-end.mdHIGHtRPC, Prisma, Pydantic type safety

Total: 6 rules, 4 references, 3 checklists, 3 examples, 1 script

Playwright Quick Start

import { test, expect } from '@playwright/test';

test('user can complete checkout', async ({ page }) => {
  await page.goto('/products');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await page.getByRole('link', { name: 'Checkout' }).click();
  await page.getByLabel('Email').fill('test@example.com');
  await page.getByRole('button', { name: 'Submit' }).click();
  await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
});

Locator Priority: getByRole() > getByLabel() > getByPlaceholder() > getByTestId()

Playwright Core

Semantic locator patterns and best practices for resilient tests.

RuleFileKey Pattern
Playwright E2Erules/e2e-playwright.mdSemantic locators, auto-wait, new 1.58+ features

Anti-patterns (FORBIDDEN):

  • Hardcoded waits: await page.waitForTimeout(2000)
  • CSS selectors for interactions: await page.click('.submit-btn')
  • XPath locators

Page Objects

Encapsulate page interactions into reusable classes.

RuleFileKey Pattern
Page Object Modelrules/e2e-page-objects.mdLocators in constructor, action methods, assertion methods
const checkout = new CheckoutPage(page);
await checkout.fillEmail('test@example.com');
await checkout.submit();
await checkout.expectConfirmation();

AI Agents

Playwright 1.58+ AI agent framework for test planning, generation, and self-healing.

RuleFileKey Pattern
AI Agentsrules/e2e-ai-agents.mdPlanner, Generator, Healer workflow
npx playwright init-agents --loop=claude    # For Claude Code

Workflow: Planner (explores app, creates specs) -> Generator (reads spec, tests live app) -> Healer (fixes failures, updates selectors).

Accessibility (Playwright)

Full-page accessibility validation with axe-core in E2E tests.

RuleFileKey Pattern
Playwright + axerules/a11y-playwright.mdWCAG 2.2 AA, interactive state testing
import AxeBuilder from '@axe-core/playwright';

test('page meets WCAG 2.2 AA', async ({ page }) => {
  await page.goto('/');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
    .analyze();
  expect(results.violations).toEqual([]);
});

Accessibility (CI/CD)

CI pipeline integration and jest-axe unit-level component testing.

RuleFileKey Pattern
CI Gates + jest-axerules/a11y-testing.mdPR blocking, component state testing

End-to-End Types

Type safety across API layers to eliminate runtime type errors.

RuleFileKey Pattern
Type Safetyrules/validation-end-to-end.mdtRPC, Zod, Pydantic, schema rejection tests

Visual Regression

Native Playwright screenshot comparison without external services.

await expect(page).toHaveScreenshot('checkout-page.png', {
  maxDiffPixels: 100,
  mask: [page.locator('.dynamic-content')],
});

See references/visual-regression.md for full configuration, CI/CD workflows, cross-platform handling, and Percy migration guide.

Key Decisions

DecisionRecommendation
E2E frameworkPlaywright 1.58+ with semantic locators
Locator strategygetByRole > getByLabel > getByTestId
BrowserChromium (Chrome for Testing in 1.58+)
Page patternPage Object Model for complex pages
Visual regressionPlaywright native toHaveScreenshot()
A11y testingaxe-core (E2E) + jest-axe (unit)
CI retries2-3 in CI, 0 locally
Flaky detectionfailOnFlakyTests: true in CI
AI agentsPlanner/Generator/Healer via init-agents
Type safetytRPC for end-to-end, Zod for runtime validation

References

ResourceDescription
references/playwright-1.57-api.mdPlaywright 1.58+ API: locators, assertions, AI agents, auth, flaky detection
references/playwright-setup.mdInstallation, MCP server, seed tests, agent initialization
references/visual-regression.mdScreenshot config, CI/CD workflows, cross-platform, Percy migration
references/a11y-testing-tools.mdjest-axe setup, Playwright axe-core, CI pipelines, manual checklists

Checklists

ChecklistDescription
checklists/e2e-checklist.mdLocator strategy, page objects, CI/CD, visual regression
checklists/e2e-testing-checklist.mdComprehensive: planning, implementation, SSE, responsive, maintenance
checklists/a11y-testing-checklist.mdAutomated + manual: keyboard, screen reader, color contrast, WCAG

Examples

ExampleDescription
examples/e2e-test-patterns.mdUser flows, page objects, auth fixtures, API mocking, multi-tab, file upload
examples/a11y-testing-examples.mdjest-axe components, Playwright axe E2E, custom rules, CI pipeline
examples/orchestkit-e2e-tests.mdOrchestKit analysis flow: page objects, SSE progress, error handling

Scripts

ScriptDescription
scripts/create-page-object.mdGenerate Playwright page object with auto-detected patterns
  • testing-unit - Unit testing patterns with mocking, fixtures, and data factories
  • test-standards-enforcer - AAA and naming enforcement
  • run-tests - Test execution orchestration

Rules (6)

Validate full-page accessibility compliance through Playwright E2E tests with axe-core — MEDIUM

Playwright + axe-core E2E

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('page has no a11y violations', async ({ page }) => {
  await page.goto('/');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
    .analyze();
  expect(results.violations).toEqual([]);
});

test('modal state has no violations', async ({ page }) => {
  await page.goto('/');
  await page.click('[data-testid="open-modal"]');
  await page.waitForSelector('[role="dialog"]');

  const results = await new AxeBuilder({ page })
    .include('[role="dialog"]')
    .withTags(['wcag2a', 'wcag2aa'])
    .analyze();
  expect(results.violations).toEqual([]);
});

Key Decisions

DecisionChoiceRationale
Test runnerPlaywright + axeFull page coverage
WCAG levelAA (wcag2aa)Industry standard
State testingTest all interactive statesModal, error, loading
Browser matrixChromium + FirefoxCross-browser coverage

Incorrect — Testing page without WCAG tags:

test('page has no violations', async ({ page }) => {
  await page.goto('/');
  const results = await new AxeBuilder({ page }).analyze();
  expect(results.violations).toEqual([]);
});

Correct — Testing with WCAG 2.2 AA compliance:

test('page meets WCAG 2.2 AA', async ({ page }) => {
  await page.goto('/');
  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
    .analyze();
  expect(results.violations).toEqual([]);
});

Enforce accessibility testing in CI pipelines and enable unit-level component testing with jest-axe — MEDIUM

CI/CD Accessibility Gates

# .github/workflows/accessibility.yml
name: Accessibility
on: [pull_request]

jobs:
  a11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci
      - run: npm run test:a11y
      - run: npm run build
      - run: npx playwright install --with-deps chromium
      - run: npm start & npx wait-on http://localhost:3000
      - run: npx playwright test e2e/accessibility

Anti-Patterns (FORBIDDEN)

// BAD: Excluding too much
new AxeBuilder({ page })
  .exclude('body')  // Defeats the purpose
  .analyze();

// BAD: No CI enforcement
// Accessibility tests exist but don't block PRs

// BAD: Manual-only testing
// Relying solely on human review

Key Decisions

DecisionChoiceRationale
CI gateBlock on violationsPrevent regression
Tagswcag2a, wcag2aa, wcag22aaFull WCAG 2.2 AA
ExclusionsThird-party widgets onlyMinimize blind spots

Incorrect — Accessibility tests exist but don't enforce in CI:

# .github/workflows/test.yml
- run: npm run test:a11y  # Runs but doesn't block on failures
- run: npm run test:unit

Correct — CI blocks PRs on accessibility violations:

# .github/workflows/accessibility.yml
on: [pull_request]
jobs:
  a11y:
    runs-on: ubuntu-latest
    steps:
      - run: npm run test:a11y  # Exits with code 1 on violations
      - run: npx playwright test e2e/accessibility  # Blocks merge

jest-axe Unit Testing

Setup

// jest.setup.ts
import { toHaveNoViolations } from 'jest-axe';
expect.extend(toHaveNoViolations);

Component Testing

import { render } from '@testing-library/react';
import { axe } from 'jest-axe';

it('has no a11y violations', async () => {
  const { container } = render(<Button>Click me</Button>);
  expect(await axe(container)).toHaveNoViolations();
});

Anti-Patterns (FORBIDDEN)

// BAD: Disabling rules globally
const results = await axe(container, {
  rules: { 'color-contrast': { enabled: false } }  // NEVER disable rules
});

// BAD: Only testing happy path
it('form is accessible', async () => {
  const { container } = render(<Form />);
  expect(await axe(container)).toHaveNoViolations();
  // Missing: error state, loading state, disabled state
});

Key Patterns

  • Test all component states (default, error, loading, disabled)
  • Never disable axe rules globally
  • Use for fast feedback in development

Incorrect — Only testing the default state:

it('form is accessible', async () => {
  const { container } = render(<LoginForm />);
  expect(await axe(container)).toHaveNoViolations();
  // Missing: error, loading, disabled states
});

Correct — Testing all component states:

it('form is accessible in all states', async () => {
  const { container, rerender } = render(<LoginForm />);
  expect(await axe(container)).toHaveNoViolations();

  rerender(<LoginForm error="Invalid email" />);
  expect(await axe(container)).toHaveNoViolations();

  rerender(<LoginForm loading={true} />);
  expect(await axe(container)).toHaveNoViolations();
});

Use Playwright AI agent framework for test planning, generation, and self-healing — HIGH

Playwright AI Agents (1.58+)

Initialize AI Agents

npx playwright init-agents --loop=claude    # For Claude Code
npx playwright init-agents --loop=vscode    # For VS Code (v1.105+)
npx playwright init-agents --loop=opencode  # For OpenCode

Generated Structure

Directory/FilePurpose
.github/Agent definitions and configuration
specs/Test plans in Markdown format
tests/seed.spec.tsSeed file for AI agents to reference

Agent Workflow

1. PLANNER   --> Explores app --> Creates specs/checkout.md
                 (uses seed.spec.ts)
2. GENERATOR --> Reads spec --> Tests live app --> Outputs tests/checkout.spec.ts
                 (verifies selectors actually work)
3. HEALER    --> Runs tests --> Fixes failures --> Updates selectors/waits
                 (self-healing)

Key Concepts

  • seed.spec.ts is required — Planner executes this to learn environment, auth, UI elements
  • Generator validates live — Actually tests app to verify selectors work
  • Healer auto-fixes — When UI changes break tests, replays and patches

Setup Requirements

// .mcp.json in project root
{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest"]
    }
  }
}

Incorrect — No seed file for AI agents to learn from:

// Missing tests/seed.spec.ts
// AI agents have no example to understand app structure
npx playwright init-agents --loop=claude

Correct — Seed file teaches agents app patterns:

// tests/seed.spec.ts
import { test } from '@playwright/test';

test('example checkout flow', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await page.getByRole('link', { name: 'Checkout' }).click();
  // Agents learn selectors and patterns from this
});

Encapsulate page interactions into reusable page object classes for maintainable E2E tests — HIGH

Page Object Model

Extract page interactions into reusable classes for maintainable E2E tests.

Pattern

// pages/CheckoutPage.ts
import { Page, Locator } from '@playwright/test';

export class CheckoutPage {
  readonly page: Page;
  readonly emailInput: Locator;
  readonly submitButton: Locator;
  readonly confirmationHeading: Locator;

  constructor(page: Page) {
    this.page = page;
    this.emailInput = page.getByLabel('Email');
    this.submitButton = page.getByRole('button', { name: 'Submit' });
    this.confirmationHeading = page.getByRole('heading', { name: 'Order confirmed' });
  }

  async fillEmail(email: string) {
    await this.emailInput.fill(email);
  }

  async submit() {
    await this.submitButton.click();
  }

  async expectConfirmation() {
    await expect(this.confirmationHeading).toBeVisible();
  }
}

Visual Regression

// Capture and compare visual snapshots
await expect(page).toHaveScreenshot('checkout-page.png', {
  maxDiffPixels: 100,
  mask: [page.locator('.dynamic-content')],
});

Critical User Journeys to Test

  1. Authentication: Signup, login, password reset
  2. Core Transaction: Purchase, booking, submission
  3. Data Operations: Create, update, delete
  4. User Settings: Profile update, preferences

Incorrect — Duplicating selectors across tests:

test('checkout flow', async ({ page }) => {
  await page.getByLabel('Email').fill('test@example.com');
  await page.getByRole('button', { name: 'Submit' }).click();
});

test('another checkout test', async ({ page }) => {
  await page.getByLabel('Email').fill('user@example.com');  // Duplicated
  await page.getByRole('button', { name: 'Submit' }).click();  // Duplicated
});

Correct — Page Object encapsulates selectors:

const checkout = new CheckoutPage(page);
await checkout.fillEmail('test@example.com');
await checkout.submit();
await checkout.expectConfirmation();

Apply semantic locator patterns and best practices for resilient Playwright E2E tests — HIGH

Playwright E2E Testing (1.58+)

Semantic Locators

// PREFERRED: Role-based locators (most resilient)
await page.getByRole('button', { name: 'Add to cart' }).click();
await page.getByRole('link', { name: 'Checkout' }).click();

// GOOD: Label-based for form controls
await page.getByLabel('Email').fill('test@example.com');

// ACCEPTABLE: Test IDs for stable anchors
await page.getByTestId('checkout-button').click();

// AVOID: CSS selectors and XPath (fragile)

Locator Priority: getByRole() > getByLabel() > getByPlaceholder() > getByTestId()

Basic Test

import { test, expect } from '@playwright/test';

test('user can complete checkout', async ({ page }) => {
  await page.goto('/products');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await page.getByRole('link', { name: 'Checkout' }).click();
  await page.getByLabel('Email').fill('test@example.com');
  await page.getByRole('button', { name: 'Submit' }).click();
  await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
});

New Features (1.58+)

// Flaky test detection
export default defineConfig({ failOnFlakyTests: true });

// Assert individual class names
await expect(page.locator('.card')).toContainClass('highlighted');

// IndexedDB storage state
await page.context().storageState({ path: 'auth.json', indexedDB: true });

Anti-Patterns (FORBIDDEN)

// NEVER use hardcoded waits
await page.waitForTimeout(2000);

// NEVER use CSS selectors for user interactions
await page.click('.submit-btn');

// ALWAYS use semantic locators + auto-wait
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('alert')).toBeVisible();

Key Decisions

DecisionRecommendation
LocatorsgetByRole > getByLabel > getByTestId
BrowserChromium (Chrome for Testing in 1.58+)
Execution5-30s per test
Retries2-3 in CI, 0 locally

Incorrect — Using hardcoded waits and CSS selectors:

await page.click('.submit-button');
await page.waitForTimeout(2000);
await expect(page.locator('.success-message')).toBeVisible();

Correct — Semantic locators with auto-wait:

await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('alert', { name: /success/i })).toBeVisible();

Validate end-to-end type safety across API layers to eliminate runtime type errors — HIGH

End-to-End Type Safety Validation

Incorrect -- type gaps between API layers:

// Manual type definitions that can drift from schema
interface User {
  id: string
  name: string
  // Missing 'email' field that database has
}

// No type connection between client and server
const response = await fetch('/api/users')
const users = await response.json() // type: any

Correct -- tRPC end-to-end type safety:

import { initTRPC } from '@trpc/server'
import { z } from 'zod'

const t = initTRPC.create()

export const appRouter = t.router({
  getUser: t.procedure
    .input(z.object({ id: z.string() }))
    .query(async ({ input }) => {
      return await db.user.findUnique({ where: { id: input.id } })
    }),

  createUser: t.procedure
    .input(z.object({ email: z.string().email(), name: z.string() }))
    .mutation(async ({ input }) => {
      return await db.user.create({ data: input })
    })
})

export type AppRouter = typeof appRouter
// Client gets full type inference from server without code generation

Correct -- Python type safety with Pydantic and NewType:

from typing import NewType
from uuid import UUID
from pydantic import BaseModel, EmailStr

AnalysisID = NewType("AnalysisID", UUID)
ArtifactID = NewType("ArtifactID", UUID)

def delete_analysis(id: AnalysisID) -> None: ...
delete_analysis(artifact_id)  # Error with mypy/ty

class CreateUserRequest(BaseModel):
    email: EmailStr
    name: str = Field(min_length=2, max_length=100)

# Type-safe extraction from untyped dict
result = {"findings": {...}, "confidence_score": 0.85}
findings: dict[str, object] | None = (
    cast("dict[str, object]", result.get("findings"))
    if isinstance(result.get("findings"), dict) else None
)

Testing type safety:

// Test that schema rejects invalid data
describe('UserSchema', () => {
  test('rejects invalid email', () => {
    const result = UserSchema.safeParse({ email: 'not-email', name: 'Test' })
    expect(result.success).toBe(false)
  })

  test('rejects missing required fields', () => {
    const result = UserSchema.safeParse({})
    expect(result.success).toBe(false)
    expect(result.error.issues).toHaveLength(2)
  })
})

Key decisions:

  • Runtime validation: Zod (best DX, TypeScript inference)
  • API layer: tRPC for end-to-end type safety without codegen
  • Exhaustive checks: assertNever for compile-time union completeness
  • Python: Pydantic v2 + NewType for branded IDs
  • Always test validation schemas reject invalid data

References (4)

A11y Testing Tools

Accessibility Testing Tools Reference

Comprehensive guide to automated and manual accessibility testing tools.

jest-axe Configuration

Installation

npm install --save-dev jest-axe @testing-library/react @testing-library/jest-dom

Setup

// test-utils/axe.ts
import { configureAxe } from 'jest-axe';

export const axe = configureAxe({
  rules: {
    // Disable rules if needed (use sparingly)
    'color-contrast': { enabled: false }, // Only if manual testing covers this
  },
  reporter: 'v2',
});
// vitest.setup.ts or jest.setup.ts
import { toHaveNoViolations } from 'jest-axe';
expect.extend(toHaveNoViolations);

Basic Usage

import { render } from '@testing-library/react';
import { axe } from './test-utils/axe';

test('Button has no accessibility violations', async () => {
  const { container } = render(<Button>Click me</Button>);
  const results = await axe(container);
  expect(results).toHaveNoViolations();
});

Component-Specific Rules

// Test form with specific WCAG level
test('Form meets WCAG 2.1 Level AA', async () => {
  const { container } = render(<ContactForm />);
  const results = await axe(container, {
    runOnly: {
      type: 'tag',
      values: ['wcag2a', 'wcag2aa', 'wcag21aa'],
    },
  });
  expect(results).toHaveNoViolations();
});

Testing Specific Rules

// Test only keyboard navigation
test('Modal is keyboard accessible', async () => {
  const { container } = render(<Modal isOpen />);
  const results = await axe(container, {
    runOnly: ['keyboard', 'focus-order-semantics'],
  });
  expect(results).toHaveNoViolations();
});

Playwright + axe-core

Installation

npm install --save-dev @axe-core/playwright

Setup

// tests/a11y.setup.ts
import { test as base } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

export const test = base.extend<{ makeAxeBuilder: () => AxeBuilder }>({
  makeAxeBuilder: async ({ page }, use) => {
    const makeAxeBuilder = () =>
      new AxeBuilder({ page })
        .withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])
        .exclude('#third-party-widget');
    await use(makeAxeBuilder);
  },
});

export { expect } from '@playwright/test';

E2E Accessibility Test

import { test, expect } from './a11y.setup';

test('homepage is accessible', async ({ page, makeAxeBuilder }) => {
  await page.goto('/');

  const accessibilityScanResults = await makeAxeBuilder().analyze();

  expect(accessibilityScanResults.violations).toEqual([]);
});

Testing After Interactions

test('modal maintains accessibility after opening', async ({ page, makeAxeBuilder }) => {
  await page.goto('/dashboard');

  // Initial state
  const initialScan = await makeAxeBuilder().analyze();
  expect(initialScan.violations).toEqual([]);

  // After opening modal
  await page.getByRole('button', { name: 'Open Settings' }).click();
  const modalScan = await makeAxeBuilder().analyze();
  expect(modalScan.violations).toEqual([]);

  // Focus should be trapped in modal
  await page.keyboard.press('Tab');
  const focusedElement = await page.evaluate(() => document.activeElement?.tagName);
  expect(focusedElement).not.toBe('BODY');
});

Excluding Regions

test('scan page excluding third-party widgets', async ({ page, makeAxeBuilder }) => {
  await page.goto('/');

  const results = await makeAxeBuilder()
    .exclude('#ads-container')
    .exclude('[data-third-party]')
    .analyze();

  expect(results.violations).toEqual([]);
});

CI/CD Integration

GitHub Actions

# .github/workflows/a11y.yml
name: Accessibility Tests

on: [push, pull_request]

jobs:
  a11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run unit accessibility tests
        run: npm run test:a11y

      - name: Install Playwright
        run: npx playwright install --with-deps chromium

      - name: Build application
        run: npm run build

      - name: Start server
        run: npm run start &
        env:
          PORT: 3000

      - name: Wait for server
        run: npx wait-on http://localhost:3000

      - name: Run E2E accessibility tests
        run: npx playwright test tests/a11y/

      - name: Upload accessibility report
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: a11y-report
          path: playwright-report/
          retention-days: 30

Pre-commit Hook

#!/bin/sh
# .husky/pre-commit

# Run accessibility tests on staged components
STAGED_FILES=$(git diff --cached --name-only --diff-filter=ACM | grep "\.tsx\?$")

if [ -n "$STAGED_FILES" ]; then
  echo "Running accessibility tests on changed components..."
  npm run test:a11y -- --findRelatedTests $STAGED_FILES
  if [ $? -ne 0 ]; then
    echo "❌ Accessibility tests failed. Please fix violations before committing."
    exit 1
  fi
fi

Package.json Scripts

{
  "scripts": {
    "test:a11y": "vitest run tests/**/*.a11y.test.{ts,tsx}",
    "test:a11y:watch": "vitest watch tests/**/*.a11y.test.{ts,tsx}",
    "test:a11y:e2e": "playwright test tests/a11y/",
    "test:a11y:all": "npm run test:a11y && npm run test:a11y:e2e"
  }
}

Manual Testing Checklist

Use this alongside automated tests for comprehensive coverage.

Keyboard Navigation

  1. Tab Order

    • Navigate entire page using only Tab/Shift+Tab
    • Verify logical focus order
    • Ensure all interactive elements are reachable
    • Check focus is visible (outline or custom indicator)
  2. Interactive Elements

    • Enter/Space activates buttons and links
    • Arrow keys navigate within widgets (tabs, menus, sliders)
    • Escape closes modals and dropdowns
    • Home/End navigate to start/end of lists
  3. Form Controls

    • All form fields reachable via keyboard
    • Labels associated with inputs
    • Error messages announced and keyboard-accessible
    • Submit works via Enter key

Screen Reader Testing

Tools:

  • macOS: VoiceOver (Cmd+F5)
  • Windows: NVDA (free) or JAWS
  • Linux: Orca

Test Scenarios:

  1. Navigate by headings (H key in screen reader)
  2. Navigate by landmarks (D key in screen reader)
  3. Form fields announce label and type
  4. Buttons announce role and state (expanded/collapsed)
  5. Dynamic content changes are announced (aria-live)
  6. Images have meaningful alt text or aria-label

Color Contrast

Tools:

  • Browser Extensions: axe DevTools, WAVE
  • Design Tools: Figma has built-in contrast checker
  • Command Line: pa11y or axe-cli

Requirements:

  • Normal text: 4.5:1 contrast ratio (WCAG AA)
  • Large text (18pt+): 3:1 contrast ratio
  • UI components: 3:1 contrast ratio

Responsive and Zoom Testing

  1. Browser Zoom

    • Test at 200% zoom (WCAG 2.1 requirement)
    • Verify no horizontal scrolling
    • Content remains readable
    • No overlapping elements
  2. Mobile Testing

    • Touch targets at least 44×44px
    • No reliance on hover states
    • Swipe gestures have keyboard alternative
    • Pinch-to-zoom enabled

Continuous Monitoring

Lighthouse CI

# lighthouserc.js
module.exports = {
  ci: {
    collect: {
      url: ['http://localhost:3000', 'http://localhost:3000/dashboard'],
      numberOfRuns: 3,
    },
    assert: {
      preset: 'lighthouse:recommended',
      assertions: {
        'categories:accessibility': ['error', { minScore: 0.95 }],
        'categories:best-practices': ['warn', { minScore: 0.9 }],
      },
    },
    upload: {
      target: 'temporary-public-storage',
    },
  },
};

axe-cli for Quick Scans

# Install
npm install -g @axe-core/cli

# Scan a URL
axe http://localhost:3000 --tags wcag2a,wcag2aa

# Save results
axe http://localhost:3000 --save results.json

# Check multiple pages
axe http://localhost:3000 \
    http://localhost:3000/dashboard \
    http://localhost:3000/profile \
    --tags wcag21aa

Common Pitfalls

  1. Automated Testing Limitations

    • Only catches ~30-40% of issues
    • Cannot verify semantic meaning
    • Cannot test keyboard navigation fully
    • Manual testing is REQUIRED
  2. False Sense of Security

    • Passing axe tests ≠ fully accessible
    • Must combine automated + manual testing
    • Screen reader testing is essential
  3. Ignoring Dynamic Content

    • Test ARIA live regions with actual updates
    • Verify focus management after route changes
    • Test loading and error states
  4. Third-Party Components

    • UI libraries may have a11y issues
    • Always test integrated components
    • Don't assume "accessible by default"

Resources

Playwright 1.57 Api

Playwright 1.58+ API Reference

Semantic Locators (2026 Best Practice)

Locator Priority

  1. getByRole() - Matches how users/assistive tech see the page
  2. getByLabel() - For form inputs with labels
  3. getByPlaceholder() - For inputs with placeholders
  4. getByText() - For text content
  5. getByTestId() - When semantic locators aren't possible

Role-Based Locators

// Buttons
await page.getByRole('button', { name: 'Submit' }).click();
await page.getByRole('button', { name: /submit/i }).click(); // Regex

// Links
await page.getByRole('link', { name: 'Home' }).click();

// Headings
await expect(page.getByRole('heading', { name: 'Welcome' })).toBeVisible();
await expect(page.getByRole('heading', { level: 1 })).toHaveText('Welcome');

// Form controls
await page.getByRole('textbox', { name: 'Email' }).fill('test@example.com');
await page.getByRole('checkbox', { name: 'Remember me' }).check();
await page.getByRole('combobox', { name: 'Country' }).selectOption('US');

// Lists
await expect(page.getByRole('list')).toContainText('Item 1');
await expect(page.getByRole('listitem')).toHaveCount(3);

// Navigation
await page.getByRole('navigation').getByRole('link', { name: 'About' }).click();

Label-Based Locators

// Form inputs with labels
await page.getByLabel('Email').fill('test@example.com');
await page.getByLabel('Password').fill('secret123');
await page.getByLabel('Remember me').check();

// Partial match
await page.getByLabel(/email/i).fill('test@example.com');

Text and Placeholder

// Text content
await page.getByText('Welcome back').click();
await page.getByText(/welcome/i).isVisible();

// Placeholder
await page.getByPlaceholder('Enter email').fill('test@example.com');

Test IDs (Fallback)

// When semantic locators aren't possible
await page.getByTestId('custom-widget').click();

// Configure test ID attribute
// playwright.config.ts
export default defineConfig({
  use: {
    testIdAttribute: 'data-test-id',
  },
});

Breaking Changes (1.58)

Removed Features

FeatureStatusMigration
_react selectorRemovedUse getByRole() or getByTestId()
_vue selectorRemovedUse getByRole() or getByTestId()
:light selector suffixRemovedUse standard CSS selectors
devtools launch optionRemovedUse args: ['--auto-open-devtools-for-tabs']
macOS 13 WebKitRemovedUpgrade to macOS 14+

Migration Examples

// React/Vue component selectors - Before
await page.locator('_react=MyComponent').click();
await page.locator('_vue=MyComponent').click();

// After - Use semantic locators or test IDs
await page.getByRole('button', { name: 'My Component' }).click();
await page.getByTestId('my-component').click();

// :light selector - Before
await page.locator('.card:light').click();

// After - Just use the selector directly
await page.locator('.card').click();

// DevTools option - Before
const browser = await chromium.launch({ devtools: true });

// After - Use args
const browser = await chromium.launch({
  args: ['--auto-open-devtools-for-tabs']
});

New Features (1.58+)

connectOverCDP with isLocal

// Optimized CDP connection for local debugging
const browser = await chromium.connectOverCDP({
  endpointURL: 'http://localhost:9222',
  isLocal: true  // NEW: Optimizes for local connections
});

// Use for connecting to locally running Chrome instances
// Reduces latency and improves reliability

Timeline in Speedboard HTML Reports

HTML reports now include an interactive timeline:

// playwright.config.ts
export default defineConfig({
  reporter: [['html', { open: 'never' }]],
});

// The HTML report shows:
// - Test execution sequence
// - Parallel test distribution
// - Time spent in each test phase
// - Performance bottlenecks

New Assertions (1.57+)

// Assert individual class names (1.57+)
await expect(page.locator('.card')).toContainClass('highlighted');
await expect(page.locator('.card')).toContainClass(['active', 'visible']);

// Visibility
await expect(page.getByRole('button')).toBeVisible();
await expect(page.getByRole('button')).toBeHidden();
await expect(page.getByRole('button')).toBeEnabled();
await expect(page.getByRole('button')).toBeDisabled();

// Text content
await expect(page.getByRole('heading')).toHaveText('Welcome');
await expect(page.getByRole('heading')).toContainText('Welcome');

// Attribute
await expect(page.getByRole('link')).toHaveAttribute('href', '/home');

// Count
await expect(page.getByRole('listitem')).toHaveCount(5);

// Screenshot
await expect(page).toHaveScreenshot('page.png');
await expect(page.locator('.hero')).toHaveScreenshot('hero.png');

AI Agents (1.58+)

Initialize AI Agents

# Initialize agents for your preferred AI tool
npx playwright init-agents --loop=claude    # For Claude Code
npx playwright init-agents --loop=vscode    # For VS Code (requires v1.105+)
npx playwright init-agents --loop=opencode  # For OpenCode

Generated Structure

Directory/FilePurpose
.github/Agent definitions and configuration
specs/Test plans in Markdown format
tests/seed.spec.tsSeed file for AI agents to reference

Configuration

// playwright.config.ts
export default defineConfig({
  use: {
    aiAgents: {
      enabled: true,
      model: 'claude-sonnet-4-6',  // or local Ollama
      autoHeal: true,              // Auto-repair on CI failures
    }
  }
});

Authentication State

Storage State

// Save auth state
await page.context().storageState({ path: 'playwright/.auth/user.json' });

// Use saved state
const context = await browser.newContext({
  storageState: 'playwright/.auth/user.json'
});

IndexedDB Support (1.57+)

// Save storage state including IndexedDB
await page.context().storageState({
  path: 'auth.json',
  indexedDB: true  // Include IndexedDB in storage state
});

// Restore with IndexedDB
const context = await browser.newContext({
  storageState: 'auth.json'  // Includes IndexedDB automatically
});

Auth Setup Project

// playwright.config.ts
export default defineConfig({
  projects: [
    {
      name: 'setup',
      testMatch: /.*\.setup\.ts/,
    },
    {
      name: 'logged-in',
      dependencies: ['setup'],
      use: {
        storageState: 'playwright/.auth/user.json',
      },
    },
  ],
});

Flaky Test Detection (1.57+)

// playwright.config.ts
export default defineConfig({
  // Fail CI if any flaky tests detected
  failOnFlakyTests: true,

  // Retry configuration
  retries: process.env.CI ? 2 : 0,

  // Web server with regex-based ready detection
  webServer: {
    command: 'npm run dev',
    wait: /ready in \d+ms/,  // Wait for this log pattern
  },
});

Visual Regression

test('visual regression', async ({ page }) => {
  await page.goto('/');

  // Full page screenshot
  await expect(page).toHaveScreenshot('homepage.png');

  // Element screenshot
  await expect(page.locator('.hero')).toHaveScreenshot('hero.png');

  // With options
  await expect(page).toHaveScreenshot('page.png', {
    maxDiffPixels: 100,
    threshold: 0.2,
  });
});

Locator Descriptions (1.57+)

// Describe locators for trace viewer
const submitBtn = page.getByRole('button', { name: 'Submit' });
submitBtn.describe('Main form submit button');

// Shows in trace viewer for debugging

Chrome for Testing (1.57+)

Playwright uses Chrome for Testing builds instead of Chromium:

# Install browsers (includes Chrome for Testing)
npx playwright install

# No code changes needed - better Chrome compatibility

Playwright Setup

Playwright Setup with Test Agents

Install and configure Playwright with autonomous test agents for Claude Code.

Prerequisites

Required: VS Code v1.105+ (released Oct 9, 2025) for agent functionality

Step 1: Install Playwright

npm install --save-dev @playwright/test
npx playwright install  # Install browsers (Chromium, Firefox, WebKit)

Step 2: Add Playwright MCP Server (CC 2.1.6)

Create or update .mcp.json in your project root:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest"]
    }
  }
}

Restart your Claude Code session to pick up the MCP configuration.

Note: The claude mcp add command is deprecated in CC 2.1.6. Configure MCPs directly via .mcp.json.

Step 3: Initialize Test Agents

# Initialize the three agents (planner, generator, healer)
npx playwright init-agents --loop=claude
# OR for VS Code: --loop=vscode
# OR for OpenCode: --loop=opencode

What this does:

  • Creates agent definition files in your project
  • Agents are Markdown-based instruction files
  • Regenerate when Playwright updates to get latest tools

Step 4: Create Seed Test

Create tests/seed.spec.ts - the planner uses this to understand your setup:

// tests/seed.spec.ts
import { test, expect } from '@playwright/test';

test.beforeEach(async ({ page }) => {
  // Your app initialization
  await page.goto('http://localhost:3000');

  // Login if needed
  // await page.getByLabel('Email').fill('test@example.com');
  // await page.getByLabel('Password').fill('password123');
  // await page.getByRole('button', { name: 'Login' }).click();
});

test('seed test - app is accessible', async ({ page }) => {
  await expect(page).toHaveTitle(/MyApp/);
  await expect(page.getByRole('navigation')).toBeVisible();
});

Why seed.spec.ts?

  • Planner executes this to learn:
    • Environment setup (fixtures, hooks)
    • Authentication flow
    • App initialization
    • Available selectors

Directory Structure

your-project/
├── specs/              <- Planner outputs test plans here (Markdown)
├── tests/              <- Generator outputs test code here (.spec.ts)
│   └── seed.spec.ts    <- Your initialization test (REQUIRED)
├── playwright.config.ts
└── .mcp.json           <- MCP server config

Basic Configuration

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,

  use: {
    baseURL: 'http://localhost:3000',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },

  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

Running Tests

npx playwright test                 # Run all tests
npx playwright test --ui            # UI mode
npx playwright test --debug         # Debug mode
npx playwright test --headed        # See browser

Browser Automation

For quick browser automation outside of Playwright tests, use agent-browser CLI:

# Quick visual verification
agent-browser open http://localhost:5173
agent-browser snapshot -i
agent-browser screenshot /tmp/screenshot.png
agent-browser close

Run agent-browser --help for full CLI docs.

Next Steps

  1. Planner: "Generate test plan for checkout flow" -> creates specs/checkout.md
  2. Generator: "Generate tests from checkout spec" -> creates tests/checkout.spec.ts
  3. Healer: Automatically fixes tests when selectors break

See references/planner-agent.md for detailed workflow.

Visual Regression

Playwright Native Visual Regression Testing

Updated Dec 2025 - Best practices for toHaveScreenshot() without external services like Percy or Chromatic.

Overview

Playwright's built-in visual regression testing uses expect(page).toHaveScreenshot() to capture and compare screenshots. This is completely free, requires no signup, and works in CI without external dependencies.

Quick Start

import { test, expect } from '@playwright/test';

test('homepage visual regression', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot('homepage.png');
});

On first run, Playwright creates a baseline screenshot. Subsequent runs compare against it.


Configuration (playwright.config.ts)

Essential Settings

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './e2e',

  // Snapshot configuration
  snapshotPathTemplate: '{testDir}/__screenshots__/{testFilePath}/{arg}{ext}',
  updateSnapshots: 'missing', // 'all' | 'changed' | 'missing' | 'none'

  expect: {
    toHaveScreenshot: {
      // Tolerance settings
      maxDiffPixelRatio: 0.01,  // Allow 1% pixel difference
      threshold: 0.2,           // Per-pixel color threshold (0-1)

      // Animation handling
      animations: 'disabled',   // Freeze CSS animations

      // Caret handling (text cursors)
      caret: 'hide',
    },
  },

  // CI-specific settings
  workers: process.env.CI ? 1 : undefined,
  retries: process.env.CI ? 2 : 0,

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    // Only run screenshots on Chromium for consistency
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
      ignoreSnapshots: true,  // Skip VRT for Firefox
    },
  ],
});

Snapshot Path Template Tokens

TokenDescriptionExample
\{testDir\}Test directorye2e
\{testFilePath\}Test file relative pathspecs/visual.spec.ts
\{testFileName\}Test file namevisual.spec.ts
\{arg\}Screenshot name argumenthomepage
\{ext\}File extension.png
\{projectName\}Project namechromium

Test Patterns

Basic Screenshot

test('page screenshot', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot('page-name.png');
});

Full Page Screenshot

test('full page screenshot', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot('full-page.png', {
    fullPage: true,
  });
});

Element Screenshot

test('component screenshot', async ({ page }) => {
  await page.goto('/');
  const header = page.locator('header');
  await expect(header).toHaveScreenshot('header.png');
});

Masking Dynamic Content

test('page with masked dynamic content', async ({ page }) => {
  await page.goto('/');

  await expect(page).toHaveScreenshot('page.png', {
    mask: [
      page.locator('[data-testid="timestamp"]'),
      page.locator('[data-testid="random-avatar"]'),
      page.locator('time'),
    ],
    maskColor: '#FF00FF',  // Pink mask (default)
  });
});

Custom Styles for Screenshots

// e2e/fixtures/screenshot.css
// Hide dynamic elements during screenshots
[data-testid="timestamp"],
[data-testid="loading-spinner"] {
  visibility: hidden !important;
}

* {
  animation: none !important;
  transition: none !important;
}
test('page with custom styles', async ({ page }) => {
  await page.goto('/');

  await expect(page).toHaveScreenshot('styled.png', {
    stylePath: './e2e/fixtures/screenshot.css',
  });
});

Responsive Viewports

const viewports = [
  { name: 'mobile', width: 375, height: 667 },
  { name: 'tablet', width: 768, height: 1024 },
  { name: 'desktop', width: 1280, height: 800 },
];

for (const viewport of viewports) {
  test(`homepage - ${viewport.name}`, async ({ page }) => {
    await page.setViewportSize({
      width: viewport.width,
      height: viewport.height
    });
    await page.goto('/');
    await expect(page).toHaveScreenshot(`homepage-${viewport.name}.png`);
  });
}

Dark Mode Testing

test('homepage dark mode', async ({ page }) => {
  await page.goto('/');

  // Toggle dark mode
  await page.evaluate(() => {
    document.documentElement.classList.add('dark');
    localStorage.setItem('theme', 'dark');
  });

  // Wait for theme to apply
  await page.waitForTimeout(100);

  await expect(page).toHaveScreenshot('homepage-dark.png');
});

Waiting for Stability

test('page after animations complete', async ({ page }) => {
  await page.goto('/');

  // Wait for network idle
  await page.waitForLoadState('networkidle');

  // Wait for specific content
  await page.waitForSelector('[data-testid="content-loaded"]');

  // Playwright auto-waits for 2 consecutive stable screenshots
  await expect(page).toHaveScreenshot('stable.png');
});

CI/CD Integration

GitHub Actions Workflow

name: Visual Regression Tests

on:
  pull_request:
    branches: [main, dev]

jobs:
  visual-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install chromium --with-deps

      - name: Run visual regression tests
        run: npx playwright test --project=chromium e2e/specs/visual-regression.spec.ts

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

      - name: Upload screenshots on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: screenshot-diffs
          path: e2e/__screenshots__/
          retention-days: 7

Handling Baseline Updates

# Separate workflow for updating baselines
name: Update Visual Baselines

on:
  workflow_dispatch:  # Manual trigger only

jobs:
  update-baselines:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup and install
        run: |
          npm ci
          npx playwright install chromium --with-deps

      - name: Update snapshots
        run: npx playwright test --update-snapshots

      - name: Commit updated snapshots
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add e2e/__screenshots__/
          git commit -m "chore: update visual regression baselines" || exit 0
          git push

Handling Cross-Platform Issues

The Problem

Screenshots differ between macOS (local) and Linux (CI) due to:

  • Font rendering differences
  • Anti-aliasing variations
  • Subpixel rendering

Solutions

Option 1: Generate baselines only in CI (Recommended)

// playwright.config.ts
export default defineConfig({
  // Only update snapshots in CI
  updateSnapshots: process.env.CI ? 'missing' : 'none',
});

Option 2: Use Docker for local development

# Run tests in same container as CI
docker run --rm -v $(pwd):/work -w /work mcr.microsoft.com/playwright:v1.58.0-jammy \
  npx playwright test --project=chromium

Option 3: Increase threshold tolerance

expect: {
  toHaveScreenshot: {
    maxDiffPixelRatio: 0.05,  // 5% tolerance
    threshold: 0.3,           // Higher per-pixel tolerance
  },
},

Debugging Failed Screenshots

View Diff Report

npx playwright show-report

Generated Files on Failure

e2e/__screenshots__/
├── homepage.png              # Expected (baseline)
├── homepage-actual.png       # Actual (current run)
└── homepage-diff.png         # Difference highlighted

Trace Viewer for Context

// playwright.config.ts
export default defineConfig({
  use: {
    trace: 'on-first-retry',  // Capture trace on failures
  },
});

Best Practices

1. Stable Selectors

// Good - semantic selectors
await page.waitForSelector('[data-testid="content"]');

// Avoid - fragile selectors
await page.waitForSelector('.css-1234xyz');

2. Wait for Stability

// Ensure page is ready before screenshot
await page.waitForLoadState('networkidle');
await page.waitForSelector('[data-loaded="true"]');

3. Mask Dynamic Content

// Always mask timestamps, avatars, random content
mask: [
  page.locator('time'),
  page.locator('[data-testid="avatar"]'),
],

4. Disable Animations

// Global in config
animations: 'disabled',

// Or per-test with CSS
stylePath: './e2e/fixtures/no-animations.css',

5. Single Browser for VRT

// Only Chromium for visual tests - most consistent
projects: [
  {
    name: 'chromium',
    use: { ...devices['Desktop Chrome'] },
  },
],

6. Meaningful Names

// Good - descriptive names
await expect(page).toHaveScreenshot('checkout-payment-form-error.png');

// Avoid - generic names
await expect(page).toHaveScreenshot('test1.png');

Migration from Percy

PercyPlaywright Native
percySnapshot(page, 'name')await expect(page).toHaveScreenshot('name.png')
.percy.ymlplaywright.config.ts expect settings
PERCY_TOKENNot needed
Cloud dashboardLocal HTML report
percy exec --Direct npx playwright test

Quick Migration Script

// Before (Percy)
import { percySnapshot } from '@percy/playwright';
await percySnapshot(page, 'Homepage - Light Mode');

// After (Playwright)
// No import needed
await expect(page).toHaveScreenshot('homepage-light.png');

Troubleshooting

Flaky Screenshots

Symptoms: Different results on each run

Solutions:

  1. Increase maxDiffPixelRatio tolerance
  2. Add explicit waits for dynamic content
  3. Mask loading spinners and animations
  4. Use animations: 'disabled'

CI vs Local Differences

Symptoms: Tests pass locally, fail in CI

Solutions:

  1. Generate baselines only in CI
  2. Use Docker locally for consistency
  3. Increase threshold for font rendering

Large Screenshot Files

Symptoms: Git repository bloat

Solutions:

  1. Use .gitattributes for LFS
  2. Compress with quality option (JPEG only)
  3. Limit screenshot dimensions
# .gitattributes
e2e/__screenshots__/**/*.png filter=lfs diff=lfs merge=lfs -text

Checklists (3)

A11y Testing Checklist

Accessibility Testing Checklist

Use this checklist to ensure comprehensive accessibility coverage.

Automated Test Coverage

Unit Tests (jest-axe)

  • All form components tested with axe
  • All interactive components (buttons, links, modals) tested
  • Custom UI widgets tested (date pickers, dropdowns, sliders)
  • Dynamic content updates tested
  • Error states tested for proper announcements
  • Loading states have appropriate ARIA attributes
  • Tests cover WCAG 2.1 Level AA tags minimum
  • No disabled rules without documented justification

E2E Tests (Playwright + axe-core)

  • Homepage scanned for violations
  • All critical user journeys include a11y scan
  • Post-interaction states scanned (after form submit, modal open)
  • Multi-step flows tested (signup, checkout, settings)
  • Error pages and 404s tested
  • Third-party widgets excluded from scan if necessary
  • Tests run in CI/CD pipeline
  • Accessibility reports archived on failure

CI/CD Integration

  • Accessibility tests run on every PR
  • Pre-commit hook runs a11y tests on changed files
  • Lighthouse CI monitors accessibility score (>95%)
  • Failed tests block deployment
  • Test results published to team (GitHub comments, Slack)

Manual Testing Requirements

Keyboard Navigation

  • Tab Navigation

    • All interactive elements reachable via Tab/Shift+Tab
    • Tab order follows visual layout (top to bottom, left to right)
    • Focus indicator visible on all focusable elements
    • No keyboard traps (can always Tab away)
  • Action Keys

    • Enter/Space activates buttons and links
    • Escape closes modals, dropdowns, menus
    • Arrow keys navigate within compound widgets (tabs, menus, sliders)
    • Home/End keys navigate to start/end where appropriate
  • Form Controls

    • All form fields accessible via keyboard
    • Enter submits forms
    • Error messages keyboard-navigable
    • Custom controls (date pickers, color pickers) keyboard-operable
  • Skip Links

    • "Skip to main content" link present and functional
    • Appears on first Tab press
    • Actually skips navigation when activated

Screen Reader Testing

Test with at least one screen reader:

  • macOS: VoiceOver (Cmd+F5)
  • Windows: NVDA (free) or JAWS
  • Linux: Orca

Content Structure

  • Headings

    • Logical heading hierarchy (h1 → h2 → h3, no skips)
    • Page has exactly one h1
    • Headings describe section content
    • Can navigate by heading (H key in screen reader)
  • Landmarks

    • &lt;header&gt;, &lt;nav&gt;, &lt;main&gt;, &lt;footer&gt; present
    • Multiple landmarks of same type have unique labels
    • Can navigate by landmark (D key in screen reader)
  • Lists

    • Navigation uses <ul> or &lt;nav&gt;
    • Related items grouped in lists
    • Screen reader announces list with item count

Interactive Elements

  • Forms

    • All inputs have associated &lt;label&gt; or aria-label
    • Required fields announced as required
    • Error messages announced when they appear
    • Field types announced (email, password, number)
    • Placeholder text not used as only label
  • Buttons and Links

    • Role announced ("button", "link")
    • Purpose clear from label alone
    • State announced (expanded/collapsed, selected)
    • Icon-only buttons have aria-label
  • Images

    • Informative images have meaningful alt text
    • Decorative images have alt="" or role="presentation"
    • Complex images have longer description (aria-describedby or caption)
  • Dynamic Content

    • Live regions announce updates (aria-live="polite" or "assertive")
    • Loading states announced
    • Success/error messages announced
    • Content changes don't lose focus position
  • Menus

    • Menu buttons announce expanded/collapsed state
    • Arrow keys navigate menu items
    • First/last items wrap or stop appropriately
    • Escape closes menu
  • Modals/Dialogs

    • Focus moves to modal on open
    • Focus trapped within modal
    • Modal title announced
    • Escape closes modal
    • Focus returns to trigger on close
  • Tabs

    • Tab role announced
    • Active tab announced as selected
    • Arrow keys navigate tabs
    • Tab panel content announced

Color and Contrast

Use browser extensions (axe DevTools, WAVE) or online tools:

  • Text Contrast

    • Normal text (< 18pt): 4.5:1 minimum ratio
    • Large text (≥ 18pt or 14pt bold): 3:1 minimum ratio
    • Passes for all text (body, headings, labels, placeholders)
  • UI Component Contrast

    • Buttons, inputs, icons: 3:1 minimum against background
    • Focus indicators: 3:1 minimum
    • Error/success states: 3:1 minimum
  • Color Independence

    • Information not conveyed by color alone
    • Links distinguishable without color (underline, icon, etc.)
    • Form errors indicated by icon + text, not just red border
    • Charts/graphs have patterns or labels, not just colors

Responsive and Zoom Testing

  • Browser Zoom (200%)

    • Test at 200% zoom level (WCAG 2.1 requirement)
    • No horizontal scrolling at 200% zoom
    • All content visible and readable
    • No overlapping or cut-off text
    • Interactive elements remain operable
  • Mobile/Touch

    • Touch targets ≥ 44×44 CSS pixels
    • Sufficient spacing between interactive elements (at least 8px)
    • No reliance on hover (all hover info accessible on tap)
    • Pinch-to-zoom enabled (no user-scalable=no)
    • Orientation works in both portrait and landscape

Animation and Motion

  • Respect Motion Preferences

    • Check prefers-reduced-motion media query
    • Disable or reduce animations when preferred
    • Test with system setting enabled (macOS, Windows)
  • No Seizure Triggers

    • No flashing content faster than 3 times per second
    • Autoplay videos have controls (pause/stop)
    • Parallax effects can be disabled

Documentation Review

  • ARIA Usage

    • ARIA only used when native HTML insufficient
    • ARIA roles match HTML semantics
    • All required ARIA properties present
    • No conflicting or redundant ARIA
  • Code Comments

    • Complex accessibility patterns documented
    • Keyboard shortcuts documented
    • Focus management documented

Cross-Browser Testing

Test in multiple browsers and assistive tech combinations:

  • Chrome + NVDA (Windows)
  • Firefox + NVDA (Windows)
  • Safari + VoiceOver (macOS)
  • Safari + VoiceOver (iOS)
  • Chrome + TalkBack (Android)

Compliance Verification

  • WCAG 2.1 Level AA

    • Automated tests pass for wcag2a, wcag2aa, wcag21aa tags
    • Manual testing confirms keyboard accessibility
    • Manual testing confirms screen reader accessibility
    • Color contrast verified
  • Legal Requirements

    • Section 508 (US federal)
    • ADA (US)
    • EN 301 549 (EU)
    • Accessibility statement page present (if required)

Continuous Monitoring

  • Lighthouse accessibility score tracked over time
  • Accessibility tests in regression suite
  • New features include a11y tests from day one
  • Team trained on accessibility best practices
  • Accessibility champion assigned
  • Regular audits scheduled (quarterly recommended)

When to Seek Expert Help

Engage an accessibility specialist if:

  • Building complex custom widgets (ARIA patterns)
  • Handling advanced screen reader interactions
  • Preparing for legal compliance audit
  • User feedback indicates accessibility issues
  • Automated tests show many violations
  • Team lacks accessibility expertise

Quick Wins for Common Issues

Missing Alt Text

<!-- Before -->
<img src="logo.png">

<!-- After -->
<img src="logo.png" alt="Company Logo">

Unlabeled Form Input

<!-- Before -->
<input type="email" placeholder="Email">

<!-- After -->
<label for="email">Email</label>
<input type="email" id="email">

Low Contrast Text

/* Before */
color: #999; /* 2.8:1 ratio */

/* After */
color: #767676; /* 4.5:1 ratio */

Keyboard Trap

// Before
<div onClick={handleClick}>Click me</div>

// After
<button onClick={handleClick}>Click me</button>

Missing Focus Indicator

/* Before */
button:focus { outline: none; }

/* After */
button:focus-visible {
  outline: 2px solid blue;
  outline-offset: 2px;
}

E2e Checklist

E2E Testing Checklist

Test Selection Checklist

Focus E2E tests on business-critical paths:

  • Authentication: Signup, login, password reset, logout
  • Core Transaction: Purchase, booking, submission, payment
  • Data Operations: Create, update, delete critical entities
  • User Settings: Profile update, preferences, notifications
  • Error Recovery: Form validation, API errors, network issues

Locator Strategy Checklist

  • Use getByRole() as primary locator strategy
  • Use getByLabel() for form inputs
  • Use getByPlaceholder() when no label available
  • Use getByTestId() only as last resort
  • AVOID CSS selectors for user interactions
  • AVOID XPath locators
  • AVOID page.click('[data-testid=...]') - use getByTestId instead

Test Implementation Checklist

For each test:

  • Clear, descriptive test name
  • Tests one user flow or scenario
  • Uses semantic locators (getByRole, getByLabel)
  • Waits for elements using Playwright's auto-wait
  • No hardcoded sleep() or wait() calls
  • Assertions use expect() with appropriate matchers
  • Test can run in isolation (no dependencies on other tests)

Page Object Checklist

For each page object:

  • Locators defined in constructor
  • Methods for user actions (login, submit, navigate)
  • Assertion methods (expectError, expectSuccess)
  • No direct page.click() calls - wrap in methods
  • TypeScript types for all methods

Configuration Checklist

  • Set baseURL in config
  • Configure browser(s) for testing
  • Set up authentication state project
  • Configure retries for CI (2-3 retries)
  • Enable failOnFlakyTests in CI
  • Set appropriate timeouts
  • Configure screenshot on failure

CI/CD Checklist

  • Tests run in CI pipeline
  • Artifacts (screenshots, traces) uploaded on failure
  • Tests parallelized with sharding
  • Auth state cached between runs
  • Web server waits for ready signal

Visual Regression Checklist

  • Screenshots stored in version control
  • Different screenshots per browser/platform
  • Mobile viewports tested
  • Dark mode tested (if applicable)
  • Threshold set for acceptable diff

Accessibility Checklist

  • axe-core integrated for a11y testing
  • Critical pages tested for violations
  • Forms have proper labels
  • Focus management tested
  • Keyboard navigation tested

Review Checklist

Before PR:

  • All tests pass locally
  • Tests are deterministic (no flakes)
  • Locators follow semantic strategy
  • No hardcoded waits
  • Test files organized logically
  • Page objects used for complex pages
  • CI configuration updated if needed

Anti-Patterns to Avoid

  • Too many E2E tests (keep it focused)
  • Testing non-critical paths
  • Hard-coded waits (await page.waitForTimeout())
  • CSS/XPath selectors for interactions
  • Tests that depend on each other
  • Tests that modify global state
  • Ignoring flaky test warnings

E2e Testing Checklist

E2E Testing Checklist

Comprehensive checklist for planning, implementing, and maintaining E2E tests with Playwright.

Pre-Implementation

Test Planning

  • Identify critical user journeys to test
  • Map out happy paths and error scenarios
  • Determine test data requirements
  • Decide on mocking strategy (API, SSE, external services)
  • Plan for visual regression testing needs
  • Identify accessibility requirements (WCAG 2.1 AA)
  • Estimate test execution time and CI impact

Environment Setup

  • Install Playwright (npm install -D @playwright/test)
  • Install browser binaries (npx playwright install)
  • Create playwright.config.ts with base URL and timeouts
  • Configure test directory structure (tests/e2e/)
  • Set up Page Object pattern structure
  • Configure CI environment (GitHub Actions, GitLab CI, etc.)
  • Set up test database/backend for integration tests

Test Data Strategy

  • Create fixtures for common test scenarios
  • Set up database seeding scripts
  • Plan API mocking approach (mock server vs route interception)
  • Create reusable test data generators
  • Handle authentication/authorization test cases
  • Plan for cleanup between tests

Test Implementation

Page Objects

  • Create base page class with common utilities
  • Implement page object for each major page/component
  • Use semantic locators (role, label, test-id)
  • Avoid brittle CSS/XPath selectors
  • Encapsulate complex interactions in helper methods
  • Add TypeScript types for type safety
  • Document page object APIs

Test Structure

  • Follow Arrange-Act-Assert (AAA) pattern
  • Use descriptive test names (should/when/given format)
  • Group related tests with test.describe()
  • Set up common state in beforeEach()
  • Clean up resources in afterEach()
  • Use test fixtures for shared setup
  • Keep tests independent (no test interdependencies)

Assertions

  • Use specific assertions (toHaveText vs toBeTruthy)
  • Assert on user-visible behavior, not implementation
  • Verify loading states appear and disappear
  • Check error messages and validation feedback
  • Validate success states and confirmations
  • Test navigation and URL changes
  • Verify data persistence across page loads

API Interactions

  • Mock external API calls for reliability
  • Test real API endpoints in integration tests
  • Handle async operations properly (promises, awaits)
  • Test timeout scenarios
  • Verify retry logic
  • Test rate limiting behavior
  • Mock SSE/WebSocket streams

SSE/Real-Time Features

  • Test SSE connection establishment
  • Verify progress updates stream correctly
  • Test reconnection on connection drop
  • Handle SSE error events
  • Test SSE completion and cleanup
  • Verify UI updates from SSE events
  • Test SSE with network throttling

Error Handling

  • Test form validation errors
  • Test API error responses (400, 500, etc.)
  • Test network failures
  • Test timeout scenarios
  • Verify error messages shown to user
  • Test retry/recovery mechanisms
  • Test graceful degradation

Loading States

  • Test loading spinners appear
  • Verify skeleton screens render
  • Test loading state timeouts
  • Check loading states disappear on completion
  • Test loading state cancellation
  • Verify loading indicators are accessible

Responsive Design

  • Test on desktop viewports (1920x1080, 1366x768)
  • Test on tablet viewports (768x1024, 1024x768)
  • Test on mobile viewports (375x667, 414x896)
  • Verify touch interactions on mobile
  • Test responsive navigation menus
  • Verify content reflow on viewport changes
  • Test orientation changes (portrait/landscape)

Accessibility

  • Test keyboard navigation (Tab, Enter, Escape, arrows)
  • Verify focus management (focus visible, focus traps)
  • Test screen reader announcements (aria-live, role=status)
  • Check ARIA labels and descriptions
  • Test color contrast (use automated tools)
  • Verify form labels and error associations
  • Test with browser accessibility extensions
  • Consider adding axe-core integration

Visual Regression

  • Identify components/pages for screenshot testing
  • Set up baseline screenshots
  • Configure pixel diff thresholds
  • Test responsive breakpoints visually
  • Test theme variations (light/dark mode)
  • Test different locales (i18n)
  • Update baselines when designs change

Code Quality

Test Maintainability

  • Avoid test duplication (use helpers, fixtures)
  • Use constants for magic strings/numbers
  • Keep tests readable (avoid over-abstraction)
  • Add comments for complex test logic
  • Refactor brittle tests
  • Remove flaky tests or fix root cause
  • Review test coverage regularly

Performance

  • Run tests in parallel where possible
  • Minimize test execution time (mock slow APIs)
  • Use test.describe.configure(\{ mode: 'parallel' \})
  • Avoid unnecessary waits (waitForTimeout)
  • Use strategic waits (waitForSelector, waitForLoadState)
  • Optimize page load times (disable unnecessary assets)
  • Profile slow tests and optimize

Flakiness Prevention

  • Use deterministic waits (waitFor* methods)
  • Avoid race conditions (wait for element visibility)
  • Handle timing issues (debounce, throttle)
  • Retry flaky tests in CI (max 2 retries)
  • Investigate and fix root cause of flakiness
  • Use test.slow() for long-running tests
  • Increase timeouts for legitimate slow operations

CI/CD Integration

Pipeline Configuration

  • Add E2E test job to CI pipeline
  • Run tests on every PR
  • Block merge on test failures
  • Run tests against staging environment
  • Configure test parallelization in CI
  • Set up test result reporting
  • Archive test artifacts (videos, screenshots, traces)

Environment Management

  • Use Docker Compose for backend services
  • Seed test database before test run
  • Run migrations before tests
  • Clean up test data after run
  • Use environment variables for config
  • Isolate test environments (per PR if possible)
  • Monitor test environment health

Monitoring & Reporting

  • Generate HTML test reports
  • Upload test artifacts to CI
  • Send notifications on test failures
  • Track test execution time trends
  • Monitor test flakiness rates
  • Set up dashboard for test metrics
  • Alert on sustained test failures

OrchestKit-Specific

Analysis Flow Tests

  • Test URL submission with validation
  • Test analysis progress SSE stream
  • Verify agent status updates (8 agents)
  • Test progress bar updates (0% to 100%)
  • Test analysis completion detection
  • Test artifact generation
  • Test navigation to artifact view

Agent Orchestration

  • Verify supervisor assigns tasks
  • Test worker agent execution
  • Verify quality gate checks
  • Test agent failure handling
  • Test partial completion scenarios
  • Verify agent status badges

Artifact Display

  • Test artifact metadata display
  • Verify quality scores shown
  • Test findings/recommendations rendering
  • Test artifact search functionality
  • Test section navigation (tabs)
  • Test download artifact feature
  • Test share/copy link feature

Error Scenarios

  • Test invalid URL submission
  • Test network timeout during analysis
  • Test SSE connection drop
  • Test analysis cancellation
  • Test concurrent analysis limit
  • Test backend service unavailable
  • Test rate limiting

Performance Tests

  • Test with large artifact (many findings)
  • Test SSE with high event frequency
  • Test concurrent analyses (multiple tabs)
  • Test long-running analysis (timeout)
  • Monitor memory leaks during SSE stream

Maintenance

Regular Tasks

  • Review and update tests after feature changes
  • Update page objects when UI changes
  • Update test data when backend schema changes
  • Refactor duplicate test code
  • Remove obsolete tests
  • Update dependencies (Playwright, browsers)
  • Review test coverage and add missing tests

When Tests Fail

  • Check if failure is legitimate regression
  • Review CI logs and screenshots
  • Download and analyze trace files
  • Reproduce locally with --debug flag
  • Fix root cause (not just update assertions)
  • Add regression test if bug found
  • Update documentation if expected behavior changed

Optimization

  • Profile slow tests and optimize
  • Reduce unnecessary API calls
  • Optimize page object selectors
  • Minimize test data setup
  • Use test fixtures for common scenarios
  • Run critical tests first (fail fast)
  • Archive old test runs

Documentation

Test Documentation

  • Document test structure in README
  • Add comments for complex test logic
  • Document page object APIs
  • Create testing guide for contributors
  • Document CI pipeline configuration
  • Maintain test data documentation
  • Document mocking strategies

Knowledge Sharing

  • Share test results in PR reviews
  • Conduct test review sessions
  • Create troubleshooting guide
  • Document common test patterns
  • Share CI optimization learnings
  • Create onboarding guide for new contributors

Quality Gates

Before Committing

  • All tests pass locally
  • New tests added for new features
  • No new flaky tests introduced
  • Test execution time acceptable
  • Code reviewed for maintainability
  • Accessibility tests pass
  • Visual regression tests updated

Before Merging PR

  • All CI tests pass
  • No flaky test failures
  • Test coverage maintained or improved
  • Test artifacts reviewed (screenshots, videos)
  • Performance impact assessed
  • Breaking changes documented

Before Production Deploy

  • Full E2E suite passes on staging
  • Performance tests pass
  • Accessibility tests pass
  • Visual regression tests reviewed
  • Smoke tests identified for post-deploy
  • Rollback plan documented

Advanced Topics

Cross-Browser Testing

  • Test on Chromium (Chrome/Edge)
  • Test on Firefox
  • Test on WebKit (Safari)
  • Handle browser-specific quirks
  • Test with different browser versions

Internationalization (i18n)

  • Test with different locales
  • Verify RTL languages (Arabic, Hebrew)
  • Test date/time formatting
  • Test currency formatting
  • Verify translations loaded correctly

Security Testing

  • Test authentication flows
  • Test authorization (role-based access)
  • Test XSS prevention
  • Test CSRF protection
  • Test input sanitization
  • Test secure headers (CSP, etc.)

Performance Testing

  • Measure page load time
  • Test Core Web Vitals (LCP, FID, CLS)
  • Test with network throttling
  • Test with CPU throttling
  • Monitor memory usage
  • Test bundle size impact

Success Metrics

  • Test coverage > 80% for critical paths
  • Test execution time < 10 minutes
  • Test flakiness rate < 2%
  • Zero P0 bugs in production from untested areas
  • All critical user journeys tested
  • 100% of new features have E2E tests
  • Test results visible in every PR
  • Tests block merge on failure

Note: This checklist is comprehensive but should be adapted to your project's specific needs. Not all items apply to every project. Prioritize based on risk, criticality, and available resources.

OrchestKit Priority:

  1. Analysis flow (URL → Progress → Artifact)
  2. SSE real-time updates
  3. Error handling and recovery
  4. Agent orchestration visibility
  5. Accessibility and responsive design

Examples (3)

A11y Testing Examples

Accessibility Testing Examples

Complete code examples for automated accessibility testing.

jest-axe Component Tests

Basic Button Test

// src/components/Button.test.tsx
import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { Button } from './Button';

expect.extend(toHaveNoViolations);

describe('Button Accessibility', () => {
  test('has no accessibility violations', async () => {
    const { container } = render(<Button>Click me</Button>);
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('disabled button is accessible', async () => {
    const { container } = render(<Button disabled>Cannot click</Button>);
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('icon-only button has accessible name', async () => {
    const { container } = render(
      <Button aria-label="Close dialog">
        <XIcon />
      </Button>
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });
});

Form Component Test

// src/components/LoginForm.test.tsx
import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { LoginForm } from './LoginForm';

expect.extend(toHaveNoViolations);

describe('LoginForm Accessibility', () => {
  test('form has no accessibility violations', async () => {
    const { container } = render(<LoginForm />);
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('form with errors is accessible', async () => {
    const { container } = render(
      <LoginForm
        errors={{
          email: 'Invalid email address',
          password: 'Password is required',
        }}
      />
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('form with loading state is accessible', async () => {
    const { container } = render(<LoginForm isLoading />);
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('meets WCAG 2.1 Level AA', async () => {
    const { container } = render(<LoginForm />);
    const results = await axe(container, {
      runOnly: {
        type: 'tag',
        values: ['wcag2a', 'wcag2aa', 'wcag21aa'],
      },
    });
    expect(results).toHaveNoViolations();
  });
});
// src/components/Modal.test.tsx
import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { Modal } from './Modal';

expect.extend(toHaveNoViolations);

describe('Modal Accessibility', () => {
  test('open modal has no violations', async () => {
    const { container } = render(
      <Modal isOpen onClose={() => {}}>
        <h2>Modal Title</h2>
        <p>Modal content</p>
      </Modal>
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('modal has proper ARIA attributes', async () => {
    const { container } = render(
      <Modal isOpen onClose={() => {}} ariaLabel="Settings">
        <p>Settings content</p>
      </Modal>
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('modal with complex content is accessible', async () => {
    const { container } = render(
      <Modal isOpen onClose={() => {}}>
        <h2>Complex Modal</h2>
        <form>
          <label htmlFor="name">Name</label>
          <input id="name" type="text" />
          <button type="submit">Save</button>
        </form>
      </Modal>
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });
});

Custom Dropdown Test

// src/components/Dropdown.test.tsx
import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { axe, toHaveNoViolations } from 'jest-axe';
import { Dropdown } from './Dropdown';

expect.extend(toHaveNoViolations);

describe('Dropdown Accessibility', () => {
  const options = [
    { value: 'apple', label: 'Apple' },
    { value: 'banana', label: 'Banana' },
    { value: 'cherry', label: 'Cherry' },
  ];

  test('closed dropdown has no violations', async () => {
    const { container } = render(
      <Dropdown label="Select fruit" options={options} />
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('open dropdown has no violations', async () => {
    const user = userEvent.setup();
    const { container } = render(
      <Dropdown label="Select fruit" options={options} />
    );

    const button = screen.getByRole('button', { name: /select fruit/i });
    await user.click(button);

    await waitFor(async () => {
      const results = await axe(container);
      expect(results).toHaveNoViolations();
    });
  });

  test('dropdown with selected value is accessible', async () => {
    const { container } = render(
      <Dropdown
        label="Select fruit"
        options={options}
        value="banana"
      />
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  test('disabled dropdown is accessible', async () => {
    const { container } = render(
      <Dropdown
        label="Select fruit"
        options={options}
        disabled
      />
    );
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });
});

Playwright + axe-core E2E Tests

Page-Level Test

// tests/a11y/homepage.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Homepage Accessibility', () => {
  test('should not have accessibility violations', async ({ page }) => {
    await page.goto('/');

    const accessibilityScanResults = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])
      .analyze();

    expect(accessibilityScanResults.violations).toEqual([]);
  });

  test('navigation menu is accessible', async ({ page }) => {
    await page.goto('/');

    // Scan only the navigation
    const results = await new AxeBuilder({ page })
      .include('nav')
      .analyze();

    expect(results.violations).toEqual([]);
  });

  test('footer is accessible', async ({ page }) => {
    await page.goto('/');

    const results = await new AxeBuilder({ page })
      .include('footer')
      .analyze();

    expect(results.violations).toEqual([]);
  });
});

User Journey Test

// tests/a11y/checkout.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Checkout Flow Accessibility', () => {
  test('entire checkout flow is accessible', async ({ page }) => {
    // Step 1: Cart page
    await page.goto('/cart');
    let results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa'])
      .analyze();
    expect(results.violations).toEqual([]);

    // Step 2: Add item and proceed
    await page.getByRole('button', { name: 'Proceed to Checkout' }).click();

    // Step 3: Shipping form
    await page.waitForURL('/checkout/shipping');
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Fill form
    await page.getByLabel('Email').fill('test@example.com');
    await page.getByLabel('Street Address').fill('123 Main St');
    await page.getByRole('button', { name: 'Continue to Payment' }).click();

    // Step 4: Payment form
    await page.waitForURL('/checkout/payment');
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Step 5: Review order
    await page.getByRole('button', { name: 'Review Order' }).click();
    await page.waitForURL('/checkout/review');
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);
  });

  test('validation errors are accessible', async ({ page }) => {
    await page.goto('/checkout/shipping');

    // Submit without filling required fields
    await page.getByRole('button', { name: 'Continue' }).click();

    // Wait for error messages to appear
    await page.waitForSelector('[role="alert"]');

    const results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);
  });
});

Dynamic Content Test

// tests/a11y/search.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Search Accessibility', () => {
  test('search interface is accessible', async ({ page }) => {
    await page.goto('/search');

    // Initial state
    let results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Type search query
    await page.getByRole('searchbox', { name: 'Search products' }).fill('laptop');

    // Wait for autocomplete suggestions
    await page.waitForSelector('[role="listbox"]');

    // Scan with suggestions visible
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Select a suggestion
    await page.getByRole('option', { name: /laptop/i }).first().click();

    // Wait for results page
    await page.waitForURL('**/search?q=laptop');

    // Scan results page
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);
  });

  test('empty search results accessible', async ({ page }) => {
    await page.goto('/search?q=nonexistentproduct123');

    const results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);
  });
});
// tests/a11y/modal.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Modal Accessibility', () => {
  test('modal maintains accessibility through interactions', async ({ page }) => {
    await page.goto('/dashboard');

    // Initial state (modal closed)
    let results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Open modal
    await page.getByRole('button', { name: 'Open Settings' }).click();
    await page.waitForSelector('[role="dialog"]');

    // Modal open state
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Interact with modal form
    await page.getByLabel('Display Name').fill('John Doe');
    await page.getByLabel('Email Notifications').check();

    // Still accessible after interactions
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);

    // Close modal
    await page.getByRole('button', { name: 'Save' }).click();
    await page.waitForSelector('[role="dialog"]', { state: 'hidden' });

    // After modal closes
    results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);
  });

  test('focus is trapped in modal', async ({ page }) => {
    await page.goto('/dashboard');
    await page.getByRole('button', { name: 'Open Settings' }).click();
    await page.waitForSelector('[role="dialog"]');

    // Tab through all elements
    const focusableElements = await page.locator('[role="dialog"] :focus-visible').count();

    for (let i = 0; i < focusableElements + 2; i++) {
      await page.keyboard.press('Tab');
    }

    // Focus should still be within modal
    const focusedElement = await page.evaluate(() => {
      const activeElement = document.activeElement;
      return activeElement?.closest('[role="dialog"]') !== null;
    });

    expect(focusedElement).toBe(true);
  });
});

Custom axe Rules

Creating a Custom Rule

// tests/utils/custom-axe-rules.ts
import { configureAxe } from 'jest-axe';

export const axeWithCustomRules = configureAxe({
  rules: {
    // Ensure all buttons have explicit type attribute
    'button-type': {
      enabled: true,
      selector: 'button:not([type])',
      any: [],
      none: [],
      all: ['button-has-type'],
    },
  },
  checks: [
    {
      id: 'button-has-type',
      evaluate: () => false,
      metadata: {
        impact: 'minor',
        messages: {
          fail: 'Button must have explicit type attribute (button, submit, or reset)',
        },
      },
    },
  ],
});

Using Custom Rules in Tests

// src/components/Form.test.tsx
import { render } from '@testing-library/react';
import { toHaveNoViolations } from 'jest-axe';
import { axeWithCustomRules } from '../tests/utils/custom-axe-rules';

expect.extend(toHaveNoViolations);

test('form buttons have explicit type', async () => {
  const { container } = render(
    <form>
      <button type="button">Cancel</button>
      <button type="submit">Submit</button>
    </form>
  );

  const results = await axeWithCustomRules(container);
  expect(results).toHaveNoViolations();
});

CI Pipeline Configuration

GitHub Actions Workflow

# .github/workflows/a11y-tests.yml
name: Accessibility Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  unit-a11y:
    name: Unit Accessibility Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run jest-axe tests
        run: npm run test:a11y:unit

      - name: Upload coverage
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage/lcov.info
          flags: accessibility

  e2e-a11y:
    name: E2E Accessibility Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps chromium

      - name: Build application
        run: npm run build
        env:
          CI: true

      - name: Start application
        run: npm run start &
        env:
          PORT: 3000
          NODE_ENV: test

      - name: Wait for application
        run: npx wait-on http://localhost:3000 --timeout 60000

      - name: Run Playwright accessibility tests
        run: npx playwright test tests/a11y/

      - name: Upload Playwright report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-a11y-report
          path: playwright-report/
          retention-days: 30

      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = fs.readFileSync('playwright-report/index.html', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '## ♿ Accessibility Test Results\n\nView full report in artifacts.'
            });

  lighthouse:
    name: Lighthouse Accessibility Audit
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Build application
        run: npm run build

      - name: Start application
        run: npm run start &

      - name: Wait for application
        run: npx wait-on http://localhost:3000

      - name: Run Lighthouse CI
        run: |
          npm install -g @lhci/cli@0.13.x
          lhci autorun
        env:
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}

      - name: Upload Lighthouse results
        uses: actions/upload-artifact@v4
        with:
          name: lighthouse-results
          path: .lighthouseci/

Package.json Test Scripts

{
  "scripts": {
    "test:a11y:unit": "vitest run --coverage src/**/*.a11y.test.{ts,tsx}",
    "test:a11y:unit:watch": "vitest watch src/**/*.a11y.test.{ts,tsx}",
    "test:a11y:e2e": "playwright test tests/a11y/",
    "test:a11y:all": "npm run test:a11y:unit && npm run test:a11y:e2e",
    "test:a11y:lighthouse": "lhci autorun"
  }
}

These examples provide a comprehensive foundation for implementing automated accessibility testing in your application.

E2e Test Patterns

E2E Test Patterns

Complete User Flow Test

import { test, expect } from '@playwright/test';

test.describe('Checkout Flow', () => {
  test('user can complete purchase', async ({ page }) => {
    // Navigate to product
    await page.goto('/products');
    await page.getByRole('link', { name: 'Premium Widget' }).click();

    // Add to cart
    await page.getByRole('button', { name: 'Add to cart' }).click();
    await expect(page.getByRole('alert')).toContainText('Added to cart');

    // Go to checkout
    await page.getByRole('link', { name: 'Cart' }).click();
    await page.getByRole('button', { name: 'Checkout' }).click();

    // Fill shipping info
    await page.getByLabel('Email').fill('test@example.com');
    await page.getByLabel('Full name').fill('Test User');
    await page.getByLabel('Address').fill('123 Test St');
    await page.getByLabel('City').fill('Test City');
    await page.getByRole('combobox', { name: 'State' }).selectOption('CA');
    await page.getByLabel('ZIP').fill('90210');

    // Fill payment
    await page.getByLabel('Card number').fill('4242424242424242');
    await page.getByLabel('Expiry').fill('12/25');
    await page.getByLabel('CVC').fill('123');

    // Submit order
    await page.getByRole('button', { name: 'Place order' }).click();

    // Verify confirmation
    await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
    await expect(page.getByText(/order #/i)).toBeVisible();
  });
});

Page Object Model

// pages/LoginPage.ts
import { Page, Locator, expect } from '@playwright/test';

export class LoginPage {
  private readonly emailInput: Locator;
  private readonly passwordInput: Locator;
  private readonly submitButton: Locator;
  private readonly errorMessage: Locator;

  constructor(private page: Page) {
    this.emailInput = page.getByLabel('Email');
    this.passwordInput = page.getByLabel('Password');
    this.submitButton = page.getByRole('button', { name: 'Sign in' });
    this.errorMessage = page.getByRole('alert');
  }

  async goto() {
    await this.page.goto('/login');
  }

  async login(email: string, password: string) {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.submitButton.click();
  }

  async expectError(message: string) {
    await expect(this.errorMessage).toContainText(message);
  }

  async expectLoggedIn() {
    await expect(this.page).toHaveURL('/dashboard');
  }
}

// tests/login.spec.ts
import { test } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';

test.describe('Login', () => {
  test('successful login', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login('user@example.com', 'password123');
    await loginPage.expectLoggedIn();
  });

  test('invalid credentials', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login('user@example.com', 'wrongpassword');
    await loginPage.expectError('Invalid email or password');
  });
});

Authentication Fixture

// fixtures/auth.ts
import { test as base, Page } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';

type AuthFixtures = {
  authenticatedPage: Page;
  adminPage: Page;
};

export const test = base.extend<AuthFixtures>({
  authenticatedPage: async ({ page }, use) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login('user@example.com', 'password123');
    await use(page);
  },
  
  adminPage: async ({ page }, use) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login('admin@example.com', 'adminpass');
    await use(page);
  },
});

// tests/dashboard.spec.ts
import { test } from '../fixtures/auth';

test('user can view dashboard', async ({ authenticatedPage }) => {
  await authenticatedPage.goto('/dashboard');
  // Already logged in
});

test('admin can access admin panel', async ({ adminPage }) => {
  await adminPage.goto('/admin');
  // Already logged in as admin
});

Visual Regression Test

import { test, expect } from '@playwright/test';

test.describe('Visual Regression', () => {
  test('homepage looks correct', async ({ page }) => {
    await page.goto('/');
    await expect(page).toHaveScreenshot('homepage.png');
  });

  test('hero section visual', async ({ page }) => {
    await page.goto('/');
    const hero = page.locator('[data-testid="hero"]');
    await expect(hero).toHaveScreenshot('hero.png');
  });

  test('responsive design - mobile', async ({ page }) => {
    await page.setViewportSize({ width: 375, height: 667 });
    await page.goto('/');
    await expect(page).toHaveScreenshot('homepage-mobile.png');
  });

  test('dark mode', async ({ page }) => {
    await page.emulateMedia({ colorScheme: 'dark' });
    await page.goto('/');
    await expect(page).toHaveScreenshot('homepage-dark.png');
  });
});

API Mocking in E2E

import { test, expect } from '@playwright/test';

test('handles API error gracefully', async ({ page }) => {
  // Mock API to return error
  await page.route('/api/users', (route) => {
    route.fulfill({
      status: 500,
      body: JSON.stringify({ error: 'Server error' }),
    });
  });

  await page.goto('/users');
  await expect(page.getByText('Unable to load users')).toBeVisible();
  await expect(page.getByRole('button', { name: 'Retry' })).toBeVisible();
});

test('shows loading state', async ({ page }) => {
  // Delay API response
  await page.route('/api/users', async (route) => {
    await new Promise((resolve) => setTimeout(resolve, 2000));
    route.fulfill({
      status: 200,
      body: JSON.stringify([{ id: 1, name: 'User' }]),
    });
  });

  await page.goto('/users');
  await expect(page.getByTestId('loading-skeleton')).toBeVisible();
  await expect(page.getByText('User')).toBeVisible({ timeout: 5000 });
});

Multi-Tab Test

import { test, expect } from '@playwright/test';

test('multi-tab checkout flow', async ({ context }) => {
  // Open two tabs
  const page1 = await context.newPage();
  const page2 = await context.newPage();

  // Add item in first tab
  await page1.goto('/products');
  await page1.getByRole('button', { name: 'Add to cart' }).click();

  // Verify cart updated in second tab
  await page2.goto('/cart');
  await expect(page2.getByRole('listitem')).toHaveCount(1);
});

File Upload Test

import { test, expect } from '@playwright/test';
import path from 'path';

test('user can upload profile photo', async ({ page }) => {
  await page.goto('/settings/profile');

  // Upload file
  const fileInput = page.locator('input[type="file"]');
  await fileInput.setInputFiles(path.join(__dirname, 'fixtures/photo.jpg'));

  // Verify preview
  await expect(page.getByAltText('Profile preview')).toBeVisible();

  // Save
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByRole('alert')).toContainText('Profile updated');
});

Accessibility Test

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Accessibility', () => {
  test('homepage has no a11y violations', async ({ page }) => {
    await page.goto('/');
    
    const results = await new AxeBuilder({ page }).analyze();
    expect(results.violations).toEqual([]);
  });

  test('login form is accessible', async ({ page }) => {
    await page.goto('/login');
    
    const results = await new AxeBuilder({ page })
      .include('[data-testid="login-form"]')
      .analyze();
    
    expect(results.violations).toEqual([]);
  });
});

Orchestkit E2e Tests

OrchestKit E2E Test Examples

Complete E2E test suite examples for OrchestKit's analysis workflow using Playwright + TypeScript.

Test Configuration

playwright.config.ts

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests/e2e',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',

  use: {
    baseURL: 'http://localhost:5173',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
    {
      name: 'mobile',
      use: { ...devices['iPhone 13'] },
    },
  ],

  webServer: {
    command: 'npm run dev',
    url: 'http://localhost:5173',
    reuseExistingServer: !process.env.CI,
  },
});

Page Objects

HomePage (URL Submission)

// tests/e2e/pages/HomePage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage } from '.claude/skills/webapp-testing/assets/playwright-test-template';

export class HomePage extends BasePage {
  readonly urlInput: Locator;
  readonly analyzeButton: Locator;
  readonly analysisTypeSelect: Locator;
  readonly recentAnalyses: Locator;

  constructor(page: Page) {
    super(page);
    this.urlInput = page.getByTestId('url-input');
    this.analyzeButton = page.getByRole('button', { name: /analyze/i });
    this.analysisTypeSelect = page.getByTestId('analysis-type-select');
    this.recentAnalyses = page.getByTestId('recent-analyses-list');
  }

  async goto(): Promise<void> {
    await super.goto('/');
    await this.waitForLoad();
  }

  async submitUrl(url: string, analysisType = 'comprehensive'): Promise<void> {
    await this.urlInput.fill(url);
    if (analysisType !== 'comprehensive') {
      await this.analysisTypeSelect.selectOption(analysisType);
    }
    await this.analyzeButton.click();
  }

  async getRecentAnalysesCount(): Promise<number> {
    return await this.recentAnalyses.locator('li').count();
  }

  async clickRecentAnalysis(index: number): Promise<void> {
    await this.recentAnalyses.locator('li').nth(index).click();
  }
}

AnalysisProgressPage (SSE Stream)

// tests/e2e/pages/AnalysisProgressPage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage, WaitHelpers } from '.claude/skills/webapp-testing/assets/playwright-test-template';

export class AnalysisProgressPage extends BasePage {
  readonly progressBar: Locator;
  readonly progressPercentage: Locator;
  readonly statusBadge: Locator;
  readonly agentCards: Locator;
  readonly errorMessage: Locator;
  readonly cancelButton: Locator;
  readonly viewArtifactButton: Locator;

  private waitHelpers: WaitHelpers;

  constructor(page: Page) {
    super(page);
    this.progressBar = page.getByTestId('analysis-progress-bar');
    this.progressPercentage = page.getByTestId('progress-percentage');
    this.statusBadge = page.getByTestId('status-badge');
    this.agentCards = page.getByTestId('agent-card');
    this.errorMessage = page.getByTestId('error-message');
    this.cancelButton = page.getByRole('button', { name: /cancel/i });
    this.viewArtifactButton = page.getByRole('button', { name: /view artifact/i });
    this.waitHelpers = new WaitHelpers(page);
  }

  async waitForAnalysisComplete(timeout = 60000): Promise<void> {
    await this.page.waitForFunction(
      () => {
        const badge = document.querySelector('[data-testid="status-badge"]');
        return badge?.textContent?.toLowerCase().includes('complete');
      },
      { timeout }
    );
  }

  async waitForProgress(percentage: number, timeout = 30000): Promise<void> {
    await this.page.waitForFunction(
      (targetPercentage) => {
        const progressText = document.querySelector('[data-testid="progress-percentage"]')?.textContent;
        const currentPercentage = parseInt(progressText || '0', 10);
        return currentPercentage >= targetPercentage;
      },
      percentage,
      { timeout }
    );
  }

  async getAgentStatus(agentName: string): Promise<'pending' | 'running' | 'completed' | 'failed'> {
    const agentCard = this.agentCards.filter({ hasText: agentName }).first();
    const statusElement = agentCard.getByTestId('agent-status');
    const status = await statusElement.textContent();
    return status?.toLowerCase() as any;
  }

  async getCompletedAgentsCount(): Promise<number> {
    return await this.agentCards.filter({ has: this.page.getByText('completed') }).count();
  }

  async cancelAnalysis(): Promise<void> {
    await this.cancelButton.click();
  }

  async goToArtifact(): Promise<void> {
    await this.viewArtifactButton.click();
  }

  async getErrorText(): Promise<string | null> {
    if (await this.errorMessage.isVisible()) {
      return await this.errorMessage.textContent();
    }
    return null;
  }
}

ArtifactPage (View Results)

// tests/e2e/pages/ArtifactPage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage } from '.claude/skills/webapp-testing/assets/playwright-test-template';

export class ArtifactPage extends BasePage {
  readonly artifactTitle: Locator;
  readonly sourceUrl: Locator;
  readonly qualityScore: Locator;
  readonly findingsSection: Locator;
  readonly downloadButton: Locator;
  readonly shareButton: Locator;
  readonly searchInput: Locator;
  readonly sectionTabs: Locator;

  constructor(page: Page) {
    super(page);
    this.artifactTitle = page.getByTestId('artifact-title');
    this.sourceUrl = page.getByTestId('source-url');
    this.qualityScore = page.getByTestId('quality-score');
    this.findingsSection = page.getByTestId('findings-section');
    this.downloadButton = page.getByRole('button', { name: /download/i });
    this.shareButton = page.getByRole('button', { name: /share/i });
    this.searchInput = page.getByTestId('artifact-search');
    this.sectionTabs = page.getByRole('tab');
  }

  async getQualityScoreValue(): Promise<number> {
    const scoreText = await this.qualityScore.textContent();
    return parseFloat(scoreText || '0');
  }

  async searchInArtifact(query: string): Promise<void> {
    await this.searchInput.fill(query);
    await this.page.waitForTimeout(300); // Debounce
  }

  async switchToTab(tabName: string): Promise<void> {
    await this.sectionTabs.filter({ hasText: tabName }).click();
  }

  async downloadArtifact(): Promise<void> {
    const downloadPromise = this.page.waitForEvent('download');
    await this.downloadButton.click();
    await downloadPromise;
  }

  async getFindingsCount(): Promise<number> {
    return await this.findingsSection.locator('[data-testid="finding-item"]').count();
  }
}

Test Suites

1. Happy Path - Complete Analysis Flow

// tests/e2e/analysis-flow.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
import { ArtifactPage } from './pages/ArtifactPage';
import { ApiMocker, CustomAssertions } from '.claude/skills/webapp-testing/assets/playwright-test-template';

test.describe('Analysis Flow - Happy Path', () => {
  test('should complete full analysis flow from URL submission to artifact view', async ({ page }) => {
    // 1. Submit URL for analysis
    const homePage = new HomePage(page);
    await homePage.goto();

    await expect(homePage.urlInput).toBeVisible();
    await homePage.submitUrl('https://example.com/article', 'comprehensive');

    // 2. Monitor progress with SSE
    const progressPage = new AnalysisProgressPage(page);
    await expect(progressPage.progressBar).toBeVisible();

    // Wait for initial progress
    await progressPage.waitForProgress(10);

    // Check at least one agent is running
    const agentStatus = await progressPage.getAgentStatus('Tech Comparator');
    expect(['running', 'completed']).toContain(agentStatus);

    // Wait for completion (with timeout for real API)
    await progressPage.waitForAnalysisComplete(90000); // 90s timeout

    // Verify all agents completed
    const completedCount = await progressPage.getCompletedAgentsCount();
    expect(completedCount).toBeGreaterThan(0);

    // 3. Navigate to artifact
    await progressPage.goToArtifact();

    // 4. Verify artifact content
    const artifactPage = new ArtifactPage(page);
    await expect(artifactPage.artifactTitle).toBeVisible();

    const qualityScore = await artifactPage.getQualityScoreValue();
    expect(qualityScore).toBeGreaterThan(0);
    expect(qualityScore).toBeLessThanOrEqual(10);

    const findingsCount = await artifactPage.getFindingsCount();
    expect(findingsCount).toBeGreaterThan(0);
  });
});

2. SSE Progress Updates

// tests/e2e/sse-progress.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
import { ApiMocker } from '.claude/skills/webapp-testing/assets/playwright-test-template';

test.describe('SSE Progress Updates', () => {
  test('should show real-time progress updates via SSE', async ({ page }) => {
    // Mock SSE stream with progress events
    const apiMocker = new ApiMocker(page);

    const sseEvents = [
      { data: { type: 'progress', percentage: 0, message: 'Starting analysis...' } },
      { data: { type: 'agent_start', agent: 'Tech Comparator' }, delay: 500 },
      { data: { type: 'progress', percentage: 25, message: 'Tech Comparator running...' } },
      { data: { type: 'agent_complete', agent: 'Tech Comparator' }, delay: 1000 },
      { data: { type: 'progress', percentage: 50, message: 'Security Auditor running...' } },
      { data: { type: 'agent_complete', agent: 'Security Auditor' }, delay: 1000 },
      { data: { type: 'progress', percentage: 100, message: 'Analysis complete!' } },
      { data: { type: 'complete', artifact_id: 'test-artifact-123' } },
    ];

    await apiMocker.mockSSE(/api\/v1\/analyses\/\d+\/stream/, sseEvents);

    // Submit analysis
    const homePage = new HomePage(page);
    await homePage.goto();
    await homePage.submitUrl('https://example.com/test');

    // Monitor progress updates
    const progressPage = new AnalysisProgressPage(page);

    // Wait for 25% progress
    await progressPage.waitForProgress(25);
    expect(await progressPage.progressPercentage.textContent()).toContain('25');

    // Wait for 50% progress
    await progressPage.waitForProgress(50);
    expect(await progressPage.progressPercentage.textContent()).toContain('50');

    // Wait for completion
    await progressPage.waitForProgress(100);
    await expect(progressPage.statusBadge).toContainText('Complete');
  });

  test('should handle SSE connection errors gracefully', async ({ page }) => {
    // Mock SSE connection failure
    await page.route(/api\/v1\/analyses\/\d+\/stream/, (route) => {
      route.abort('failed');
    });

    const homePage = new HomePage(page);
    await homePage.goto();
    await homePage.submitUrl('https://example.com/test');

    const progressPage = new AnalysisProgressPage(page);

    // Should show error message
    await expect(progressPage.errorMessage).toBeVisible();
    const errorText = await progressPage.getErrorText();
    expect(errorText).toContain('connection');
  });
});

3. Error Handling

// tests/e2e/error-handling.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
import { ApiMocker, CustomAssertions } from '.claude/skills/webapp-testing/assets/playwright-test-template';

test.describe('Error Handling', () => {
  test('should show validation error for invalid URL', async ({ page }) => {
    const homePage = new HomePage(page);
    await homePage.goto();

    await homePage.submitUrl('not-a-valid-url');

    const assertions = new CustomAssertions(page);
    await assertions.expectToast('Please enter a valid URL', 'error');
  });

  test('should handle API error during analysis submission', async ({ page }) => {
    const apiMocker = new ApiMocker(page);
    await apiMocker.mockError(/api\/v1\/analyses/, 500, 'Internal server error');

    const homePage = new HomePage(page);
    await homePage.goto();
    await homePage.submitUrl('https://example.com/test');

    const assertions = new CustomAssertions(page);
    await assertions.expectToast('Failed to start analysis', 'error');
  });

  test('should handle analysis failure from backend', async ({ page }) => {
    const apiMocker = new ApiMocker(page);

    // Mock successful submission
    await apiMocker.mockSuccess(/api\/v1\/analyses$/, {
      id: 123,
      status: 'processing',
      url: 'https://example.com/test',
    });

    // Mock SSE with failure event
    await apiMocker.mockSSE(/api\/v1\/analyses\/123\/stream/, [
      { data: { type: 'progress', percentage: 10 } },
      { data: { type: 'error', message: 'Failed to fetch content' } },
    ]);

    const homePage = new HomePage(page);
    await homePage.goto();
    await homePage.submitUrl('https://example.com/test');

    const progressPage = new AnalysisProgressPage(page);
    await expect(progressPage.errorMessage).toBeVisible();
    const errorText = await progressPage.getErrorText();
    expect(errorText).toContain('Failed to fetch content');
  });

  test('should allow retry after failed analysis', async ({ page }) => {
    const homePage = new HomePage(page);
    const progressPage = new AnalysisProgressPage(page);

    await homePage.goto();
    await homePage.submitUrl('https://example.com/test');

    // Wait for error state
    await expect(progressPage.errorMessage).toBeVisible();

    // Click retry button
    const retryButton = page.getByRole('button', { name: /retry/i });
    await retryButton.click();

    // Should restart analysis
    await expect(progressPage.progressBar).toBeVisible();
  });
});

4. Cancellation & Cleanup

// tests/e2e/cancellation.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';

test.describe('Analysis Cancellation', () => {
  test('should cancel in-progress analysis', async ({ page }) => {
    const homePage = new HomePage(page);
    await homePage.goto();
    await homePage.submitUrl('https://example.com/long-analysis');

    const progressPage = new AnalysisProgressPage(page);

    // Wait for analysis to start
    await progressPage.waitForProgress(10);

    // Cancel analysis
    await progressPage.cancelAnalysis();

    // Confirm cancellation in dialog
    page.on('dialog', dialog => dialog.accept());

    // Should redirect back to home
    await expect(page).toHaveURL('/');

    // Should show cancellation toast
    const assertions = new CustomAssertions(page);
    await assertions.expectToast('Analysis cancelled', 'info');
  });

  test('should not allow cancellation of completed analysis', async ({ page }) => {
    // Navigate to completed analysis
    await page.goto('/analysis/completed-123');

    const progressPage = new AnalysisProgressPage(page);

    // Cancel button should be disabled or hidden
    await expect(progressPage.cancelButton).not.toBeVisible();
  });
});

5. Responsive & Mobile

// tests/e2e/responsive.spec.ts
import { test, expect, devices } from '@playwright/test';
import { HomePage } from './pages/HomePage';

test.describe('Responsive Design', () => {
  test.use({ ...devices['iPhone 13'] });

  test('should work on mobile viewport', async ({ page }) => {
    const homePage = new HomePage(page);
    await homePage.goto();

    // URL input should be visible and usable
    await expect(homePage.urlInput).toBeVisible();
    await homePage.urlInput.fill('https://example.com/mobile-test');

    // Button should be tappable
    await homePage.analyzeButton.click();

    // Progress page should be mobile-friendly
    const progressBar = page.getByTestId('analysis-progress-bar');
    await expect(progressBar).toBeVisible();

    // Agent cards should stack vertically
    const agentCards = page.getByTestId('agent-card');
    const firstCard = agentCards.first();
    const secondCard = agentCards.nth(1);

    const firstBox = await firstCard.boundingBox();
    const secondBox = await secondCard.boundingBox();

    // Second card should be below first (Y coordinate)
    expect(secondBox!.y).toBeGreaterThan(firstBox!.y + firstBox!.height);
  });
});

6. Accessibility

// tests/e2e/accessibility.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';

test.describe('Accessibility', () => {
  test('should be keyboard navigable', async ({ page }) => {
    const homePage = new HomePage(page);
    await homePage.goto();

    // Tab to URL input
    await page.keyboard.press('Tab');
    await expect(homePage.urlInput).toBeFocused();

    // Type URL
    await page.keyboard.type('https://example.com/test');

    // Tab to analyze button
    await page.keyboard.press('Tab');
    await expect(homePage.analyzeButton).toBeFocused();

    // Press Enter to submit
    await page.keyboard.press('Enter');

    // Should navigate to progress page
    const progressPage = new AnalysisProgressPage(page);
    await expect(progressPage.progressBar).toBeVisible();
  });

  test('should have proper ARIA labels', async ({ page }) => {
    const homePage = new HomePage(page);
    await homePage.goto();

    // URL input should have aria-label
    await expect(homePage.urlInput).toHaveAttribute('aria-label');

    // Submit button should have accessible name
    const buttonName = await homePage.analyzeButton.getAttribute('aria-label');
    expect(buttonName).toBeTruthy();
  });

  test('should announce progress updates to screen readers', async ({ page }) => {
    await page.goto('/analysis/123');

    const progressPage = new AnalysisProgressPage(page);

    // Progress region should have aria-live
    await expect(progressPage.progressBar).toHaveAttribute('aria-live', 'polite');

    // Status updates should have role="status"
    const statusRegion = page.getByTestId('status-updates');
    await expect(statusRegion).toHaveAttribute('role', 'status');
  });
});

Running Tests

# Install Playwright
npm install -D @playwright/test
npx playwright install

# Run all tests
npx playwright test

# Run specific suite
npx playwright test tests/e2e/analysis-flow.spec.ts

# Run in UI mode (interactive)
npx playwright test --ui

# Run in headed mode (see browser)
npx playwright test --headed

# Run on specific browser
npx playwright test --project=chromium

# Debug mode
npx playwright test --debug

# Generate test report
npx playwright show-report

CI Integration

# .github/workflows/e2e-tests.yml
name: E2E Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis:7
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Start backend
        run: |
          cd backend
          poetry install
          poetry run uvicorn app.main:app --host 0.0.0.0 --port 8500 &
          sleep 5

      - name: Start frontend
        run: |
          npm run build
          npm run preview &
          sleep 3

      - name: Run E2E tests
        run: npx playwright test

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

Best Practices

  1. Use Page Objects - Encapsulate page logic, improve maintainability
  2. Mock External APIs - Fast, reliable tests without network dependencies
  3. Wait Strategically - Use waitForSelector, avoid arbitrary timeouts
  4. Test Real Flows - Mirror actual user journeys
  5. Handle Async - SSE streams, debounced inputs, loading states
  6. Accessibility First - Test keyboard nav, ARIA, screen reader announcements
  7. Visual Regression - Screenshot testing for UI consistency
  8. CI Integration - Run tests on every PR, block merges on failures
Edit on GitHub

Last updated on

On this page

E2E Testing PatternsQuick ReferencePlaywright Quick StartPlaywright CorePage ObjectsAI AgentsAccessibility (Playwright)Accessibility (CI/CD)End-to-End TypesVisual RegressionKey DecisionsReferencesChecklistsExamplesScriptsRelated SkillsRules (6)Validate full-page accessibility compliance through Playwright E2E tests with axe-core — MEDIUMPlaywright + axe-core E2EKey DecisionsEnforce accessibility testing in CI pipelines and enable unit-level component testing with jest-axe — MEDIUMCI/CD Accessibility GatesAnti-Patterns (FORBIDDEN)Key Decisionsjest-axe Unit TestingSetupComponent TestingAnti-Patterns (FORBIDDEN)Key PatternsUse Playwright AI agent framework for test planning, generation, and self-healing — HIGHPlaywright AI Agents (1.58+)Initialize AI AgentsGenerated StructureAgent WorkflowKey ConceptsSetup RequirementsEncapsulate page interactions into reusable page object classes for maintainable E2E tests — HIGHPage Object ModelPatternVisual RegressionCritical User Journeys to TestApply semantic locator patterns and best practices for resilient Playwright E2E tests — HIGHPlaywright E2E Testing (1.58+)Semantic LocatorsBasic TestNew Features (1.58+)Anti-Patterns (FORBIDDEN)Key DecisionsValidate end-to-end type safety across API layers to eliminate runtime type errors — HIGHEnd-to-End Type Safety ValidationReferences (4)A11y Testing ToolsAccessibility Testing Tools Referencejest-axe ConfigurationInstallationSetupBasic UsageComponent-Specific RulesTesting Specific RulesPlaywright + axe-coreInstallationSetupE2E Accessibility TestTesting After InteractionsExcluding RegionsCI/CD IntegrationGitHub ActionsPre-commit HookPackage.json ScriptsManual Testing ChecklistKeyboard NavigationScreen Reader TestingColor ContrastResponsive and Zoom TestingContinuous MonitoringLighthouse CIaxe-cli for Quick ScansCommon PitfallsResourcesPlaywright 1.57 ApiPlaywright 1.58+ API ReferenceSemantic Locators (2026 Best Practice)Locator PriorityRole-Based LocatorsLabel-Based LocatorsText and PlaceholderTest IDs (Fallback)Breaking Changes (1.58)Removed FeaturesMigration ExamplesNew Features (1.58+)connectOverCDP with isLocalTimeline in Speedboard HTML ReportsNew Assertions (1.57+)AI Agents (1.58+)Initialize AI AgentsGenerated StructureConfigurationAuthentication StateStorage StateIndexedDB Support (1.57+)Auth Setup ProjectFlaky Test Detection (1.57+)Visual RegressionLocator Descriptions (1.57+)Chrome for Testing (1.57+)External LinksPlaywright SetupPlaywright Setup with Test AgentsPrerequisitesStep 1: Install PlaywrightStep 2: Add Playwright MCP Server (CC 2.1.6)Step 3: Initialize Test AgentsStep 4: Create Seed TestDirectory StructureBasic ConfigurationRunning TestsBrowser AutomationNext StepsVisual RegressionPlaywright Native Visual Regression TestingOverviewQuick StartConfiguration (playwright.config.ts)Essential SettingsSnapshot Path Template TokensTest PatternsBasic ScreenshotFull Page ScreenshotElement ScreenshotMasking Dynamic ContentCustom Styles for ScreenshotsResponsive ViewportsDark Mode TestingWaiting for StabilityCI/CD IntegrationGitHub Actions WorkflowHandling Baseline UpdatesHandling Cross-Platform IssuesThe ProblemSolutionsDebugging Failed ScreenshotsView Diff ReportGenerated Files on FailureTrace Viewer for ContextBest Practices1. Stable Selectors2. Wait for Stability3. Mask Dynamic Content4. Disable Animations5. Single Browser for VRT6. Meaningful NamesMigration from PercyQuick Migration ScriptTroubleshootingFlaky ScreenshotsCI vs Local DifferencesLarge Screenshot FilesChecklists (3)A11y Testing ChecklistAccessibility Testing ChecklistAutomated Test CoverageUnit Tests (jest-axe)E2E Tests (Playwright + axe-core)CI/CD IntegrationManual Testing RequirementsKeyboard NavigationScreen Reader TestingContent StructureInteractive ElementsNavigationColor and ContrastResponsive and Zoom TestingAnimation and MotionDocumentation ReviewCross-Browser TestingCompliance VerificationContinuous MonitoringWhen to Seek Expert HelpQuick Wins for Common IssuesMissing Alt TextUnlabeled Form InputLow Contrast TextKeyboard TrapMissing Focus IndicatorE2e ChecklistE2E Testing ChecklistTest Selection ChecklistLocator Strategy ChecklistTest Implementation ChecklistPage Object ChecklistConfiguration ChecklistCI/CD ChecklistVisual Regression ChecklistAccessibility ChecklistReview ChecklistAnti-Patterns to AvoidE2e Testing ChecklistE2E Testing ChecklistPre-ImplementationTest PlanningEnvironment SetupTest Data StrategyTest ImplementationPage ObjectsTest StructureAssertionsAPI InteractionsSSE/Real-Time FeaturesError HandlingLoading StatesResponsive DesignAccessibilityVisual RegressionCode QualityTest MaintainabilityPerformanceFlakiness PreventionCI/CD IntegrationPipeline ConfigurationEnvironment ManagementMonitoring & ReportingOrchestKit-SpecificAnalysis Flow TestsAgent OrchestrationArtifact DisplayError ScenariosPerformance TestsMaintenanceRegular TasksWhen Tests FailOptimizationDocumentationTest DocumentationKnowledge SharingQuality GatesBefore CommittingBefore Merging PRBefore Production DeployAdvanced TopicsCross-Browser TestingInternationalization (i18n)Security TestingPerformance TestingSuccess MetricsExamples (3)A11y Testing ExamplesAccessibility Testing Examplesjest-axe Component TestsBasic Button TestForm Component TestModal Component TestCustom Dropdown TestPlaywright + axe-core E2E TestsPage-Level TestUser Journey TestDynamic Content TestModal Interaction TestCustom axe RulesCreating a Custom RuleUsing Custom Rules in TestsCI Pipeline ConfigurationGitHub Actions WorkflowPackage.json Test ScriptsE2e Test PatternsE2E Test PatternsComplete User Flow TestPage Object ModelAuthentication FixtureVisual Regression TestAPI Mocking in E2EMulti-Tab TestFile Upload TestAccessibility TestOrchestkit E2e TestsOrchestKit E2E Test ExamplesTest Configurationplaywright.config.tsPage ObjectsHomePage (URL Submission)AnalysisProgressPage (SSE Stream)ArtifactPage (View Results)Test Suites1. Happy Path - Complete Analysis Flow2. SSE Progress Updates3. Error Handling4. Cancellation & Cleanup5. Responsive & Mobile6. AccessibilityRunning TestsCI IntegrationBest Practices