Testing E2e
End-to-end testing patterns with Playwright — page objects, AI agent testing, visual regression, accessibility testing with axe-core, and CI integration. Use when writing E2E tests, setting up Playwright, implementing visual regression, or testing accessibility.
Primary Agent: test-generator
E2E Testing Patterns
End-to-end testing with Playwright 1.58+, visual regression, accessibility, and AI agent workflows.
Quick Reference
| Category | Rules | Impact | When to Use |
|---|---|---|---|
| Playwright Core | rules/e2e-playwright.md | HIGH | Semantic locators, auto-wait, flaky detection |
| Page Objects | rules/e2e-page-objects.md | HIGH | Encapsulate page interactions, visual regression |
| AI Agents | rules/e2e-ai-agents.md | HIGH | Planner/Generator/Healer, init-agents |
| A11y Playwright | rules/a11y-playwright.md | MEDIUM | Full-page axe-core scanning with WCAG 2.2 AA |
| A11y CI/CD | rules/a11y-testing.md | MEDIUM | CI gates, jest-axe unit tests, PR blocking |
| End-to-End Types | rules/validation-end-to-end.md | HIGH | tRPC, Prisma, Pydantic type safety |
Total: 6 rules, 4 references, 3 checklists, 3 examples, 1 script
Playwright Quick Start
import { test, expect } from '@playwright/test';
test('user can complete checkout', async ({ page }) => {
await page.goto('/products');
await page.getByRole('button', { name: 'Add to cart' }).click();
await page.getByRole('link', { name: 'Checkout' }).click();
await page.getByLabel('Email').fill('test@example.com');
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
});Locator Priority: getByRole() > getByLabel() > getByPlaceholder() > getByTestId()
Playwright Core
Semantic locator patterns and best practices for resilient tests.
| Rule | File | Key Pattern |
|---|---|---|
| Playwright E2E | rules/e2e-playwright.md | Semantic locators, auto-wait, new 1.58+ features |
Anti-patterns (FORBIDDEN):
- Hardcoded waits:
await page.waitForTimeout(2000) - CSS selectors for interactions:
await page.click('.submit-btn') - XPath locators
Page Objects
Encapsulate page interactions into reusable classes.
| Rule | File | Key Pattern |
|---|---|---|
| Page Object Model | rules/e2e-page-objects.md | Locators in constructor, action methods, assertion methods |
const checkout = new CheckoutPage(page);
await checkout.fillEmail('test@example.com');
await checkout.submit();
await checkout.expectConfirmation();AI Agents
Playwright 1.58+ AI agent framework for test planning, generation, and self-healing.
| Rule | File | Key Pattern |
|---|---|---|
| AI Agents | rules/e2e-ai-agents.md | Planner, Generator, Healer workflow |
npx playwright init-agents --loop=claude # For Claude CodeWorkflow: Planner (explores app, creates specs) -> Generator (reads spec, tests live app) -> Healer (fixes failures, updates selectors).
Accessibility (Playwright)
Full-page accessibility validation with axe-core in E2E tests.
| Rule | File | Key Pattern |
|---|---|---|
| Playwright + axe | rules/a11y-playwright.md | WCAG 2.2 AA, interactive state testing |
import AxeBuilder from '@axe-core/playwright';
test('page meets WCAG 2.2 AA', async ({ page }) => {
await page.goto('/');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
.analyze();
expect(results.violations).toEqual([]);
});Accessibility (CI/CD)
CI pipeline integration and jest-axe unit-level component testing.
| Rule | File | Key Pattern |
|---|---|---|
| CI Gates + jest-axe | rules/a11y-testing.md | PR blocking, component state testing |
End-to-End Types
Type safety across API layers to eliminate runtime type errors.
| Rule | File | Key Pattern |
|---|---|---|
| Type Safety | rules/validation-end-to-end.md | tRPC, Zod, Pydantic, schema rejection tests |
Visual Regression
Native Playwright screenshot comparison without external services.
await expect(page).toHaveScreenshot('checkout-page.png', {
maxDiffPixels: 100,
mask: [page.locator('.dynamic-content')],
});See references/visual-regression.md for full configuration, CI/CD workflows, cross-platform handling, and Percy migration guide.
Key Decisions
| Decision | Recommendation |
|---|---|
| E2E framework | Playwright 1.58+ with semantic locators |
| Locator strategy | getByRole > getByLabel > getByTestId |
| Browser | Chromium (Chrome for Testing in 1.58+) |
| Page pattern | Page Object Model for complex pages |
| Visual regression | Playwright native toHaveScreenshot() |
| A11y testing | axe-core (E2E) + jest-axe (unit) |
| CI retries | 2-3 in CI, 0 locally |
| Flaky detection | failOnFlakyTests: true in CI |
| AI agents | Planner/Generator/Healer via init-agents |
| Type safety | tRPC for end-to-end, Zod for runtime validation |
References
| Resource | Description |
|---|---|
references/playwright-1.57-api.md | Playwright 1.58+ API: locators, assertions, AI agents, auth, flaky detection |
references/playwright-setup.md | Installation, MCP server, seed tests, agent initialization |
references/visual-regression.md | Screenshot config, CI/CD workflows, cross-platform, Percy migration |
references/a11y-testing-tools.md | jest-axe setup, Playwright axe-core, CI pipelines, manual checklists |
Checklists
| Checklist | Description |
|---|---|
checklists/e2e-checklist.md | Locator strategy, page objects, CI/CD, visual regression |
checklists/e2e-testing-checklist.md | Comprehensive: planning, implementation, SSE, responsive, maintenance |
checklists/a11y-testing-checklist.md | Automated + manual: keyboard, screen reader, color contrast, WCAG |
Examples
| Example | Description |
|---|---|
examples/e2e-test-patterns.md | User flows, page objects, auth fixtures, API mocking, multi-tab, file upload |
examples/a11y-testing-examples.md | jest-axe components, Playwright axe E2E, custom rules, CI pipeline |
examples/orchestkit-e2e-tests.md | OrchestKit analysis flow: page objects, SSE progress, error handling |
Scripts
| Script | Description |
|---|---|
scripts/create-page-object.md | Generate Playwright page object with auto-detected patterns |
Related Skills
testing-unit- Unit testing patterns with mocking, fixtures, and data factoriestest-standards-enforcer- AAA and naming enforcementrun-tests- Test execution orchestration
Rules (6)
Validate full-page accessibility compliance through Playwright E2E tests with axe-core — MEDIUM
Playwright + axe-core E2E
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test('page has no a11y violations', async ({ page }) => {
await page.goto('/');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
.analyze();
expect(results.violations).toEqual([]);
});
test('modal state has no violations', async ({ page }) => {
await page.goto('/');
await page.click('[data-testid="open-modal"]');
await page.waitForSelector('[role="dialog"]');
const results = await new AxeBuilder({ page })
.include('[role="dialog"]')
.withTags(['wcag2a', 'wcag2aa'])
.analyze();
expect(results.violations).toEqual([]);
});Key Decisions
| Decision | Choice | Rationale |
|---|---|---|
| Test runner | Playwright + axe | Full page coverage |
| WCAG level | AA (wcag2aa) | Industry standard |
| State testing | Test all interactive states | Modal, error, loading |
| Browser matrix | Chromium + Firefox | Cross-browser coverage |
Incorrect — Testing page without WCAG tags:
test('page has no violations', async ({ page }) => {
await page.goto('/');
const results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});Correct — Testing with WCAG 2.2 AA compliance:
test('page meets WCAG 2.2 AA', async ({ page }) => {
await page.goto('/');
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag22aa'])
.analyze();
expect(results.violations).toEqual([]);
});Enforce accessibility testing in CI pipelines and enable unit-level component testing with jest-axe — MEDIUM
CI/CD Accessibility Gates
# .github/workflows/accessibility.yml
name: Accessibility
on: [pull_request]
jobs:
a11y:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '20', cache: 'npm' }
- run: npm ci
- run: npm run test:a11y
- run: npm run build
- run: npx playwright install --with-deps chromium
- run: npm start & npx wait-on http://localhost:3000
- run: npx playwright test e2e/accessibilityAnti-Patterns (FORBIDDEN)
// BAD: Excluding too much
new AxeBuilder({ page })
.exclude('body') // Defeats the purpose
.analyze();
// BAD: No CI enforcement
// Accessibility tests exist but don't block PRs
// BAD: Manual-only testing
// Relying solely on human reviewKey Decisions
| Decision | Choice | Rationale |
|---|---|---|
| CI gate | Block on violations | Prevent regression |
| Tags | wcag2a, wcag2aa, wcag22aa | Full WCAG 2.2 AA |
| Exclusions | Third-party widgets only | Minimize blind spots |
Incorrect — Accessibility tests exist but don't enforce in CI:
# .github/workflows/test.yml
- run: npm run test:a11y # Runs but doesn't block on failures
- run: npm run test:unitCorrect — CI blocks PRs on accessibility violations:
# .github/workflows/accessibility.yml
on: [pull_request]
jobs:
a11y:
runs-on: ubuntu-latest
steps:
- run: npm run test:a11y # Exits with code 1 on violations
- run: npx playwright test e2e/accessibility # Blocks mergejest-axe Unit Testing
Setup
// jest.setup.ts
import { toHaveNoViolations } from 'jest-axe';
expect.extend(toHaveNoViolations);Component Testing
import { render } from '@testing-library/react';
import { axe } from 'jest-axe';
it('has no a11y violations', async () => {
const { container } = render(<Button>Click me</Button>);
expect(await axe(container)).toHaveNoViolations();
});Anti-Patterns (FORBIDDEN)
// BAD: Disabling rules globally
const results = await axe(container, {
rules: { 'color-contrast': { enabled: false } } // NEVER disable rules
});
// BAD: Only testing happy path
it('form is accessible', async () => {
const { container } = render(<Form />);
expect(await axe(container)).toHaveNoViolations();
// Missing: error state, loading state, disabled state
});Key Patterns
- Test all component states (default, error, loading, disabled)
- Never disable axe rules globally
- Use for fast feedback in development
Incorrect — Only testing the default state:
it('form is accessible', async () => {
const { container } = render(<LoginForm />);
expect(await axe(container)).toHaveNoViolations();
// Missing: error, loading, disabled states
});Correct — Testing all component states:
it('form is accessible in all states', async () => {
const { container, rerender } = render(<LoginForm />);
expect(await axe(container)).toHaveNoViolations();
rerender(<LoginForm error="Invalid email" />);
expect(await axe(container)).toHaveNoViolations();
rerender(<LoginForm loading={true} />);
expect(await axe(container)).toHaveNoViolations();
});Use Playwright AI agent framework for test planning, generation, and self-healing — HIGH
Playwright AI Agents (1.58+)
Initialize AI Agents
npx playwright init-agents --loop=claude # For Claude Code
npx playwright init-agents --loop=vscode # For VS Code (v1.105+)
npx playwright init-agents --loop=opencode # For OpenCodeGenerated Structure
| Directory/File | Purpose |
|---|---|
.github/ | Agent definitions and configuration |
specs/ | Test plans in Markdown format |
tests/seed.spec.ts | Seed file for AI agents to reference |
Agent Workflow
1. PLANNER --> Explores app --> Creates specs/checkout.md
(uses seed.spec.ts)
2. GENERATOR --> Reads spec --> Tests live app --> Outputs tests/checkout.spec.ts
(verifies selectors actually work)
3. HEALER --> Runs tests --> Fixes failures --> Updates selectors/waits
(self-healing)Key Concepts
- seed.spec.ts is required — Planner executes this to learn environment, auth, UI elements
- Generator validates live — Actually tests app to verify selectors work
- Healer auto-fixes — When UI changes break tests, replays and patches
Setup Requirements
// .mcp.json in project root
{
"mcpServers": {
"playwright": {
"command": "npx",
"args": ["-y", "@playwright/mcp@latest"]
}
}
}Incorrect — No seed file for AI agents to learn from:
// Missing tests/seed.spec.ts
// AI agents have no example to understand app structure
npx playwright init-agents --loop=claudeCorrect — Seed file teaches agents app patterns:
// tests/seed.spec.ts
import { test } from '@playwright/test';
test('example checkout flow', async ({ page }) => {
await page.goto('/');
await page.getByRole('button', { name: 'Add to cart' }).click();
await page.getByRole('link', { name: 'Checkout' }).click();
// Agents learn selectors and patterns from this
});Encapsulate page interactions into reusable page object classes for maintainable E2E tests — HIGH
Page Object Model
Extract page interactions into reusable classes for maintainable E2E tests.
Pattern
// pages/CheckoutPage.ts
import { Page, Locator } from '@playwright/test';
export class CheckoutPage {
readonly page: Page;
readonly emailInput: Locator;
readonly submitButton: Locator;
readonly confirmationHeading: Locator;
constructor(page: Page) {
this.page = page;
this.emailInput = page.getByLabel('Email');
this.submitButton = page.getByRole('button', { name: 'Submit' });
this.confirmationHeading = page.getByRole('heading', { name: 'Order confirmed' });
}
async fillEmail(email: string) {
await this.emailInput.fill(email);
}
async submit() {
await this.submitButton.click();
}
async expectConfirmation() {
await expect(this.confirmationHeading).toBeVisible();
}
}Visual Regression
// Capture and compare visual snapshots
await expect(page).toHaveScreenshot('checkout-page.png', {
maxDiffPixels: 100,
mask: [page.locator('.dynamic-content')],
});Critical User Journeys to Test
- Authentication: Signup, login, password reset
- Core Transaction: Purchase, booking, submission
- Data Operations: Create, update, delete
- User Settings: Profile update, preferences
Incorrect — Duplicating selectors across tests:
test('checkout flow', async ({ page }) => {
await page.getByLabel('Email').fill('test@example.com');
await page.getByRole('button', { name: 'Submit' }).click();
});
test('another checkout test', async ({ page }) => {
await page.getByLabel('Email').fill('user@example.com'); // Duplicated
await page.getByRole('button', { name: 'Submit' }).click(); // Duplicated
});Correct — Page Object encapsulates selectors:
const checkout = new CheckoutPage(page);
await checkout.fillEmail('test@example.com');
await checkout.submit();
await checkout.expectConfirmation();Apply semantic locator patterns and best practices for resilient Playwright E2E tests — HIGH
Playwright E2E Testing (1.58+)
Semantic Locators
// PREFERRED: Role-based locators (most resilient)
await page.getByRole('button', { name: 'Add to cart' }).click();
await page.getByRole('link', { name: 'Checkout' }).click();
// GOOD: Label-based for form controls
await page.getByLabel('Email').fill('test@example.com');
// ACCEPTABLE: Test IDs for stable anchors
await page.getByTestId('checkout-button').click();
// AVOID: CSS selectors and XPath (fragile)Locator Priority: getByRole() > getByLabel() > getByPlaceholder() > getByTestId()
Basic Test
import { test, expect } from '@playwright/test';
test('user can complete checkout', async ({ page }) => {
await page.goto('/products');
await page.getByRole('button', { name: 'Add to cart' }).click();
await page.getByRole('link', { name: 'Checkout' }).click();
await page.getByLabel('Email').fill('test@example.com');
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
});New Features (1.58+)
// Flaky test detection
export default defineConfig({ failOnFlakyTests: true });
// Assert individual class names
await expect(page.locator('.card')).toContainClass('highlighted');
// IndexedDB storage state
await page.context().storageState({ path: 'auth.json', indexedDB: true });Anti-Patterns (FORBIDDEN)
// NEVER use hardcoded waits
await page.waitForTimeout(2000);
// NEVER use CSS selectors for user interactions
await page.click('.submit-btn');
// ALWAYS use semantic locators + auto-wait
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('alert')).toBeVisible();Key Decisions
| Decision | Recommendation |
|---|---|
| Locators | getByRole > getByLabel > getByTestId |
| Browser | Chromium (Chrome for Testing in 1.58+) |
| Execution | 5-30s per test |
| Retries | 2-3 in CI, 0 locally |
Incorrect — Using hardcoded waits and CSS selectors:
await page.click('.submit-button');
await page.waitForTimeout(2000);
await expect(page.locator('.success-message')).toBeVisible();Correct — Semantic locators with auto-wait:
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('alert', { name: /success/i })).toBeVisible();Validate end-to-end type safety across API layers to eliminate runtime type errors — HIGH
End-to-End Type Safety Validation
Incorrect -- type gaps between API layers:
// Manual type definitions that can drift from schema
interface User {
id: string
name: string
// Missing 'email' field that database has
}
// No type connection between client and server
const response = await fetch('/api/users')
const users = await response.json() // type: anyCorrect -- tRPC end-to-end type safety:
import { initTRPC } from '@trpc/server'
import { z } from 'zod'
const t = initTRPC.create()
export const appRouter = t.router({
getUser: t.procedure
.input(z.object({ id: z.string() }))
.query(async ({ input }) => {
return await db.user.findUnique({ where: { id: input.id } })
}),
createUser: t.procedure
.input(z.object({ email: z.string().email(), name: z.string() }))
.mutation(async ({ input }) => {
return await db.user.create({ data: input })
})
})
export type AppRouter = typeof appRouter
// Client gets full type inference from server without code generationCorrect -- Python type safety with Pydantic and NewType:
from typing import NewType
from uuid import UUID
from pydantic import BaseModel, EmailStr
AnalysisID = NewType("AnalysisID", UUID)
ArtifactID = NewType("ArtifactID", UUID)
def delete_analysis(id: AnalysisID) -> None: ...
delete_analysis(artifact_id) # Error with mypy/ty
class CreateUserRequest(BaseModel):
email: EmailStr
name: str = Field(min_length=2, max_length=100)
# Type-safe extraction from untyped dict
result = {"findings": {...}, "confidence_score": 0.85}
findings: dict[str, object] | None = (
cast("dict[str, object]", result.get("findings"))
if isinstance(result.get("findings"), dict) else None
)Testing type safety:
// Test that schema rejects invalid data
describe('UserSchema', () => {
test('rejects invalid email', () => {
const result = UserSchema.safeParse({ email: 'not-email', name: 'Test' })
expect(result.success).toBe(false)
})
test('rejects missing required fields', () => {
const result = UserSchema.safeParse({})
expect(result.success).toBe(false)
expect(result.error.issues).toHaveLength(2)
})
})Key decisions:
- Runtime validation: Zod (best DX, TypeScript inference)
- API layer: tRPC for end-to-end type safety without codegen
- Exhaustive checks: assertNever for compile-time union completeness
- Python: Pydantic v2 + NewType for branded IDs
- Always test validation schemas reject invalid data
References (4)
A11y Testing Tools
Accessibility Testing Tools Reference
Comprehensive guide to automated and manual accessibility testing tools.
jest-axe Configuration
Installation
npm install --save-dev jest-axe @testing-library/react @testing-library/jest-domSetup
// test-utils/axe.ts
import { configureAxe } from 'jest-axe';
export const axe = configureAxe({
rules: {
// Disable rules if needed (use sparingly)
'color-contrast': { enabled: false }, // Only if manual testing covers this
},
reporter: 'v2',
});// vitest.setup.ts or jest.setup.ts
import { toHaveNoViolations } from 'jest-axe';
expect.extend(toHaveNoViolations);Basic Usage
import { render } from '@testing-library/react';
import { axe } from './test-utils/axe';
test('Button has no accessibility violations', async () => {
const { container } = render(<Button>Click me</Button>);
const results = await axe(container);
expect(results).toHaveNoViolations();
});Component-Specific Rules
// Test form with specific WCAG level
test('Form meets WCAG 2.1 Level AA', async () => {
const { container } = render(<ContactForm />);
const results = await axe(container, {
runOnly: {
type: 'tag',
values: ['wcag2a', 'wcag2aa', 'wcag21aa'],
},
});
expect(results).toHaveNoViolations();
});Testing Specific Rules
// Test only keyboard navigation
test('Modal is keyboard accessible', async () => {
const { container } = render(<Modal isOpen />);
const results = await axe(container, {
runOnly: ['keyboard', 'focus-order-semantics'],
});
expect(results).toHaveNoViolations();
});Playwright + axe-core
Installation
npm install --save-dev @axe-core/playwrightSetup
// tests/a11y.setup.ts
import { test as base } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
export const test = base.extend<{ makeAxeBuilder: () => AxeBuilder }>({
makeAxeBuilder: async ({ page }, use) => {
const makeAxeBuilder = () =>
new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])
.exclude('#third-party-widget');
await use(makeAxeBuilder);
},
});
export { expect } from '@playwright/test';E2E Accessibility Test
import { test, expect } from './a11y.setup';
test('homepage is accessible', async ({ page, makeAxeBuilder }) => {
await page.goto('/');
const accessibilityScanResults = await makeAxeBuilder().analyze();
expect(accessibilityScanResults.violations).toEqual([]);
});Testing After Interactions
test('modal maintains accessibility after opening', async ({ page, makeAxeBuilder }) => {
await page.goto('/dashboard');
// Initial state
const initialScan = await makeAxeBuilder().analyze();
expect(initialScan.violations).toEqual([]);
// After opening modal
await page.getByRole('button', { name: 'Open Settings' }).click();
const modalScan = await makeAxeBuilder().analyze();
expect(modalScan.violations).toEqual([]);
// Focus should be trapped in modal
await page.keyboard.press('Tab');
const focusedElement = await page.evaluate(() => document.activeElement?.tagName);
expect(focusedElement).not.toBe('BODY');
});Excluding Regions
test('scan page excluding third-party widgets', async ({ page, makeAxeBuilder }) => {
await page.goto('/');
const results = await makeAxeBuilder()
.exclude('#ads-container')
.exclude('[data-third-party]')
.analyze();
expect(results.violations).toEqual([]);
});CI/CD Integration
GitHub Actions
# .github/workflows/a11y.yml
name: Accessibility Tests
on: [push, pull_request]
jobs:
a11y:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run unit accessibility tests
run: npm run test:a11y
- name: Install Playwright
run: npx playwright install --with-deps chromium
- name: Build application
run: npm run build
- name: Start server
run: npm run start &
env:
PORT: 3000
- name: Wait for server
run: npx wait-on http://localhost:3000
- name: Run E2E accessibility tests
run: npx playwright test tests/a11y/
- name: Upload accessibility report
if: failure()
uses: actions/upload-artifact@v4
with:
name: a11y-report
path: playwright-report/
retention-days: 30Pre-commit Hook
#!/bin/sh
# .husky/pre-commit
# Run accessibility tests on staged components
STAGED_FILES=$(git diff --cached --name-only --diff-filter=ACM | grep "\.tsx\?$")
if [ -n "$STAGED_FILES" ]; then
echo "Running accessibility tests on changed components..."
npm run test:a11y -- --findRelatedTests $STAGED_FILES
if [ $? -ne 0 ]; then
echo "❌ Accessibility tests failed. Please fix violations before committing."
exit 1
fi
fiPackage.json Scripts
{
"scripts": {
"test:a11y": "vitest run tests/**/*.a11y.test.{ts,tsx}",
"test:a11y:watch": "vitest watch tests/**/*.a11y.test.{ts,tsx}",
"test:a11y:e2e": "playwright test tests/a11y/",
"test:a11y:all": "npm run test:a11y && npm run test:a11y:e2e"
}
}Manual Testing Checklist
Use this alongside automated tests for comprehensive coverage.
Keyboard Navigation
-
Tab Order
- Navigate entire page using only Tab/Shift+Tab
- Verify logical focus order
- Ensure all interactive elements are reachable
- Check focus is visible (outline or custom indicator)
-
Interactive Elements
- Enter/Space activates buttons and links
- Arrow keys navigate within widgets (tabs, menus, sliders)
- Escape closes modals and dropdowns
- Home/End navigate to start/end of lists
-
Form Controls
- All form fields reachable via keyboard
- Labels associated with inputs
- Error messages announced and keyboard-accessible
- Submit works via Enter key
Screen Reader Testing
Tools:
- macOS: VoiceOver (Cmd+F5)
- Windows: NVDA (free) or JAWS
- Linux: Orca
Test Scenarios:
- Navigate by headings (H key in screen reader)
- Navigate by landmarks (D key in screen reader)
- Form fields announce label and type
- Buttons announce role and state (expanded/collapsed)
- Dynamic content changes are announced (aria-live)
- Images have meaningful alt text or aria-label
Color Contrast
Tools:
- Browser Extensions: axe DevTools, WAVE
- Design Tools: Figma has built-in contrast checker
- Command Line:
pa11yoraxe-cli
Requirements:
- Normal text: 4.5:1 contrast ratio (WCAG AA)
- Large text (18pt+): 3:1 contrast ratio
- UI components: 3:1 contrast ratio
Responsive and Zoom Testing
-
Browser Zoom
- Test at 200% zoom (WCAG 2.1 requirement)
- Verify no horizontal scrolling
- Content remains readable
- No overlapping elements
-
Mobile Testing
- Touch targets at least 44×44px
- No reliance on hover states
- Swipe gestures have keyboard alternative
- Pinch-to-zoom enabled
Continuous Monitoring
Lighthouse CI
# lighthouserc.js
module.exports = {
ci: {
collect: {
url: ['http://localhost:3000', 'http://localhost:3000/dashboard'],
numberOfRuns: 3,
},
assert: {
preset: 'lighthouse:recommended',
assertions: {
'categories:accessibility': ['error', { minScore: 0.95 }],
'categories:best-practices': ['warn', { minScore: 0.9 }],
},
},
upload: {
target: 'temporary-public-storage',
},
},
};axe-cli for Quick Scans
# Install
npm install -g @axe-core/cli
# Scan a URL
axe http://localhost:3000 --tags wcag2a,wcag2aa
# Save results
axe http://localhost:3000 --save results.json
# Check multiple pages
axe http://localhost:3000 \
http://localhost:3000/dashboard \
http://localhost:3000/profile \
--tags wcag21aaCommon Pitfalls
-
Automated Testing Limitations
- Only catches ~30-40% of issues
- Cannot verify semantic meaning
- Cannot test keyboard navigation fully
- Manual testing is REQUIRED
-
False Sense of Security
- Passing axe tests ≠ fully accessible
- Must combine automated + manual testing
- Screen reader testing is essential
-
Ignoring Dynamic Content
- Test ARIA live regions with actual updates
- Verify focus management after route changes
- Test loading and error states
-
Third-Party Components
- UI libraries may have a11y issues
- Always test integrated components
- Don't assume "accessible by default"
Resources
- WCAG 2.1 Guidelines: https://www.w3.org/WAI/WCAG21/quickref/
- axe Rules: https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md
- WebAIM: https://webaim.org/articles/
- A11y Project Checklist: https://www.a11yproject.com/checklist/
Playwright 1.57 Api
Playwright 1.58+ API Reference
Semantic Locators (2026 Best Practice)
Locator Priority
getByRole()- Matches how users/assistive tech see the pagegetByLabel()- For form inputs with labelsgetByPlaceholder()- For inputs with placeholdersgetByText()- For text contentgetByTestId()- When semantic locators aren't possible
Role-Based Locators
// Buttons
await page.getByRole('button', { name: 'Submit' }).click();
await page.getByRole('button', { name: /submit/i }).click(); // Regex
// Links
await page.getByRole('link', { name: 'Home' }).click();
// Headings
await expect(page.getByRole('heading', { name: 'Welcome' })).toBeVisible();
await expect(page.getByRole('heading', { level: 1 })).toHaveText('Welcome');
// Form controls
await page.getByRole('textbox', { name: 'Email' }).fill('test@example.com');
await page.getByRole('checkbox', { name: 'Remember me' }).check();
await page.getByRole('combobox', { name: 'Country' }).selectOption('US');
// Lists
await expect(page.getByRole('list')).toContainText('Item 1');
await expect(page.getByRole('listitem')).toHaveCount(3);
// Navigation
await page.getByRole('navigation').getByRole('link', { name: 'About' }).click();Label-Based Locators
// Form inputs with labels
await page.getByLabel('Email').fill('test@example.com');
await page.getByLabel('Password').fill('secret123');
await page.getByLabel('Remember me').check();
// Partial match
await page.getByLabel(/email/i).fill('test@example.com');Text and Placeholder
// Text content
await page.getByText('Welcome back').click();
await page.getByText(/welcome/i).isVisible();
// Placeholder
await page.getByPlaceholder('Enter email').fill('test@example.com');Test IDs (Fallback)
// When semantic locators aren't possible
await page.getByTestId('custom-widget').click();
// Configure test ID attribute
// playwright.config.ts
export default defineConfig({
use: {
testIdAttribute: 'data-test-id',
},
});Breaking Changes (1.58)
Removed Features
| Feature | Status | Migration |
|---|---|---|
_react selector | Removed | Use getByRole() or getByTestId() |
_vue selector | Removed | Use getByRole() or getByTestId() |
:light selector suffix | Removed | Use standard CSS selectors |
devtools launch option | Removed | Use args: ['--auto-open-devtools-for-tabs'] |
| macOS 13 WebKit | Removed | Upgrade to macOS 14+ |
Migration Examples
// React/Vue component selectors - Before
await page.locator('_react=MyComponent').click();
await page.locator('_vue=MyComponent').click();
// After - Use semantic locators or test IDs
await page.getByRole('button', { name: 'My Component' }).click();
await page.getByTestId('my-component').click();
// :light selector - Before
await page.locator('.card:light').click();
// After - Just use the selector directly
await page.locator('.card').click();
// DevTools option - Before
const browser = await chromium.launch({ devtools: true });
// After - Use args
const browser = await chromium.launch({
args: ['--auto-open-devtools-for-tabs']
});New Features (1.58+)
connectOverCDP with isLocal
// Optimized CDP connection for local debugging
const browser = await chromium.connectOverCDP({
endpointURL: 'http://localhost:9222',
isLocal: true // NEW: Optimizes for local connections
});
// Use for connecting to locally running Chrome instances
// Reduces latency and improves reliabilityTimeline in Speedboard HTML Reports
HTML reports now include an interactive timeline:
// playwright.config.ts
export default defineConfig({
reporter: [['html', { open: 'never' }]],
});
// The HTML report shows:
// - Test execution sequence
// - Parallel test distribution
// - Time spent in each test phase
// - Performance bottlenecksNew Assertions (1.57+)
// Assert individual class names (1.57+)
await expect(page.locator('.card')).toContainClass('highlighted');
await expect(page.locator('.card')).toContainClass(['active', 'visible']);
// Visibility
await expect(page.getByRole('button')).toBeVisible();
await expect(page.getByRole('button')).toBeHidden();
await expect(page.getByRole('button')).toBeEnabled();
await expect(page.getByRole('button')).toBeDisabled();
// Text content
await expect(page.getByRole('heading')).toHaveText('Welcome');
await expect(page.getByRole('heading')).toContainText('Welcome');
// Attribute
await expect(page.getByRole('link')).toHaveAttribute('href', '/home');
// Count
await expect(page.getByRole('listitem')).toHaveCount(5);
// Screenshot
await expect(page).toHaveScreenshot('page.png');
await expect(page.locator('.hero')).toHaveScreenshot('hero.png');AI Agents (1.58+)
Initialize AI Agents
# Initialize agents for your preferred AI tool
npx playwright init-agents --loop=claude # For Claude Code
npx playwright init-agents --loop=vscode # For VS Code (requires v1.105+)
npx playwright init-agents --loop=opencode # For OpenCodeGenerated Structure
| Directory/File | Purpose |
|---|---|
.github/ | Agent definitions and configuration |
specs/ | Test plans in Markdown format |
tests/seed.spec.ts | Seed file for AI agents to reference |
Configuration
// playwright.config.ts
export default defineConfig({
use: {
aiAgents: {
enabled: true,
model: 'claude-sonnet-4-6', // or local Ollama
autoHeal: true, // Auto-repair on CI failures
}
}
});Authentication State
Storage State
// Save auth state
await page.context().storageState({ path: 'playwright/.auth/user.json' });
// Use saved state
const context = await browser.newContext({
storageState: 'playwright/.auth/user.json'
});IndexedDB Support (1.57+)
// Save storage state including IndexedDB
await page.context().storageState({
path: 'auth.json',
indexedDB: true // Include IndexedDB in storage state
});
// Restore with IndexedDB
const context = await browser.newContext({
storageState: 'auth.json' // Includes IndexedDB automatically
});Auth Setup Project
// playwright.config.ts
export default defineConfig({
projects: [
{
name: 'setup',
testMatch: /.*\.setup\.ts/,
},
{
name: 'logged-in',
dependencies: ['setup'],
use: {
storageState: 'playwright/.auth/user.json',
},
},
],
});Flaky Test Detection (1.57+)
// playwright.config.ts
export default defineConfig({
// Fail CI if any flaky tests detected
failOnFlakyTests: true,
// Retry configuration
retries: process.env.CI ? 2 : 0,
// Web server with regex-based ready detection
webServer: {
command: 'npm run dev',
wait: /ready in \d+ms/, // Wait for this log pattern
},
});Visual Regression
test('visual regression', async ({ page }) => {
await page.goto('/');
// Full page screenshot
await expect(page).toHaveScreenshot('homepage.png');
// Element screenshot
await expect(page.locator('.hero')).toHaveScreenshot('hero.png');
// With options
await expect(page).toHaveScreenshot('page.png', {
maxDiffPixels: 100,
threshold: 0.2,
});
});Locator Descriptions (1.57+)
// Describe locators for trace viewer
const submitBtn = page.getByRole('button', { name: 'Submit' });
submitBtn.describe('Main form submit button');
// Shows in trace viewer for debuggingChrome for Testing (1.57+)
Playwright uses Chrome for Testing builds instead of Chromium:
# Install browsers (includes Chrome for Testing)
npx playwright install
# No code changes needed - better Chrome compatibilityExternal Links
Playwright Setup
Playwright Setup with Test Agents
Install and configure Playwright with autonomous test agents for Claude Code.
Prerequisites
Required: VS Code v1.105+ (released Oct 9, 2025) for agent functionality
Step 1: Install Playwright
npm install --save-dev @playwright/test
npx playwright install # Install browsers (Chromium, Firefox, WebKit)Step 2: Add Playwright MCP Server (CC 2.1.6)
Create or update .mcp.json in your project root:
{
"mcpServers": {
"playwright": {
"command": "npx",
"args": ["-y", "@playwright/mcp@latest"]
}
}
}Restart your Claude Code session to pick up the MCP configuration.
Note: The
claude mcp addcommand is deprecated in CC 2.1.6. Configure MCPs directly via.mcp.json.
Step 3: Initialize Test Agents
# Initialize the three agents (planner, generator, healer)
npx playwright init-agents --loop=claude
# OR for VS Code: --loop=vscode
# OR for OpenCode: --loop=opencodeWhat this does:
- Creates agent definition files in your project
- Agents are Markdown-based instruction files
- Regenerate when Playwright updates to get latest tools
Step 4: Create Seed Test
Create tests/seed.spec.ts - the planner uses this to understand your setup:
// tests/seed.spec.ts
import { test, expect } from '@playwright/test';
test.beforeEach(async ({ page }) => {
// Your app initialization
await page.goto('http://localhost:3000');
// Login if needed
// await page.getByLabel('Email').fill('test@example.com');
// await page.getByLabel('Password').fill('password123');
// await page.getByRole('button', { name: 'Login' }).click();
});
test('seed test - app is accessible', async ({ page }) => {
await expect(page).toHaveTitle(/MyApp/);
await expect(page.getByRole('navigation')).toBeVisible();
});Why seed.spec.ts?
- Planner executes this to learn:
- Environment setup (fixtures, hooks)
- Authentication flow
- App initialization
- Available selectors
Directory Structure
your-project/
├── specs/ <- Planner outputs test plans here (Markdown)
├── tests/ <- Generator outputs test code here (.spec.ts)
│ └── seed.spec.ts <- Your initialization test (REQUIRED)
├── playwright.config.ts
└── .mcp.json <- MCP server configBasic Configuration
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests',
fullyParallel: true,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
use: {
baseURL: 'http://localhost:3000',
trace: 'on-first-retry',
screenshot: 'only-on-failure',
},
projects: [
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
],
});Running Tests
npx playwright test # Run all tests
npx playwright test --ui # UI mode
npx playwright test --debug # Debug mode
npx playwright test --headed # See browserBrowser Automation
For quick browser automation outside of Playwright tests, use agent-browser CLI:
# Quick visual verification
agent-browser open http://localhost:5173
agent-browser snapshot -i
agent-browser screenshot /tmp/screenshot.png
agent-browser closeRun agent-browser --help for full CLI docs.
Next Steps
- Planner: "Generate test plan for checkout flow" -> creates
specs/checkout.md - Generator: "Generate tests from checkout spec" -> creates
tests/checkout.spec.ts - Healer: Automatically fixes tests when selectors break
See references/planner-agent.md for detailed workflow.
Visual Regression
Playwright Native Visual Regression Testing
Updated Dec 2025 - Best practices for
toHaveScreenshot()without external services like Percy or Chromatic.
Overview
Playwright's built-in visual regression testing uses expect(page).toHaveScreenshot() to capture and compare screenshots. This is completely free, requires no signup, and works in CI without external dependencies.
Quick Start
import { test, expect } from '@playwright/test';
test('homepage visual regression', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('homepage.png');
});On first run, Playwright creates a baseline screenshot. Subsequent runs compare against it.
Configuration (playwright.config.ts)
Essential Settings
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
// Snapshot configuration
snapshotPathTemplate: '{testDir}/__screenshots__/{testFilePath}/{arg}{ext}',
updateSnapshots: 'missing', // 'all' | 'changed' | 'missing' | 'none'
expect: {
toHaveScreenshot: {
// Tolerance settings
maxDiffPixelRatio: 0.01, // Allow 1% pixel difference
threshold: 0.2, // Per-pixel color threshold (0-1)
// Animation handling
animations: 'disabled', // Freeze CSS animations
// Caret handling (text cursors)
caret: 'hide',
},
},
// CI-specific settings
workers: process.env.CI ? 1 : undefined,
retries: process.env.CI ? 2 : 0,
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
// Only run screenshots on Chromium for consistency
{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
ignoreSnapshots: true, // Skip VRT for Firefox
},
],
});Snapshot Path Template Tokens
| Token | Description | Example |
|---|---|---|
\{testDir\} | Test directory | e2e |
\{testFilePath\} | Test file relative path | specs/visual.spec.ts |
\{testFileName\} | Test file name | visual.spec.ts |
\{arg\} | Screenshot name argument | homepage |
\{ext\} | File extension | .png |
\{projectName\} | Project name | chromium |
Test Patterns
Basic Screenshot
test('page screenshot', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('page-name.png');
});Full Page Screenshot
test('full page screenshot', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('full-page.png', {
fullPage: true,
});
});Element Screenshot
test('component screenshot', async ({ page }) => {
await page.goto('/');
const header = page.locator('header');
await expect(header).toHaveScreenshot('header.png');
});Masking Dynamic Content
test('page with masked dynamic content', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('page.png', {
mask: [
page.locator('[data-testid="timestamp"]'),
page.locator('[data-testid="random-avatar"]'),
page.locator('time'),
],
maskColor: '#FF00FF', // Pink mask (default)
});
});Custom Styles for Screenshots
// e2e/fixtures/screenshot.css
// Hide dynamic elements during screenshots
[data-testid="timestamp"],
[data-testid="loading-spinner"] {
visibility: hidden !important;
}
* {
animation: none !important;
transition: none !important;
}test('page with custom styles', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('styled.png', {
stylePath: './e2e/fixtures/screenshot.css',
});
});Responsive Viewports
const viewports = [
{ name: 'mobile', width: 375, height: 667 },
{ name: 'tablet', width: 768, height: 1024 },
{ name: 'desktop', width: 1280, height: 800 },
];
for (const viewport of viewports) {
test(`homepage - ${viewport.name}`, async ({ page }) => {
await page.setViewportSize({
width: viewport.width,
height: viewport.height
});
await page.goto('/');
await expect(page).toHaveScreenshot(`homepage-${viewport.name}.png`);
});
}Dark Mode Testing
test('homepage dark mode', async ({ page }) => {
await page.goto('/');
// Toggle dark mode
await page.evaluate(() => {
document.documentElement.classList.add('dark');
localStorage.setItem('theme', 'dark');
});
// Wait for theme to apply
await page.waitForTimeout(100);
await expect(page).toHaveScreenshot('homepage-dark.png');
});Waiting for Stability
test('page after animations complete', async ({ page }) => {
await page.goto('/');
// Wait for network idle
await page.waitForLoadState('networkidle');
// Wait for specific content
await page.waitForSelector('[data-testid="content-loaded"]');
// Playwright auto-waits for 2 consecutive stable screenshots
await expect(page).toHaveScreenshot('stable.png');
});CI/CD Integration
GitHub Actions Workflow
name: Visual Regression Tests
on:
pull_request:
branches: [main, dev]
jobs:
visual-regression:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Playwright browsers
run: npx playwright install chromium --with-deps
- name: Run visual regression tests
run: npx playwright test --project=chromium e2e/specs/visual-regression.spec.ts
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: playwright-report
path: playwright-report/
retention-days: 7
- name: Upload screenshots on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: screenshot-diffs
path: e2e/__screenshots__/
retention-days: 7Handling Baseline Updates
# Separate workflow for updating baselines
name: Update Visual Baselines
on:
workflow_dispatch: # Manual trigger only
jobs:
update-baselines:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup and install
run: |
npm ci
npx playwright install chromium --with-deps
- name: Update snapshots
run: npx playwright test --update-snapshots
- name: Commit updated snapshots
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add e2e/__screenshots__/
git commit -m "chore: update visual regression baselines" || exit 0
git pushHandling Cross-Platform Issues
The Problem
Screenshots differ between macOS (local) and Linux (CI) due to:
- Font rendering differences
- Anti-aliasing variations
- Subpixel rendering
Solutions
Option 1: Generate baselines only in CI (Recommended)
// playwright.config.ts
export default defineConfig({
// Only update snapshots in CI
updateSnapshots: process.env.CI ? 'missing' : 'none',
});Option 2: Use Docker for local development
# Run tests in same container as CI
docker run --rm -v $(pwd):/work -w /work mcr.microsoft.com/playwright:v1.58.0-jammy \
npx playwright test --project=chromiumOption 3: Increase threshold tolerance
expect: {
toHaveScreenshot: {
maxDiffPixelRatio: 0.05, // 5% tolerance
threshold: 0.3, // Higher per-pixel tolerance
},
},Debugging Failed Screenshots
View Diff Report
npx playwright show-reportGenerated Files on Failure
e2e/__screenshots__/
├── homepage.png # Expected (baseline)
├── homepage-actual.png # Actual (current run)
└── homepage-diff.png # Difference highlightedTrace Viewer for Context
// playwright.config.ts
export default defineConfig({
use: {
trace: 'on-first-retry', // Capture trace on failures
},
});Best Practices
1. Stable Selectors
// Good - semantic selectors
await page.waitForSelector('[data-testid="content"]');
// Avoid - fragile selectors
await page.waitForSelector('.css-1234xyz');2. Wait for Stability
// Ensure page is ready before screenshot
await page.waitForLoadState('networkidle');
await page.waitForSelector('[data-loaded="true"]');3. Mask Dynamic Content
// Always mask timestamps, avatars, random content
mask: [
page.locator('time'),
page.locator('[data-testid="avatar"]'),
],4. Disable Animations
// Global in config
animations: 'disabled',
// Or per-test with CSS
stylePath: './e2e/fixtures/no-animations.css',5. Single Browser for VRT
// Only Chromium for visual tests - most consistent
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
],6. Meaningful Names
// Good - descriptive names
await expect(page).toHaveScreenshot('checkout-payment-form-error.png');
// Avoid - generic names
await expect(page).toHaveScreenshot('test1.png');Migration from Percy
| Percy | Playwright Native |
|---|---|
percySnapshot(page, 'name') | await expect(page).toHaveScreenshot('name.png') |
.percy.yml | playwright.config.ts expect settings |
PERCY_TOKEN | Not needed |
| Cloud dashboard | Local HTML report |
percy exec -- | Direct npx playwright test |
Quick Migration Script
// Before (Percy)
import { percySnapshot } from '@percy/playwright';
await percySnapshot(page, 'Homepage - Light Mode');
// After (Playwright)
// No import needed
await expect(page).toHaveScreenshot('homepage-light.png');Troubleshooting
Flaky Screenshots
Symptoms: Different results on each run
Solutions:
- Increase
maxDiffPixelRatiotolerance - Add explicit waits for dynamic content
- Mask loading spinners and animations
- Use
animations: 'disabled'
CI vs Local Differences
Symptoms: Tests pass locally, fail in CI
Solutions:
- Generate baselines only in CI
- Use Docker locally for consistency
- Increase threshold for font rendering
Large Screenshot Files
Symptoms: Git repository bloat
Solutions:
- Use
.gitattributesfor LFS - Compress with
qualityoption (JPEG only) - Limit screenshot dimensions
# .gitattributes
e2e/__screenshots__/**/*.png filter=lfs diff=lfs merge=lfs -textChecklists (3)
A11y Testing Checklist
Accessibility Testing Checklist
Use this checklist to ensure comprehensive accessibility coverage.
Automated Test Coverage
Unit Tests (jest-axe)
- All form components tested with axe
- All interactive components (buttons, links, modals) tested
- Custom UI widgets tested (date pickers, dropdowns, sliders)
- Dynamic content updates tested
- Error states tested for proper announcements
- Loading states have appropriate ARIA attributes
- Tests cover WCAG 2.1 Level AA tags minimum
- No disabled rules without documented justification
E2E Tests (Playwright + axe-core)
- Homepage scanned for violations
- All critical user journeys include a11y scan
- Post-interaction states scanned (after form submit, modal open)
- Multi-step flows tested (signup, checkout, settings)
- Error pages and 404s tested
- Third-party widgets excluded from scan if necessary
- Tests run in CI/CD pipeline
- Accessibility reports archived on failure
CI/CD Integration
- Accessibility tests run on every PR
- Pre-commit hook runs a11y tests on changed files
- Lighthouse CI monitors accessibility score (>95%)
- Failed tests block deployment
- Test results published to team (GitHub comments, Slack)
Manual Testing Requirements
Keyboard Navigation
-
Tab Navigation
- All interactive elements reachable via Tab/Shift+Tab
- Tab order follows visual layout (top to bottom, left to right)
- Focus indicator visible on all focusable elements
- No keyboard traps (can always Tab away)
-
Action Keys
- Enter/Space activates buttons and links
- Escape closes modals, dropdowns, menus
- Arrow keys navigate within compound widgets (tabs, menus, sliders)
- Home/End keys navigate to start/end where appropriate
-
Form Controls
- All form fields accessible via keyboard
- Enter submits forms
- Error messages keyboard-navigable
- Custom controls (date pickers, color pickers) keyboard-operable
-
Skip Links
- "Skip to main content" link present and functional
- Appears on first Tab press
- Actually skips navigation when activated
Screen Reader Testing
Test with at least one screen reader:
- macOS: VoiceOver (Cmd+F5)
- Windows: NVDA (free) or JAWS
- Linux: Orca
Content Structure
-
Headings
- Logical heading hierarchy (h1 → h2 → h3, no skips)
- Page has exactly one h1
- Headings describe section content
- Can navigate by heading (H key in screen reader)
-
Landmarks
-
<header>,<nav>,<main>,<footer>present - Multiple landmarks of same type have unique labels
- Can navigate by landmark (D key in screen reader)
-
-
Lists
- Navigation uses
<ul>or<nav> - Related items grouped in lists
- Screen reader announces list with item count
- Navigation uses
Interactive Elements
-
Forms
- All inputs have associated
<label>oraria-label - Required fields announced as required
- Error messages announced when they appear
- Field types announced (email, password, number)
- Placeholder text not used as only label
- All inputs have associated
-
Buttons and Links
- Role announced ("button", "link")
- Purpose clear from label alone
- State announced (expanded/collapsed, selected)
- Icon-only buttons have
aria-label
-
Images
- Informative images have meaningful
alttext - Decorative images have
alt=""orrole="presentation" - Complex images have longer description (
aria-describedbyor caption)
- Informative images have meaningful
-
Dynamic Content
- Live regions announce updates (
aria-live="polite"or"assertive") - Loading states announced
- Success/error messages announced
- Content changes don't lose focus position
- Live regions announce updates (
Navigation
-
Menus
- Menu buttons announce expanded/collapsed state
- Arrow keys navigate menu items
- First/last items wrap or stop appropriately
- Escape closes menu
-
Modals/Dialogs
- Focus moves to modal on open
- Focus trapped within modal
- Modal title announced
- Escape closes modal
- Focus returns to trigger on close
-
Tabs
- Tab role announced
- Active tab announced as selected
- Arrow keys navigate tabs
- Tab panel content announced
Color and Contrast
Use browser extensions (axe DevTools, WAVE) or online tools:
-
Text Contrast
- Normal text (< 18pt): 4.5:1 minimum ratio
- Large text (≥ 18pt or 14pt bold): 3:1 minimum ratio
- Passes for all text (body, headings, labels, placeholders)
-
UI Component Contrast
- Buttons, inputs, icons: 3:1 minimum against background
- Focus indicators: 3:1 minimum
- Error/success states: 3:1 minimum
-
Color Independence
- Information not conveyed by color alone
- Links distinguishable without color (underline, icon, etc.)
- Form errors indicated by icon + text, not just red border
- Charts/graphs have patterns or labels, not just colors
Responsive and Zoom Testing
-
Browser Zoom (200%)
- Test at 200% zoom level (WCAG 2.1 requirement)
- No horizontal scrolling at 200% zoom
- All content visible and readable
- No overlapping or cut-off text
- Interactive elements remain operable
-
Mobile/Touch
- Touch targets ≥ 44×44 CSS pixels
- Sufficient spacing between interactive elements (at least 8px)
- No reliance on hover (all hover info accessible on tap)
- Pinch-to-zoom enabled (no
user-scalable=no) - Orientation works in both portrait and landscape
Animation and Motion
-
Respect Motion Preferences
- Check
prefers-reduced-motionmedia query - Disable or reduce animations when preferred
- Test with system setting enabled (macOS, Windows)
- Check
-
No Seizure Triggers
- No flashing content faster than 3 times per second
- Autoplay videos have controls (pause/stop)
- Parallax effects can be disabled
Documentation Review
-
ARIA Usage
- ARIA only used when native HTML insufficient
- ARIA roles match HTML semantics
- All required ARIA properties present
- No conflicting or redundant ARIA
-
Code Comments
- Complex accessibility patterns documented
- Keyboard shortcuts documented
- Focus management documented
Cross-Browser Testing
Test in multiple browsers and assistive tech combinations:
- Chrome + NVDA (Windows)
- Firefox + NVDA (Windows)
- Safari + VoiceOver (macOS)
- Safari + VoiceOver (iOS)
- Chrome + TalkBack (Android)
Compliance Verification
-
WCAG 2.1 Level AA
- Automated tests pass for wcag2a, wcag2aa, wcag21aa tags
- Manual testing confirms keyboard accessibility
- Manual testing confirms screen reader accessibility
- Color contrast verified
-
Legal Requirements
- Section 508 (US federal)
- ADA (US)
- EN 301 549 (EU)
- Accessibility statement page present (if required)
Continuous Monitoring
- Lighthouse accessibility score tracked over time
- Accessibility tests in regression suite
- New features include a11y tests from day one
- Team trained on accessibility best practices
- Accessibility champion assigned
- Regular audits scheduled (quarterly recommended)
When to Seek Expert Help
Engage an accessibility specialist if:
- Building complex custom widgets (ARIA patterns)
- Handling advanced screen reader interactions
- Preparing for legal compliance audit
- User feedback indicates accessibility issues
- Automated tests show many violations
- Team lacks accessibility expertise
Quick Wins for Common Issues
Missing Alt Text
<!-- Before -->
<img src="logo.png">
<!-- After -->
<img src="logo.png" alt="Company Logo">Unlabeled Form Input
<!-- Before -->
<input type="email" placeholder="Email">
<!-- After -->
<label for="email">Email</label>
<input type="email" id="email">Low Contrast Text
/* Before */
color: #999; /* 2.8:1 ratio */
/* After */
color: #767676; /* 4.5:1 ratio */Keyboard Trap
// Before
<div onClick={handleClick}>Click me</div>
// After
<button onClick={handleClick}>Click me</button>Missing Focus Indicator
/* Before */
button:focus { outline: none; }
/* After */
button:focus-visible {
outline: 2px solid blue;
outline-offset: 2px;
}E2e Checklist
E2E Testing Checklist
Test Selection Checklist
Focus E2E tests on business-critical paths:
- Authentication: Signup, login, password reset, logout
- Core Transaction: Purchase, booking, submission, payment
- Data Operations: Create, update, delete critical entities
- User Settings: Profile update, preferences, notifications
- Error Recovery: Form validation, API errors, network issues
Locator Strategy Checklist
- Use
getByRole()as primary locator strategy - Use
getByLabel()for form inputs - Use
getByPlaceholder()when no label available - Use
getByTestId()only as last resort - AVOID CSS selectors for user interactions
- AVOID XPath locators
- AVOID
page.click('[data-testid=...]')- usegetByTestIdinstead
Test Implementation Checklist
For each test:
- Clear, descriptive test name
- Tests one user flow or scenario
- Uses semantic locators (getByRole, getByLabel)
- Waits for elements using Playwright's auto-wait
- No hardcoded
sleep()orwait()calls - Assertions use
expect()with appropriate matchers - Test can run in isolation (no dependencies on other tests)
Page Object Checklist
For each page object:
- Locators defined in constructor
- Methods for user actions (login, submit, navigate)
- Assertion methods (expectError, expectSuccess)
- No direct
page.click()calls - wrap in methods - TypeScript types for all methods
Configuration Checklist
- Set
baseURLin config - Configure browser(s) for testing
- Set up authentication state project
- Configure retries for CI (2-3 retries)
- Enable
failOnFlakyTestsin CI - Set appropriate timeouts
- Configure screenshot on failure
CI/CD Checklist
- Tests run in CI pipeline
- Artifacts (screenshots, traces) uploaded on failure
- Tests parallelized with sharding
- Auth state cached between runs
- Web server waits for ready signal
Visual Regression Checklist
- Screenshots stored in version control
- Different screenshots per browser/platform
- Mobile viewports tested
- Dark mode tested (if applicable)
- Threshold set for acceptable diff
Accessibility Checklist
- axe-core integrated for a11y testing
- Critical pages tested for violations
- Forms have proper labels
- Focus management tested
- Keyboard navigation tested
Review Checklist
Before PR:
- All tests pass locally
- Tests are deterministic (no flakes)
- Locators follow semantic strategy
- No hardcoded waits
- Test files organized logically
- Page objects used for complex pages
- CI configuration updated if needed
Anti-Patterns to Avoid
- Too many E2E tests (keep it focused)
- Testing non-critical paths
- Hard-coded waits (
await page.waitForTimeout()) - CSS/XPath selectors for interactions
- Tests that depend on each other
- Tests that modify global state
- Ignoring flaky test warnings
E2e Testing Checklist
E2E Testing Checklist
Comprehensive checklist for planning, implementing, and maintaining E2E tests with Playwright.
Pre-Implementation
Test Planning
- Identify critical user journeys to test
- Map out happy paths and error scenarios
- Determine test data requirements
- Decide on mocking strategy (API, SSE, external services)
- Plan for visual regression testing needs
- Identify accessibility requirements (WCAG 2.1 AA)
- Estimate test execution time and CI impact
Environment Setup
- Install Playwright (
npm install -D @playwright/test) - Install browser binaries (
npx playwright install) - Create
playwright.config.tswith base URL and timeouts - Configure test directory structure (
tests/e2e/) - Set up Page Object pattern structure
- Configure CI environment (GitHub Actions, GitLab CI, etc.)
- Set up test database/backend for integration tests
Test Data Strategy
- Create fixtures for common test scenarios
- Set up database seeding scripts
- Plan API mocking approach (mock server vs route interception)
- Create reusable test data generators
- Handle authentication/authorization test cases
- Plan for cleanup between tests
Test Implementation
Page Objects
- Create base page class with common utilities
- Implement page object for each major page/component
- Use semantic locators (role, label, test-id)
- Avoid brittle CSS/XPath selectors
- Encapsulate complex interactions in helper methods
- Add TypeScript types for type safety
- Document page object APIs
Test Structure
- Follow Arrange-Act-Assert (AAA) pattern
- Use descriptive test names (should/when/given format)
- Group related tests with
test.describe() - Set up common state in
beforeEach() - Clean up resources in
afterEach() - Use test fixtures for shared setup
- Keep tests independent (no test interdependencies)
Assertions
- Use specific assertions (
toHaveTextvstoBeTruthy) - Assert on user-visible behavior, not implementation
- Verify loading states appear and disappear
- Check error messages and validation feedback
- Validate success states and confirmations
- Test navigation and URL changes
- Verify data persistence across page loads
API Interactions
- Mock external API calls for reliability
- Test real API endpoints in integration tests
- Handle async operations properly (promises, awaits)
- Test timeout scenarios
- Verify retry logic
- Test rate limiting behavior
- Mock SSE/WebSocket streams
SSE/Real-Time Features
- Test SSE connection establishment
- Verify progress updates stream correctly
- Test reconnection on connection drop
- Handle SSE error events
- Test SSE completion and cleanup
- Verify UI updates from SSE events
- Test SSE with network throttling
Error Handling
- Test form validation errors
- Test API error responses (400, 500, etc.)
- Test network failures
- Test timeout scenarios
- Verify error messages shown to user
- Test retry/recovery mechanisms
- Test graceful degradation
Loading States
- Test loading spinners appear
- Verify skeleton screens render
- Test loading state timeouts
- Check loading states disappear on completion
- Test loading state cancellation
- Verify loading indicators are accessible
Responsive Design
- Test on desktop viewports (1920x1080, 1366x768)
- Test on tablet viewports (768x1024, 1024x768)
- Test on mobile viewports (375x667, 414x896)
- Verify touch interactions on mobile
- Test responsive navigation menus
- Verify content reflow on viewport changes
- Test orientation changes (portrait/landscape)
Accessibility
- Test keyboard navigation (Tab, Enter, Escape, arrows)
- Verify focus management (focus visible, focus traps)
- Test screen reader announcements (aria-live, role=status)
- Check ARIA labels and descriptions
- Test color contrast (use automated tools)
- Verify form labels and error associations
- Test with browser accessibility extensions
- Consider adding axe-core integration
Visual Regression
- Identify components/pages for screenshot testing
- Set up baseline screenshots
- Configure pixel diff thresholds
- Test responsive breakpoints visually
- Test theme variations (light/dark mode)
- Test different locales (i18n)
- Update baselines when designs change
Code Quality
Test Maintainability
- Avoid test duplication (use helpers, fixtures)
- Use constants for magic strings/numbers
- Keep tests readable (avoid over-abstraction)
- Add comments for complex test logic
- Refactor brittle tests
- Remove flaky tests or fix root cause
- Review test coverage regularly
Performance
- Run tests in parallel where possible
- Minimize test execution time (mock slow APIs)
- Use
test.describe.configure(\{ mode: 'parallel' \}) - Avoid unnecessary waits (
waitForTimeout) - Use strategic waits (
waitForSelector,waitForLoadState) - Optimize page load times (disable unnecessary assets)
- Profile slow tests and optimize
Flakiness Prevention
- Use deterministic waits (waitFor* methods)
- Avoid race conditions (wait for element visibility)
- Handle timing issues (debounce, throttle)
- Retry flaky tests in CI (max 2 retries)
- Investigate and fix root cause of flakiness
- Use
test.slow()for long-running tests - Increase timeouts for legitimate slow operations
CI/CD Integration
Pipeline Configuration
- Add E2E test job to CI pipeline
- Run tests on every PR
- Block merge on test failures
- Run tests against staging environment
- Configure test parallelization in CI
- Set up test result reporting
- Archive test artifacts (videos, screenshots, traces)
Environment Management
- Use Docker Compose for backend services
- Seed test database before test run
- Run migrations before tests
- Clean up test data after run
- Use environment variables for config
- Isolate test environments (per PR if possible)
- Monitor test environment health
Monitoring & Reporting
- Generate HTML test reports
- Upload test artifacts to CI
- Send notifications on test failures
- Track test execution time trends
- Monitor test flakiness rates
- Set up dashboard for test metrics
- Alert on sustained test failures
OrchestKit-Specific
Analysis Flow Tests
- Test URL submission with validation
- Test analysis progress SSE stream
- Verify agent status updates (8 agents)
- Test progress bar updates (0% to 100%)
- Test analysis completion detection
- Test artifact generation
- Test navigation to artifact view
Agent Orchestration
- Verify supervisor assigns tasks
- Test worker agent execution
- Verify quality gate checks
- Test agent failure handling
- Test partial completion scenarios
- Verify agent status badges
Artifact Display
- Test artifact metadata display
- Verify quality scores shown
- Test findings/recommendations rendering
- Test artifact search functionality
- Test section navigation (tabs)
- Test download artifact feature
- Test share/copy link feature
Error Scenarios
- Test invalid URL submission
- Test network timeout during analysis
- Test SSE connection drop
- Test analysis cancellation
- Test concurrent analysis limit
- Test backend service unavailable
- Test rate limiting
Performance Tests
- Test with large artifact (many findings)
- Test SSE with high event frequency
- Test concurrent analyses (multiple tabs)
- Test long-running analysis (timeout)
- Monitor memory leaks during SSE stream
Maintenance
Regular Tasks
- Review and update tests after feature changes
- Update page objects when UI changes
- Update test data when backend schema changes
- Refactor duplicate test code
- Remove obsolete tests
- Update dependencies (Playwright, browsers)
- Review test coverage and add missing tests
When Tests Fail
- Check if failure is legitimate regression
- Review CI logs and screenshots
- Download and analyze trace files
- Reproduce locally with
--debugflag - Fix root cause (not just update assertions)
- Add regression test if bug found
- Update documentation if expected behavior changed
Optimization
- Profile slow tests and optimize
- Reduce unnecessary API calls
- Optimize page object selectors
- Minimize test data setup
- Use test fixtures for common scenarios
- Run critical tests first (fail fast)
- Archive old test runs
Documentation
Test Documentation
- Document test structure in README
- Add comments for complex test logic
- Document page object APIs
- Create testing guide for contributors
- Document CI pipeline configuration
- Maintain test data documentation
- Document mocking strategies
Knowledge Sharing
- Share test results in PR reviews
- Conduct test review sessions
- Create troubleshooting guide
- Document common test patterns
- Share CI optimization learnings
- Create onboarding guide for new contributors
Quality Gates
Before Committing
- All tests pass locally
- New tests added for new features
- No new flaky tests introduced
- Test execution time acceptable
- Code reviewed for maintainability
- Accessibility tests pass
- Visual regression tests updated
Before Merging PR
- All CI tests pass
- No flaky test failures
- Test coverage maintained or improved
- Test artifacts reviewed (screenshots, videos)
- Performance impact assessed
- Breaking changes documented
Before Production Deploy
- Full E2E suite passes on staging
- Performance tests pass
- Accessibility tests pass
- Visual regression tests reviewed
- Smoke tests identified for post-deploy
- Rollback plan documented
Advanced Topics
Cross-Browser Testing
- Test on Chromium (Chrome/Edge)
- Test on Firefox
- Test on WebKit (Safari)
- Handle browser-specific quirks
- Test with different browser versions
Internationalization (i18n)
- Test with different locales
- Verify RTL languages (Arabic, Hebrew)
- Test date/time formatting
- Test currency formatting
- Verify translations loaded correctly
Security Testing
- Test authentication flows
- Test authorization (role-based access)
- Test XSS prevention
- Test CSRF protection
- Test input sanitization
- Test secure headers (CSP, etc.)
Performance Testing
- Measure page load time
- Test Core Web Vitals (LCP, FID, CLS)
- Test with network throttling
- Test with CPU throttling
- Monitor memory usage
- Test bundle size impact
Success Metrics
- Test coverage > 80% for critical paths
- Test execution time < 10 minutes
- Test flakiness rate < 2%
- Zero P0 bugs in production from untested areas
- All critical user journeys tested
- 100% of new features have E2E tests
- Test results visible in every PR
- Tests block merge on failure
Note: This checklist is comprehensive but should be adapted to your project's specific needs. Not all items apply to every project. Prioritize based on risk, criticality, and available resources.
OrchestKit Priority:
- Analysis flow (URL → Progress → Artifact)
- SSE real-time updates
- Error handling and recovery
- Agent orchestration visibility
- Accessibility and responsive design
Examples (3)
A11y Testing Examples
Accessibility Testing Examples
Complete code examples for automated accessibility testing.
jest-axe Component Tests
Basic Button Test
// src/components/Button.test.tsx
import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { Button } from './Button';
expect.extend(toHaveNoViolations);
describe('Button Accessibility', () => {
test('has no accessibility violations', async () => {
const { container } = render(<Button>Click me</Button>);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('disabled button is accessible', async () => {
const { container } = render(<Button disabled>Cannot click</Button>);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('icon-only button has accessible name', async () => {
const { container } = render(
<Button aria-label="Close dialog">
<XIcon />
</Button>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
});Form Component Test
// src/components/LoginForm.test.tsx
import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { LoginForm } from './LoginForm';
expect.extend(toHaveNoViolations);
describe('LoginForm Accessibility', () => {
test('form has no accessibility violations', async () => {
const { container } = render(<LoginForm />);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('form with errors is accessible', async () => {
const { container } = render(
<LoginForm
errors={{
email: 'Invalid email address',
password: 'Password is required',
}}
/>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('form with loading state is accessible', async () => {
const { container } = render(<LoginForm isLoading />);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('meets WCAG 2.1 Level AA', async () => {
const { container } = render(<LoginForm />);
const results = await axe(container, {
runOnly: {
type: 'tag',
values: ['wcag2a', 'wcag2aa', 'wcag21aa'],
},
});
expect(results).toHaveNoViolations();
});
});Modal Component Test
// src/components/Modal.test.tsx
import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { Modal } from './Modal';
expect.extend(toHaveNoViolations);
describe('Modal Accessibility', () => {
test('open modal has no violations', async () => {
const { container } = render(
<Modal isOpen onClose={() => {}}>
<h2>Modal Title</h2>
<p>Modal content</p>
</Modal>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('modal has proper ARIA attributes', async () => {
const { container } = render(
<Modal isOpen onClose={() => {}} ariaLabel="Settings">
<p>Settings content</p>
</Modal>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('modal with complex content is accessible', async () => {
const { container } = render(
<Modal isOpen onClose={() => {}}>
<h2>Complex Modal</h2>
<form>
<label htmlFor="name">Name</label>
<input id="name" type="text" />
<button type="submit">Save</button>
</form>
</Modal>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
});Custom Dropdown Test
// src/components/Dropdown.test.tsx
import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { axe, toHaveNoViolations } from 'jest-axe';
import { Dropdown } from './Dropdown';
expect.extend(toHaveNoViolations);
describe('Dropdown Accessibility', () => {
const options = [
{ value: 'apple', label: 'Apple' },
{ value: 'banana', label: 'Banana' },
{ value: 'cherry', label: 'Cherry' },
];
test('closed dropdown has no violations', async () => {
const { container } = render(
<Dropdown label="Select fruit" options={options} />
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('open dropdown has no violations', async () => {
const user = userEvent.setup();
const { container } = render(
<Dropdown label="Select fruit" options={options} />
);
const button = screen.getByRole('button', { name: /select fruit/i });
await user.click(button);
await waitFor(async () => {
const results = await axe(container);
expect(results).toHaveNoViolations();
});
});
test('dropdown with selected value is accessible', async () => {
const { container } = render(
<Dropdown
label="Select fruit"
options={options}
value="banana"
/>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
test('disabled dropdown is accessible', async () => {
const { container } = render(
<Dropdown
label="Select fruit"
options={options}
disabled
/>
);
const results = await axe(container);
expect(results).toHaveNoViolations();
});
});Playwright + axe-core E2E Tests
Page-Level Test
// tests/a11y/homepage.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test.describe('Homepage Accessibility', () => {
test('should not have accessibility violations', async ({ page }) => {
await page.goto('/');
const accessibilityScanResults = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])
.analyze();
expect(accessibilityScanResults.violations).toEqual([]);
});
test('navigation menu is accessible', async ({ page }) => {
await page.goto('/');
// Scan only the navigation
const results = await new AxeBuilder({ page })
.include('nav')
.analyze();
expect(results.violations).toEqual([]);
});
test('footer is accessible', async ({ page }) => {
await page.goto('/');
const results = await new AxeBuilder({ page })
.include('footer')
.analyze();
expect(results.violations).toEqual([]);
});
});User Journey Test
// tests/a11y/checkout.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test.describe('Checkout Flow Accessibility', () => {
test('entire checkout flow is accessible', async ({ page }) => {
// Step 1: Cart page
await page.goto('/cart');
let results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa'])
.analyze();
expect(results.violations).toEqual([]);
// Step 2: Add item and proceed
await page.getByRole('button', { name: 'Proceed to Checkout' }).click();
// Step 3: Shipping form
await page.waitForURL('/checkout/shipping');
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Fill form
await page.getByLabel('Email').fill('test@example.com');
await page.getByLabel('Street Address').fill('123 Main St');
await page.getByRole('button', { name: 'Continue to Payment' }).click();
// Step 4: Payment form
await page.waitForURL('/checkout/payment');
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Step 5: Review order
await page.getByRole('button', { name: 'Review Order' }).click();
await page.waitForURL('/checkout/review');
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});
test('validation errors are accessible', async ({ page }) => {
await page.goto('/checkout/shipping');
// Submit without filling required fields
await page.getByRole('button', { name: 'Continue' }).click();
// Wait for error messages to appear
await page.waitForSelector('[role="alert"]');
const results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});
});Dynamic Content Test
// tests/a11y/search.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test.describe('Search Accessibility', () => {
test('search interface is accessible', async ({ page }) => {
await page.goto('/search');
// Initial state
let results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Type search query
await page.getByRole('searchbox', { name: 'Search products' }).fill('laptop');
// Wait for autocomplete suggestions
await page.waitForSelector('[role="listbox"]');
// Scan with suggestions visible
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Select a suggestion
await page.getByRole('option', { name: /laptop/i }).first().click();
// Wait for results page
await page.waitForURL('**/search?q=laptop');
// Scan results page
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});
test('empty search results accessible', async ({ page }) => {
await page.goto('/search?q=nonexistentproduct123');
const results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});
});Modal Interaction Test
// tests/a11y/modal.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test.describe('Modal Accessibility', () => {
test('modal maintains accessibility through interactions', async ({ page }) => {
await page.goto('/dashboard');
// Initial state (modal closed)
let results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Open modal
await page.getByRole('button', { name: 'Open Settings' }).click();
await page.waitForSelector('[role="dialog"]');
// Modal open state
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Interact with modal form
await page.getByLabel('Display Name').fill('John Doe');
await page.getByLabel('Email Notifications').check();
// Still accessible after interactions
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
// Close modal
await page.getByRole('button', { name: 'Save' }).click();
await page.waitForSelector('[role="dialog"]', { state: 'hidden' });
// After modal closes
results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});
test('focus is trapped in modal', async ({ page }) => {
await page.goto('/dashboard');
await page.getByRole('button', { name: 'Open Settings' }).click();
await page.waitForSelector('[role="dialog"]');
// Tab through all elements
const focusableElements = await page.locator('[role="dialog"] :focus-visible').count();
for (let i = 0; i < focusableElements + 2; i++) {
await page.keyboard.press('Tab');
}
// Focus should still be within modal
const focusedElement = await page.evaluate(() => {
const activeElement = document.activeElement;
return activeElement?.closest('[role="dialog"]') !== null;
});
expect(focusedElement).toBe(true);
});
});Custom axe Rules
Creating a Custom Rule
// tests/utils/custom-axe-rules.ts
import { configureAxe } from 'jest-axe';
export const axeWithCustomRules = configureAxe({
rules: {
// Ensure all buttons have explicit type attribute
'button-type': {
enabled: true,
selector: 'button:not([type])',
any: [],
none: [],
all: ['button-has-type'],
},
},
checks: [
{
id: 'button-has-type',
evaluate: () => false,
metadata: {
impact: 'minor',
messages: {
fail: 'Button must have explicit type attribute (button, submit, or reset)',
},
},
},
],
});Using Custom Rules in Tests
// src/components/Form.test.tsx
import { render } from '@testing-library/react';
import { toHaveNoViolations } from 'jest-axe';
import { axeWithCustomRules } from '../tests/utils/custom-axe-rules';
expect.extend(toHaveNoViolations);
test('form buttons have explicit type', async () => {
const { container } = render(
<form>
<button type="button">Cancel</button>
<button type="submit">Submit</button>
</form>
);
const results = await axeWithCustomRules(container);
expect(results).toHaveNoViolations();
});CI Pipeline Configuration
GitHub Actions Workflow
# .github/workflows/a11y-tests.yml
name: Accessibility Tests
on:
push:
branches: [main, develop]
pull_request:
branches: [main, develop]
jobs:
unit-a11y:
name: Unit Accessibility Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run jest-axe tests
run: npm run test:a11y:unit
- name: Upload coverage
uses: codecov/codecov-action@v4
with:
files: ./coverage/lcov.info
flags: accessibility
e2e-a11y:
name: E2E Accessibility Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Playwright
run: npx playwright install --with-deps chromium
- name: Build application
run: npm run build
env:
CI: true
- name: Start application
run: npm run start &
env:
PORT: 3000
NODE_ENV: test
- name: Wait for application
run: npx wait-on http://localhost:3000 --timeout 60000
- name: Run Playwright accessibility tests
run: npx playwright test tests/a11y/
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@v4
with:
name: playwright-a11y-report
path: playwright-report/
retention-days: 30
- name: Comment PR with results
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const report = fs.readFileSync('playwright-report/index.html', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '## ♿ Accessibility Test Results\n\nView full report in artifacts.'
});
lighthouse:
name: Lighthouse Accessibility Audit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Build application
run: npm run build
- name: Start application
run: npm run start &
- name: Wait for application
run: npx wait-on http://localhost:3000
- name: Run Lighthouse CI
run: |
npm install -g @lhci/cli@0.13.x
lhci autorun
env:
LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}
- name: Upload Lighthouse results
uses: actions/upload-artifact@v4
with:
name: lighthouse-results
path: .lighthouseci/Package.json Test Scripts
{
"scripts": {
"test:a11y:unit": "vitest run --coverage src/**/*.a11y.test.{ts,tsx}",
"test:a11y:unit:watch": "vitest watch src/**/*.a11y.test.{ts,tsx}",
"test:a11y:e2e": "playwright test tests/a11y/",
"test:a11y:all": "npm run test:a11y:unit && npm run test:a11y:e2e",
"test:a11y:lighthouse": "lhci autorun"
}
}These examples provide a comprehensive foundation for implementing automated accessibility testing in your application.
E2e Test Patterns
E2E Test Patterns
Complete User Flow Test
import { test, expect } from '@playwright/test';
test.describe('Checkout Flow', () => {
test('user can complete purchase', async ({ page }) => {
// Navigate to product
await page.goto('/products');
await page.getByRole('link', { name: 'Premium Widget' }).click();
// Add to cart
await page.getByRole('button', { name: 'Add to cart' }).click();
await expect(page.getByRole('alert')).toContainText('Added to cart');
// Go to checkout
await page.getByRole('link', { name: 'Cart' }).click();
await page.getByRole('button', { name: 'Checkout' }).click();
// Fill shipping info
await page.getByLabel('Email').fill('test@example.com');
await page.getByLabel('Full name').fill('Test User');
await page.getByLabel('Address').fill('123 Test St');
await page.getByLabel('City').fill('Test City');
await page.getByRole('combobox', { name: 'State' }).selectOption('CA');
await page.getByLabel('ZIP').fill('90210');
// Fill payment
await page.getByLabel('Card number').fill('4242424242424242');
await page.getByLabel('Expiry').fill('12/25');
await page.getByLabel('CVC').fill('123');
// Submit order
await page.getByRole('button', { name: 'Place order' }).click();
// Verify confirmation
await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
await expect(page.getByText(/order #/i)).toBeVisible();
});
});Page Object Model
// pages/LoginPage.ts
import { Page, Locator, expect } from '@playwright/test';
export class LoginPage {
private readonly emailInput: Locator;
private readonly passwordInput: Locator;
private readonly submitButton: Locator;
private readonly errorMessage: Locator;
constructor(private page: Page) {
this.emailInput = page.getByLabel('Email');
this.passwordInput = page.getByLabel('Password');
this.submitButton = page.getByRole('button', { name: 'Sign in' });
this.errorMessage = page.getByRole('alert');
}
async goto() {
await this.page.goto('/login');
}
async login(email: string, password: string) {
await this.emailInput.fill(email);
await this.passwordInput.fill(password);
await this.submitButton.click();
}
async expectError(message: string) {
await expect(this.errorMessage).toContainText(message);
}
async expectLoggedIn() {
await expect(this.page).toHaveURL('/dashboard');
}
}
// tests/login.spec.ts
import { test } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';
test.describe('Login', () => {
test('successful login', async ({ page }) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await loginPage.login('user@example.com', 'password123');
await loginPage.expectLoggedIn();
});
test('invalid credentials', async ({ page }) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await loginPage.login('user@example.com', 'wrongpassword');
await loginPage.expectError('Invalid email or password');
});
});Authentication Fixture
// fixtures/auth.ts
import { test as base, Page } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';
type AuthFixtures = {
authenticatedPage: Page;
adminPage: Page;
};
export const test = base.extend<AuthFixtures>({
authenticatedPage: async ({ page }, use) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await loginPage.login('user@example.com', 'password123');
await use(page);
},
adminPage: async ({ page }, use) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await loginPage.login('admin@example.com', 'adminpass');
await use(page);
},
});
// tests/dashboard.spec.ts
import { test } from '../fixtures/auth';
test('user can view dashboard', async ({ authenticatedPage }) => {
await authenticatedPage.goto('/dashboard');
// Already logged in
});
test('admin can access admin panel', async ({ adminPage }) => {
await adminPage.goto('/admin');
// Already logged in as admin
});Visual Regression Test
import { test, expect } from '@playwright/test';
test.describe('Visual Regression', () => {
test('homepage looks correct', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('homepage.png');
});
test('hero section visual', async ({ page }) => {
await page.goto('/');
const hero = page.locator('[data-testid="hero"]');
await expect(hero).toHaveScreenshot('hero.png');
});
test('responsive design - mobile', async ({ page }) => {
await page.setViewportSize({ width: 375, height: 667 });
await page.goto('/');
await expect(page).toHaveScreenshot('homepage-mobile.png');
});
test('dark mode', async ({ page }) => {
await page.emulateMedia({ colorScheme: 'dark' });
await page.goto('/');
await expect(page).toHaveScreenshot('homepage-dark.png');
});
});API Mocking in E2E
import { test, expect } from '@playwright/test';
test('handles API error gracefully', async ({ page }) => {
// Mock API to return error
await page.route('/api/users', (route) => {
route.fulfill({
status: 500,
body: JSON.stringify({ error: 'Server error' }),
});
});
await page.goto('/users');
await expect(page.getByText('Unable to load users')).toBeVisible();
await expect(page.getByRole('button', { name: 'Retry' })).toBeVisible();
});
test('shows loading state', async ({ page }) => {
// Delay API response
await page.route('/api/users', async (route) => {
await new Promise((resolve) => setTimeout(resolve, 2000));
route.fulfill({
status: 200,
body: JSON.stringify([{ id: 1, name: 'User' }]),
});
});
await page.goto('/users');
await expect(page.getByTestId('loading-skeleton')).toBeVisible();
await expect(page.getByText('User')).toBeVisible({ timeout: 5000 });
});Multi-Tab Test
import { test, expect } from '@playwright/test';
test('multi-tab checkout flow', async ({ context }) => {
// Open two tabs
const page1 = await context.newPage();
const page2 = await context.newPage();
// Add item in first tab
await page1.goto('/products');
await page1.getByRole('button', { name: 'Add to cart' }).click();
// Verify cart updated in second tab
await page2.goto('/cart');
await expect(page2.getByRole('listitem')).toHaveCount(1);
});File Upload Test
import { test, expect } from '@playwright/test';
import path from 'path';
test('user can upload profile photo', async ({ page }) => {
await page.goto('/settings/profile');
// Upload file
const fileInput = page.locator('input[type="file"]');
await fileInput.setInputFiles(path.join(__dirname, 'fixtures/photo.jpg'));
// Verify preview
await expect(page.getByAltText('Profile preview')).toBeVisible();
// Save
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByRole('alert')).toContainText('Profile updated');
});Accessibility Test
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test.describe('Accessibility', () => {
test('homepage has no a11y violations', async ({ page }) => {
await page.goto('/');
const results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);
});
test('login form is accessible', async ({ page }) => {
await page.goto('/login');
const results = await new AxeBuilder({ page })
.include('[data-testid="login-form"]')
.analyze();
expect(results.violations).toEqual([]);
});
});Orchestkit E2e Tests
OrchestKit E2E Test Examples
Complete E2E test suite examples for OrchestKit's analysis workflow using Playwright + TypeScript.
Test Configuration
playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './tests/e2e',
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: 'html',
use: {
baseURL: 'http://localhost:5173',
trace: 'on-first-retry',
screenshot: 'only-on-failure',
},
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},
{
name: 'mobile',
use: { ...devices['iPhone 13'] },
},
],
webServer: {
command: 'npm run dev',
url: 'http://localhost:5173',
reuseExistingServer: !process.env.CI,
},
});Page Objects
HomePage (URL Submission)
// tests/e2e/pages/HomePage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage } from '.claude/skills/webapp-testing/assets/playwright-test-template';
export class HomePage extends BasePage {
readonly urlInput: Locator;
readonly analyzeButton: Locator;
readonly analysisTypeSelect: Locator;
readonly recentAnalyses: Locator;
constructor(page: Page) {
super(page);
this.urlInput = page.getByTestId('url-input');
this.analyzeButton = page.getByRole('button', { name: /analyze/i });
this.analysisTypeSelect = page.getByTestId('analysis-type-select');
this.recentAnalyses = page.getByTestId('recent-analyses-list');
}
async goto(): Promise<void> {
await super.goto('/');
await this.waitForLoad();
}
async submitUrl(url: string, analysisType = 'comprehensive'): Promise<void> {
await this.urlInput.fill(url);
if (analysisType !== 'comprehensive') {
await this.analysisTypeSelect.selectOption(analysisType);
}
await this.analyzeButton.click();
}
async getRecentAnalysesCount(): Promise<number> {
return await this.recentAnalyses.locator('li').count();
}
async clickRecentAnalysis(index: number): Promise<void> {
await this.recentAnalyses.locator('li').nth(index).click();
}
}AnalysisProgressPage (SSE Stream)
// tests/e2e/pages/AnalysisProgressPage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage, WaitHelpers } from '.claude/skills/webapp-testing/assets/playwright-test-template';
export class AnalysisProgressPage extends BasePage {
readonly progressBar: Locator;
readonly progressPercentage: Locator;
readonly statusBadge: Locator;
readonly agentCards: Locator;
readonly errorMessage: Locator;
readonly cancelButton: Locator;
readonly viewArtifactButton: Locator;
private waitHelpers: WaitHelpers;
constructor(page: Page) {
super(page);
this.progressBar = page.getByTestId('analysis-progress-bar');
this.progressPercentage = page.getByTestId('progress-percentage');
this.statusBadge = page.getByTestId('status-badge');
this.agentCards = page.getByTestId('agent-card');
this.errorMessage = page.getByTestId('error-message');
this.cancelButton = page.getByRole('button', { name: /cancel/i });
this.viewArtifactButton = page.getByRole('button', { name: /view artifact/i });
this.waitHelpers = new WaitHelpers(page);
}
async waitForAnalysisComplete(timeout = 60000): Promise<void> {
await this.page.waitForFunction(
() => {
const badge = document.querySelector('[data-testid="status-badge"]');
return badge?.textContent?.toLowerCase().includes('complete');
},
{ timeout }
);
}
async waitForProgress(percentage: number, timeout = 30000): Promise<void> {
await this.page.waitForFunction(
(targetPercentage) => {
const progressText = document.querySelector('[data-testid="progress-percentage"]')?.textContent;
const currentPercentage = parseInt(progressText || '0', 10);
return currentPercentage >= targetPercentage;
},
percentage,
{ timeout }
);
}
async getAgentStatus(agentName: string): Promise<'pending' | 'running' | 'completed' | 'failed'> {
const agentCard = this.agentCards.filter({ hasText: agentName }).first();
const statusElement = agentCard.getByTestId('agent-status');
const status = await statusElement.textContent();
return status?.toLowerCase() as any;
}
async getCompletedAgentsCount(): Promise<number> {
return await this.agentCards.filter({ has: this.page.getByText('completed') }).count();
}
async cancelAnalysis(): Promise<void> {
await this.cancelButton.click();
}
async goToArtifact(): Promise<void> {
await this.viewArtifactButton.click();
}
async getErrorText(): Promise<string | null> {
if (await this.errorMessage.isVisible()) {
return await this.errorMessage.textContent();
}
return null;
}
}ArtifactPage (View Results)
// tests/e2e/pages/ArtifactPage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage } from '.claude/skills/webapp-testing/assets/playwright-test-template';
export class ArtifactPage extends BasePage {
readonly artifactTitle: Locator;
readonly sourceUrl: Locator;
readonly qualityScore: Locator;
readonly findingsSection: Locator;
readonly downloadButton: Locator;
readonly shareButton: Locator;
readonly searchInput: Locator;
readonly sectionTabs: Locator;
constructor(page: Page) {
super(page);
this.artifactTitle = page.getByTestId('artifact-title');
this.sourceUrl = page.getByTestId('source-url');
this.qualityScore = page.getByTestId('quality-score');
this.findingsSection = page.getByTestId('findings-section');
this.downloadButton = page.getByRole('button', { name: /download/i });
this.shareButton = page.getByRole('button', { name: /share/i });
this.searchInput = page.getByTestId('artifact-search');
this.sectionTabs = page.getByRole('tab');
}
async getQualityScoreValue(): Promise<number> {
const scoreText = await this.qualityScore.textContent();
return parseFloat(scoreText || '0');
}
async searchInArtifact(query: string): Promise<void> {
await this.searchInput.fill(query);
await this.page.waitForTimeout(300); // Debounce
}
async switchToTab(tabName: string): Promise<void> {
await this.sectionTabs.filter({ hasText: tabName }).click();
}
async downloadArtifact(): Promise<void> {
const downloadPromise = this.page.waitForEvent('download');
await this.downloadButton.click();
await downloadPromise;
}
async getFindingsCount(): Promise<number> {
return await this.findingsSection.locator('[data-testid="finding-item"]').count();
}
}Test Suites
1. Happy Path - Complete Analysis Flow
// tests/e2e/analysis-flow.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
import { ArtifactPage } from './pages/ArtifactPage';
import { ApiMocker, CustomAssertions } from '.claude/skills/webapp-testing/assets/playwright-test-template';
test.describe('Analysis Flow - Happy Path', () => {
test('should complete full analysis flow from URL submission to artifact view', async ({ page }) => {
// 1. Submit URL for analysis
const homePage = new HomePage(page);
await homePage.goto();
await expect(homePage.urlInput).toBeVisible();
await homePage.submitUrl('https://example.com/article', 'comprehensive');
// 2. Monitor progress with SSE
const progressPage = new AnalysisProgressPage(page);
await expect(progressPage.progressBar).toBeVisible();
// Wait for initial progress
await progressPage.waitForProgress(10);
// Check at least one agent is running
const agentStatus = await progressPage.getAgentStatus('Tech Comparator');
expect(['running', 'completed']).toContain(agentStatus);
// Wait for completion (with timeout for real API)
await progressPage.waitForAnalysisComplete(90000); // 90s timeout
// Verify all agents completed
const completedCount = await progressPage.getCompletedAgentsCount();
expect(completedCount).toBeGreaterThan(0);
// 3. Navigate to artifact
await progressPage.goToArtifact();
// 4. Verify artifact content
const artifactPage = new ArtifactPage(page);
await expect(artifactPage.artifactTitle).toBeVisible();
const qualityScore = await artifactPage.getQualityScoreValue();
expect(qualityScore).toBeGreaterThan(0);
expect(qualityScore).toBeLessThanOrEqual(10);
const findingsCount = await artifactPage.getFindingsCount();
expect(findingsCount).toBeGreaterThan(0);
});
});2. SSE Progress Updates
// tests/e2e/sse-progress.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
import { ApiMocker } from '.claude/skills/webapp-testing/assets/playwright-test-template';
test.describe('SSE Progress Updates', () => {
test('should show real-time progress updates via SSE', async ({ page }) => {
// Mock SSE stream with progress events
const apiMocker = new ApiMocker(page);
const sseEvents = [
{ data: { type: 'progress', percentage: 0, message: 'Starting analysis...' } },
{ data: { type: 'agent_start', agent: 'Tech Comparator' }, delay: 500 },
{ data: { type: 'progress', percentage: 25, message: 'Tech Comparator running...' } },
{ data: { type: 'agent_complete', agent: 'Tech Comparator' }, delay: 1000 },
{ data: { type: 'progress', percentage: 50, message: 'Security Auditor running...' } },
{ data: { type: 'agent_complete', agent: 'Security Auditor' }, delay: 1000 },
{ data: { type: 'progress', percentage: 100, message: 'Analysis complete!' } },
{ data: { type: 'complete', artifact_id: 'test-artifact-123' } },
];
await apiMocker.mockSSE(/api\/v1\/analyses\/\d+\/stream/, sseEvents);
// Submit analysis
const homePage = new HomePage(page);
await homePage.goto();
await homePage.submitUrl('https://example.com/test');
// Monitor progress updates
const progressPage = new AnalysisProgressPage(page);
// Wait for 25% progress
await progressPage.waitForProgress(25);
expect(await progressPage.progressPercentage.textContent()).toContain('25');
// Wait for 50% progress
await progressPage.waitForProgress(50);
expect(await progressPage.progressPercentage.textContent()).toContain('50');
// Wait for completion
await progressPage.waitForProgress(100);
await expect(progressPage.statusBadge).toContainText('Complete');
});
test('should handle SSE connection errors gracefully', async ({ page }) => {
// Mock SSE connection failure
await page.route(/api\/v1\/analyses\/\d+\/stream/, (route) => {
route.abort('failed');
});
const homePage = new HomePage(page);
await homePage.goto();
await homePage.submitUrl('https://example.com/test');
const progressPage = new AnalysisProgressPage(page);
// Should show error message
await expect(progressPage.errorMessage).toBeVisible();
const errorText = await progressPage.getErrorText();
expect(errorText).toContain('connection');
});
});3. Error Handling
// tests/e2e/error-handling.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
import { ApiMocker, CustomAssertions } from '.claude/skills/webapp-testing/assets/playwright-test-template';
test.describe('Error Handling', () => {
test('should show validation error for invalid URL', async ({ page }) => {
const homePage = new HomePage(page);
await homePage.goto();
await homePage.submitUrl('not-a-valid-url');
const assertions = new CustomAssertions(page);
await assertions.expectToast('Please enter a valid URL', 'error');
});
test('should handle API error during analysis submission', async ({ page }) => {
const apiMocker = new ApiMocker(page);
await apiMocker.mockError(/api\/v1\/analyses/, 500, 'Internal server error');
const homePage = new HomePage(page);
await homePage.goto();
await homePage.submitUrl('https://example.com/test');
const assertions = new CustomAssertions(page);
await assertions.expectToast('Failed to start analysis', 'error');
});
test('should handle analysis failure from backend', async ({ page }) => {
const apiMocker = new ApiMocker(page);
// Mock successful submission
await apiMocker.mockSuccess(/api\/v1\/analyses$/, {
id: 123,
status: 'processing',
url: 'https://example.com/test',
});
// Mock SSE with failure event
await apiMocker.mockSSE(/api\/v1\/analyses\/123\/stream/, [
{ data: { type: 'progress', percentage: 10 } },
{ data: { type: 'error', message: 'Failed to fetch content' } },
]);
const homePage = new HomePage(page);
await homePage.goto();
await homePage.submitUrl('https://example.com/test');
const progressPage = new AnalysisProgressPage(page);
await expect(progressPage.errorMessage).toBeVisible();
const errorText = await progressPage.getErrorText();
expect(errorText).toContain('Failed to fetch content');
});
test('should allow retry after failed analysis', async ({ page }) => {
const homePage = new HomePage(page);
const progressPage = new AnalysisProgressPage(page);
await homePage.goto();
await homePage.submitUrl('https://example.com/test');
// Wait for error state
await expect(progressPage.errorMessage).toBeVisible();
// Click retry button
const retryButton = page.getByRole('button', { name: /retry/i });
await retryButton.click();
// Should restart analysis
await expect(progressPage.progressBar).toBeVisible();
});
});4. Cancellation & Cleanup
// tests/e2e/cancellation.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
test.describe('Analysis Cancellation', () => {
test('should cancel in-progress analysis', async ({ page }) => {
const homePage = new HomePage(page);
await homePage.goto();
await homePage.submitUrl('https://example.com/long-analysis');
const progressPage = new AnalysisProgressPage(page);
// Wait for analysis to start
await progressPage.waitForProgress(10);
// Cancel analysis
await progressPage.cancelAnalysis();
// Confirm cancellation in dialog
page.on('dialog', dialog => dialog.accept());
// Should redirect back to home
await expect(page).toHaveURL('/');
// Should show cancellation toast
const assertions = new CustomAssertions(page);
await assertions.expectToast('Analysis cancelled', 'info');
});
test('should not allow cancellation of completed analysis', async ({ page }) => {
// Navigate to completed analysis
await page.goto('/analysis/completed-123');
const progressPage = new AnalysisProgressPage(page);
// Cancel button should be disabled or hidden
await expect(progressPage.cancelButton).not.toBeVisible();
});
});5. Responsive & Mobile
// tests/e2e/responsive.spec.ts
import { test, expect, devices } from '@playwright/test';
import { HomePage } from './pages/HomePage';
test.describe('Responsive Design', () => {
test.use({ ...devices['iPhone 13'] });
test('should work on mobile viewport', async ({ page }) => {
const homePage = new HomePage(page);
await homePage.goto();
// URL input should be visible and usable
await expect(homePage.urlInput).toBeVisible();
await homePage.urlInput.fill('https://example.com/mobile-test');
// Button should be tappable
await homePage.analyzeButton.click();
// Progress page should be mobile-friendly
const progressBar = page.getByTestId('analysis-progress-bar');
await expect(progressBar).toBeVisible();
// Agent cards should stack vertically
const agentCards = page.getByTestId('agent-card');
const firstCard = agentCards.first();
const secondCard = agentCards.nth(1);
const firstBox = await firstCard.boundingBox();
const secondBox = await secondCard.boundingBox();
// Second card should be below first (Y coordinate)
expect(secondBox!.y).toBeGreaterThan(firstBox!.y + firstBox!.height);
});
});6. Accessibility
// tests/e2e/accessibility.spec.ts
import { test, expect } from '@playwright/test';
import { HomePage } from './pages/HomePage';
import { AnalysisProgressPage } from './pages/AnalysisProgressPage';
test.describe('Accessibility', () => {
test('should be keyboard navigable', async ({ page }) => {
const homePage = new HomePage(page);
await homePage.goto();
// Tab to URL input
await page.keyboard.press('Tab');
await expect(homePage.urlInput).toBeFocused();
// Type URL
await page.keyboard.type('https://example.com/test');
// Tab to analyze button
await page.keyboard.press('Tab');
await expect(homePage.analyzeButton).toBeFocused();
// Press Enter to submit
await page.keyboard.press('Enter');
// Should navigate to progress page
const progressPage = new AnalysisProgressPage(page);
await expect(progressPage.progressBar).toBeVisible();
});
test('should have proper ARIA labels', async ({ page }) => {
const homePage = new HomePage(page);
await homePage.goto();
// URL input should have aria-label
await expect(homePage.urlInput).toHaveAttribute('aria-label');
// Submit button should have accessible name
const buttonName = await homePage.analyzeButton.getAttribute('aria-label');
expect(buttonName).toBeTruthy();
});
test('should announce progress updates to screen readers', async ({ page }) => {
await page.goto('/analysis/123');
const progressPage = new AnalysisProgressPage(page);
// Progress region should have aria-live
await expect(progressPage.progressBar).toHaveAttribute('aria-live', 'polite');
// Status updates should have role="status"
const statusRegion = page.getByTestId('status-updates');
await expect(statusRegion).toHaveAttribute('role', 'status');
});
});Running Tests
# Install Playwright
npm install -D @playwright/test
npx playwright install
# Run all tests
npx playwright test
# Run specific suite
npx playwright test tests/e2e/analysis-flow.spec.ts
# Run in UI mode (interactive)
npx playwright test --ui
# Run in headed mode (see browser)
npx playwright test --headed
# Run on specific browser
npx playwright test --project=chromium
# Debug mode
npx playwright test --debug
# Generate test report
npx playwright show-reportCI Integration
# .github/workflows/e2e-tests.yml
name: E2E Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis:7
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install dependencies
run: npm ci
- name: Install Playwright browsers
run: npx playwright install --with-deps
- name: Start backend
run: |
cd backend
poetry install
poetry run uvicorn app.main:app --host 0.0.0.0 --port 8500 &
sleep 5
- name: Start frontend
run: |
npm run build
npm run preview &
sleep 3
- name: Run E2E tests
run: npx playwright test
- name: Upload test results
if: always()
uses: actions/upload-artifact@v3
with:
name: playwright-report
path: playwright-report/
retention-days: 30Best Practices
- Use Page Objects - Encapsulate page logic, improve maintainability
- Mock External APIs - Fast, reliable tests without network dependencies
- Wait Strategically - Use
waitForSelector, avoid arbitrary timeouts - Test Real Flows - Mirror actual user journeys
- Handle Async - SSE streams, debounced inputs, loading states
- Accessibility First - Test keyboard nav, ARIA, screen reader announcements
- Visual Regression - Screenshot testing for UI consistency
- CI Integration - Run tests on every PR, block merges on failures
Task Dependency Patterns
CC 2.1.16 Task Management patterns with TaskCreate, TaskUpdate, TaskGet, TaskList tools. Decompose complex work into trackable tasks with dependency chains. Use when managing multi-step implementations, coordinating parallel work, or tracking completion status.
Testing Integration
Integration and contract testing patterns — API endpoint tests, component integration, database testing, Pact contract verification, property-based testing, and Zod schema validation. Use when testing API boundaries, verifying contracts, or validating cross-service integration.
Last updated on