Metrics Architect

{
  "metrics_framework": {
    "feature": "Multi-Agent Workflow Builder",
    "date": "2026-01-02",
    "version": "1.0"
  },
  "okrs": [
    {
      "objective": "Make workflow creation effortless for AI engineers",
      "key_results": [
        {"kr": "Time to first workflow < 30 minutes (from 2+ hours)", "target": "30 min", "baseline": "120 min"},
        {"kr": "70% of new users create a workflow in first session", "target": "70%", "baseline": "N/A"},
        {"kr": "NPS for workflow builder > 50", "target": "50", "baseline": "N/A"}
      ]
    }
  ],
  "kpis": {
    "leading_indicators": [
      {"kpi": "Canvas interactions/session", "definition": "Drag, drop, connect actions", "target": "> 20", "why_leading": "Engagement predicts completion"},
      {"kpi": "Template usage rate", "definition": "% workflows started from template", "target": "> 60%", "why_leading": "Templates reduce friction"}
    ],
    "lagging_indicators": [
      {"kpi": "Workflow completion rate", "definition": "% started workflows that export code", "target": "> 50%"},
      {"kpi": "Return usage (7-day)", "definition": "% users who return within 7 days", "target": "> 40%"},
      {"kpi": "Support tickets", "definition": "Workflow-related tickets/week", "target": "< 10"}
    ]
  },
  "instrumentation_plan": {
    "events_to_track": [
      {"event": "workflow_builder_opened", "properties": ["source", "user_tier"]},
      {"event": "node_added", "properties": ["node_type", "from_template"]},
      {"event": "edge_created", "properties": ["edge_type", "is_conditional"]},
      {"event": "workflow_exported", "properties": ["node_count", "export_format", "duration_seconds"]},
      {"event": "workflow_error", "properties": ["error_type", "node_count"]}
    ],
    "tool_recommendation": "PostHog (already in stack) or Langfuse for LLM-specific",
    "dashboard_views": ["Funnel: Open → Add Node → Connect → Export", "Retention cohorts", "Error rates by node type"]
  },
  "hypothesis_validation": {
    "hypothesis": "AI engineers will build workflows 3x faster with visual builder",
    "experiment_design": {
      "type": "Before/After comparison",
      "control": "Time to build supervisor-worker manually (baseline: 2 hours)",
      "treatment": "Time to build same pattern with visual builder",
      "success_metric": "< 40 minutes (3x improvement)",
      "sample_size": "20 users",
      "duration": "2 weeks post-launch"
    }
  },
  "guardrail_metrics": [
    {"metric": "Code export errors", "threshold": "< 5%", "action_if_breached": "Pause rollout, fix generator"},
    {"metric": "Page load time", "threshold": "< 3s", "action_if_breached": "Optimize bundle size"}
  ],
  "review_cadence": {
    "daily": ["Export errors", "Active users"],
    "weekly": ["Completion rate", "NPS samples"],
    "monthly": ["Full OKR review", "Retention analysis"]
  },
  "received_from": "requirements-translator",
  "handoff_to": "TECHNICAL_IMPLEMENTATION"
}

OBJECTIVE (Qualitative, inspirational)
"Make workflow creation effortless"

KEY RESULTS (Quantitative, measurable)
├── KR1: Time to first workflow < 30 min (from 2+ hours)
├── KR2: 70% create workflow in first session
└── KR3: NPS > 50 for workflow builder

Rules:
- 3-5 KRs per Objective
- Each KR is binary pass/fail
- Ambitious but achievable (70% success = well-calibrated)

LEADING INDICATORS (Predict future outcomes)
├── Early signals of success/failure
├── Actionable (can influence with changes)
├── Examples: engagement, activation, early retention

LAGGING INDICATORS (Confirm outcomes)
├── Final results after the fact
├── Harder to influence directly
├── Examples: revenue, churn, NPS

CONNECTION:
Leading ──► predicts ──► Lagging
Engagement ──► predicts ──► Retention
Activation ──► predicts ──► Revenue

┌─────────────────────────────────────────────────────┐
│              NORTH STAR METRIC                       │
│         "Workflows exported per week"                │
├─────────────────────────────────────────────────────┤
│                                                     │
│  INPUT METRICS (drivers)                            │
│  ├── New users trying builder                       │
│  ├── Template usage rate                            │
│  ├── Canvas engagement (actions/session)            │
│  └── Error rate (inverse)                           │
│                                                     │
│  OUTPUT METRICS (outcomes)                          │
│  ├── User retention                                 │
│  ├── Revenue per user                               │
│  └── Word of mouth / referrals                      │
│                                                     │
└─────────────────────────────────────────────────────┘

EVENT NAMING: noun_verb (snake_case)
├── workflow_created
├── node_added
├── workflow_exported
└── error_occurred

PROPERTIES: Always include
├── user_id (for cohort analysis)
├── session_id (for funnel analysis)
├── timestamp (for time analysis)
├── source (attribution)
└── feature_version (for A/B)

TAXONOMY:
├── page_viewed (page_name, referrer)
├── button_clicked (button_name, context)
├── feature_used (feature_name, details)
├── error_occurred (error_type, context)
└── flow_completed (flow_name, duration)

## Hypothesis
[Statement in form: "We believe X will cause Y"]

## Metrics
- Primary: [One metric to decide success]
- Secondary: [Supporting metrics]
- Guardrails: [Metrics that shouldn't degrade]

## Design
- Type: A/B Test / Before-After / Cohort
- Sample size: [Calculated for statistical power]
- Duration: [Minimum runtime]
- Segments: [Who's included/excluded]

## Success Criteria
- Primary metric improves by X% with p < 0.05
- No guardrail metric degrades by more than Y%

## Rollout Plan
1. 10% → validate instrumentation
2. 50% → gather statistical significance
3. 100% → full rollout if successful

PERFORMANCE GUARDRAILS
├── Page load time < 3s
├── API latency p99 < 500ms
├── Error rate < 1%

BUSINESS GUARDRAILS
├── Conversion rate doesn't drop > 5%
├── Support tickets don't increase > 20%
├── NPS doesn't drop > 10 points

ALERT THRESHOLDS
├── Warning: 80% of guardrail
├── Critical: guardrail breached
└── Action: pause rollout, investigate

# Check existing metrics discussions
gh issue list --search "metrics OR analytics OR KPI" --limit 20

# Look for experiment-related issues
gh issue list --label "experiment" --state all

# Check milestone for metrics alignment
gh milestone view "v2.0" --json title,description

Task Result Fields:
├── token_count    — Total tokens consumed by the agent
├── tool_uses      — Number of tool invocations
├── duration_ms    — Wall-clock execution time
└── model          — Model used (sonnet, opus, haiku)

Cost Efficiency KPIs:
├── tokens_per_tool_use = token_count / tool_uses
├── cost_per_task = token_count × model_rate
├── throughput = tasks_completed / duration_ms
└── model_efficiency = quality_score / cost_per_task