User Research That Engineers Can Actually Run
You check your product analytics dashboard and everything looks healthy. DAU is stable. Page views are up. The feature your team shipped last sprint has “adoption” because people are visiting the page. You report the numbers in the sprint retro. Everyone feels good.
Then you watch a session replay. Users land on the page, click the same button three times, stare at the screen for 40 seconds, and leave to accomplish the task in a spreadsheet instead. Counting the people who entered the store. Not the people who bought anything. Your “adoption” metric was measuring frustration, not success.
- Page views measure visits, not outcomes. A user rage-clicking a broken button registers as “engagement” by every standard metric. Counting the angry button-pressers as “engaged users.” Task completion rate is the metric that actually signals value.
- Session replay turns vague bug reports into reproducible issues. The security camera for your UI. Record DOM mutations, not video. Redact PII automatically. Sample at 5-10% for general traffic, 100% for error sessions.
- A/B tests need behavioral guardrails, not just conversion lifts. A variant that boosts signups but quietly tanks retention is a net loss.
- Five users in 30-minute sessions uncover roughly 80% of usability issues. Engineering teams can run this quarterly without a dedicated researcher.
- Quantitative data shows what happened. Qualitative data explains why. Neither alone produces good product decisions. Both together compound.
Nielsen Norman Group’s research methodology guidelines lay out the evidence-based approach. Jakob Nielsen’s usability heuristics have held up for decades. The tools changed. The principles haven’t.
Beyond Page Views: Metrics That Signal Intent
Standard analytics answers “what happened.” Clicks. Page views. Bounce rate. Session duration. All useful for traffic reporting, all useless for knowing if users actually accomplished anything.
Intent signals live one layer deeper.
Rage clicks fire when a user clicks the same element 3+ times in 1-2 seconds. Something was supposed to happen and didn’t. Track these and you’ll find dozens of broken or misleading UI elements per quarter that standard analytics misses. They’re the most reliable signal of user frustration you can automate.
Dead clicks on non-interactive elements reveal a different problem: the element looks clickable but isn’t, or there’s a feature gap the user expects to exist. A card without a link, a label that resembles a button, a row that should expand but doesn’t.
Scroll depth tells you what content might as well not exist. If 70% of users never scroll past 25% of your settings page, everything below that fold is invisible. You could delete it and nobody would notice.
Task completion rate ties all of this together. Did the user accomplish what they came to do? Not “did they visit the page” but “did they successfully change their notification preferences” or “did they create a new project.” This single metric tells you more than any combination of page views and session duration.
Intent signals tell you where users struggle. But they don’t tell you why. For that, you need experiments.
A/B Testing Infrastructure That Produces Valid Results
Most A/B tests in production are so underpowered they measure noise instead of signal. Three problems keep showing up, and each one on its own invalidates results.
Underpowered tests. For a 5% minimum detectable effect (MDE) at 80% statistical power and 95% confidence, you need roughly 3,200 users per variant. For a 2% MDE, roughly 20,000 per variant. Most teams run with a few hundred users and declare results after three days. The math doesn’t care about your sprint cadence.
Peeking. Checking the p-value daily and stopping when it crosses 0.05 blows up your false positive rate. The stats only work at the pre-committed sample size. Use sequential testing frameworks (like those in Statsig or Eppo) that account for continuous monitoring, or commit to the full sample size and walk away.
Novelty effects. UI changes inflate engagement metrics in the first week from sheer curiosity, not genuine preference. A button color change might show a lift for five days and then regress to baseline. Run for minimum two full business cycles, typically 2-4 weeks.
| Validity Pitfall | What Goes Wrong | Minimum Threshold | How to Enforce |
|---|---|---|---|
| Underpowered sample | Effect is real but sample is too small to detect it. Test “fails” when it should have passed | Calculate required sample size BEFORE starting. Typically 1,000-10,000 per variant depending on effect size | Power calculator in experiment config. Block test start if projected traffic is insufficient |
| Peeking at results | Checking results daily inflates false positive rate from 5% to 30%+. Early “winners” are noise | Run for the full pre-calculated duration. No early stopping without sequential testing correction | Lock dashboard until minimum runtime. Use sequential testing (always valid p-values) if early stopping is needed |
| Multiple comparisons | Testing 10 metrics without correction. At least one will be “significant” by chance (50% probability) | Apply Bonferroni or Benjamini-Hochberg correction. Or pre-declare a single primary metric | Experiment platform enforces primary metric declaration. Secondary metrics flagged as exploratory |
Don’t: Measure only conversion rate. A variant that lifts signups but silently increases churn by the same amount is a net loss. You celebrate the lift in sprint review and discover the damage a quarter later.
Do: Attach guardrail metrics (error rate, P95 latency, revenue per session, 7-day retention) to every experiment. Automatically flag any test where a guardrail degrades beyond a threshold. The guardrail prevents shipping a “win” that quietly destroys something more valuable.
Now you know what works. But even a perfect A/B test can’t tell you why. For that, you need to watch people use your product.
Session Replay at Scale
Session replay connects the dots between aggregate metrics and individual experience. A 40% drop-off on step 3 of your onboarding flow is a number. Watching five replays of users on step 3 reveals that the form asks for information they don’t have yet, or that the “Next” button is below the fold on smaller screens, or that a validation error clears the entire form.
High-traffic apps generate a lot of replay data. How you handle it matters.
Sampling. Record 5-10% of general sessions, 100% of sessions with errors, and 100% of sessions from high-value accounts. This keeps storage costs manageable while guaranteeing you capture every failure.
Privacy masking. Mask all form inputs by default on the client side. GDPR compliance depends on masking happening in the recording layer, not retroactively in storage. The legal exposure from recording unmasked PII and then trying to redact it later is much worse than masking at capture.
Integration. Connect replays to your analytics pipeline so rage clicks, error encounters, and abandonment points automatically bookmark relevant sessions. An engineer investigating a bug should be able to click from an error log directly to the session replay where it happened.
The Quantitative-Qualitative Loop
Analytics surfaces a pattern: 40% drop-off on step 3. Replays show users scrolling past the CTA. Interviews reveal the step asks for information they don’t have yet. The fix addresses the root cause. Analytics confirms the improvement.
Without qualitative data, teams guess. “Make the button bigger.” “Add a tooltip.” “Change the color.” The actual issue was that users needed their API key to proceed and most of them didn’t know where to find it. A link to the API keys page fixed the drop-off. A completely different fix than anything analytics alone would have suggested.
| Signal source | What it reveals | What it misses |
|---|---|---|
| Product analytics | Aggregate behavior patterns, funnel drop-offs, feature usage frequency | Why users behave that way |
| Session replay | Individual user journeys, rage clicks, confusion points | Whether the behavior is representative |
| Usability testing | Root causes of friction, mental model mismatches, unspoken expectations | Scale. Five users is deep but narrow. |
| JTBD interviews | The actual job the user is trying to accomplish, workarounds they’ve built | Current-state behavior (interviews capture intent, not action) |
Each source fills a gap the others leave. Rely only on analytics and you build confidently in the wrong direction. Rely only on interviews and you build for what people say they want instead of what they actually do.
Usability Testing Without a Dedicated Researcher
Five users testing a specific task flow for 20-30 minutes uncovers roughly 80% of usability issues, per Nielsen’s foundational research . Engineering teams can run this quarterly with a structured protocol.
- Define 3-5 task scenarios mapped to real user goals. Not “click the settings button” but “change your notification preferences so you only get emails about critical alerts.” The task must have a clear success state.
- Recruit 5 participants from your target user base. Users who signed up in the last 30 days work well because they’re past initial confusion but haven’t developed workarounds for broken flows.
- Separate the moderator from the builder. The person who built the feature observes silently. A different team member reads the task and asks the participant to think aloud. Moderating your own feature creates unconscious steering toward the “right” path. The struggle is the data.
- Record screen plus audio. Review later for patterns across all five participants. A single confused user is anecdotal. Three users failing at the same point is a design problem.
- Debrief within 24 hours while observations are fresh. Categorize issues by severity: task failure (couldn’t complete), confusion (completed but with difficulty), and cosmetic (completed easily, minor friction).
It’s never about headcount. It’s about willingness to watch someone struggle with something you built and not jump in to help.
Jobs-to-be-Done for Technical Products
“Add CSV export for the audit log.” Sounds like a feature request. It’s actually a solution to a problem you haven’t investigated yet.
The real job: “Prepare compliance evidence for the quarterly audit.” That job might be better served by an automated compliance report, a direct integration with the auditor’s platform, or a pre-built evidence package that eliminates the spreadsheet step entirely. Jobs-to-be-Done (JTBD) is the interview framework that peels back the feature request to find the real task underneath.
Walk through the last time the user performed the job, chronologically. What triggered it. What tools they opened. Where they got stuck. The workarounds are gold. Every workaround is a feature request expressed as behavior instead of words, and workarounds reveal the job with far more honesty than direct questions about what users “want.”
Instrumenting the Product Analytics Stack
How you instrument decides whether you get useful data or dashboards nobody acts on. Three layers, each doing something different.
Event collection (Segment, RudderStack) decouples instrumentation from analytics vendor. Switch vendors by changing a destination config, not 200 event calls scattered across your codebase. If you are wiring analytics events directly to a vendor SDK, you are one vendor migration away from rewriting your entire tracking layer.
Product analytics (Amplitude, PostHog) translates raw events into behavioral insights. Under 50 engineers, PostHog reduces integration overhead because it bundles session replay, feature flags, and A/B testing into a single platform. Larger teams with specialized needs tend toward Amplitude for behavioral analysis alongside dedicated tools for experimentation.
Custom events are where product decisions actually live. project_created, alert_configured, onboarding_step_3_abandoned. Treat your tracking plan like a database schema. Review changes in pull requests. Breaking a tracking plan mid-experiment invalidates every running test. Connecting custom events to UX design practices
closes the measurement-to-iteration loop that most teams leave open.
What the Industry Gets Wrong About User Research
“More data means better product decisions.” More data without qualitative context produces more confident wrong decisions. Teams drown in dashboards showing vanity metrics while the actual user experience goes unobserved. Ten million page views tell you less about product quality than five 30-minute user sessions.
“You need a dedicated UX researcher to run useful research.” Five users uncover roughly 80% of usability issues. Engineering teams with a structured protocol, a screen recorder, and the discipline to observe without defending their choices produce actionable findings. The constraint is willingness, not headcount.
“A/B testing is the gold standard for product decisions.” Only when the sample size and runtime justify the conclusion. A few hundred users over three days, and someone declares a winner in the sprint review. Confirmation bias with a statistics veneer. Not experimentation.
Those healthy DAU numbers on the sprint retro slide? Five user sessions would have revealed that people visit, get confused, and leave to finish the job in a spreadsheet. Instrument intent signals. Run valid experiments. Watch real users struggle. The teams that do this don’t ship fewer features. They ship fewer features nobody uses.