Post-Test Data Interpretation

As UX researchers, we often encounter a common challenge: deciding whether one design truly outperforms another. Maybe one version of an interface feels faster or looks cleaner. But how do we know if those differences are meaningful - or just the result of chance? To answer that, we turn to statistical comparisons. When comparing numeric metrics like task time or SUS scores, one of the first decisions is whether you’re working with the same users across both designs or two separate groups. If it's the same users, a paired t-test helps isolate the design effect by removing between-subject variability. For independent groups, a two-sample t-test is appropriate, though it requires more participants to detect small effects due to added variability. Binary outcomes like task success or conversion are another common case. If different users are tested on each version, a two-proportion z-test is suitable. But when the same users attempt tasks under both designs, McNemar’s test allows you to evaluate whether the observed success rates differ in a meaningful way. Task time data in UX is often skewed, which violates assumptions of normality. A good workaround is to log-transform the data before calculating confidence intervals, and then back-transform the results to interpret them on the original scale. It gives you a more reliable estimate of the typical time range without being overly influenced by outliers. Statistical significance is only part of the story. Once you establish that a difference is real, the next question is: how big is the difference? For continuous metrics, Cohen’s d is the most common effect size measure, helping you interpret results beyond p-values. For binary data, metrics like risk difference, risk ratio, and odds ratio offer insight into how much more likely users are to succeed or convert with one design over another. Before interpreting any test results, it’s also important to check a few assumptions: are your groups independent, are the data roughly normal (or corrected for skew), and are variances reasonably equal across groups? Fortunately, most statistical tests are fairly robust, especially when sample sizes are balanced. If you're working in R, I’ve included code in the carousel. This walkthrough follows the frequentist approach to comparing designs. I’ll also be sharing a follow-up soon on how to tackle the same questions using Bayesian methods.

Automate post test 𝙖𝙣𝙖𝙡𝙮𝙨𝙞𝙨 𝙖𝙣𝙙 𝙞𝙣𝙩𝙚𝙧𝙥𝙧𝙚𝙩𝙖𝙩𝙞𝙤𝙣 using an AI "debate club" We're starting to do that at Speero, pushed by the brilliant idea and execution of Vasisth S to create a "debate club" of AIs to solve this. This AI-powered “debate club” approach for post-A/B test analysis & interpretation is genuinely innovative—and it addresses several limitations of using AI for interpreting experimental data. Here's why this is both cool and practically useful IMO: First, what's going on in the diagram: • Two LLMs (for & against): The system uses separate large language models (here, versions of LLaMA) to argue in favor of and against the tested variant, interpreting the same data from opposing perspectives. • Merge + Map Steps: These steps combine the arguments and reformat them into a structure suitable for a judging model. • Judge LLM: A third model acts as a neutral “judge,” evaluating both arguments and deciding which is more compelling—ultimately guiding the post-test interpretation. This is where we can add constraints, templates and protocols to consider org-specific goal tree maps and priorities per team/swimlane/etc.    Why this is cool and how it solves some AI pitfalls in normal analysis: 1. Traditional AI often just describes, not interprets • Most AI-based experiment reporting today regurgitates metrics (e.g., uplift %, p-values), but fails to tell you why something worked or didn't—or how to act on it. • This approach encourages causal reasoning and competing narratives, which mimic human decision-making more closely. 2. Bias reduction via structured disagreement • LLMs are prone to confirmation bias—they tend to affirm what’s in the prompt. • Forcing models into “for” and “against” roles ensures balanced exploration, reducing blind acceptance of the test outcome. 3. Math weaknesses compensated by reasoning framing • LLMs struggle with advanced statistical interpretation (e.g., p-hacking, false positives). • But when structured as an argumentative task, they focus on reasoning from observed patterns, not raw statistical computation—an area where LLMs are stronger (so I've been told...I'm pretending to be an expert here). 4. Action-oriented insights • The “judge” model synthesizes arguments and produces a final interpretation, which can be more actionable than a stats dashboard. • This helps non-technical stakeholders (product managers, marketers) understand what to do next without needing a background in statistics.    Thoughts? Want to set this up for your org? DM me. We're working hard to structure AI playbooks like this into Speero's 'Growth Experimentation' product, where we focus on delivering the promise of 5X velocity and impact. Not just saying this to sell and promote, but in a 'say it until we make it' sort of way. Need to hold ourselves accountable :)

“Let’s break it down by segment. There might be something there.” 🚨 That’s the sound of p-hacking creeping in. When you slice your test data after the fact without planning and justification, you’re no longer analyzing. You’re fishing. 🎣 The more cuts you make, the more likely you are to find a lift that’s not actually there. And if you act on that? You’re not optimizing. The solution? ✅ Predefine your segments ✅ Limit comparisons ✅ Control for false discovery ✅ Be okay with saying: “This didn’t move the needle yet.” That doesn’t mean post-test segmentation is useless though. It just means it’s hypothesis fuel, not a conclusion. Here’s how we handle it at Chirpy: If we notice a meaningful signal in a post-hoc segment (e.g., returning users on mobile had a notably different response)… 🔍 We don’t declare a win. Instead, we ask questions like: 🧐 Is this pattern consistent with research data? 🧐 Is there a plausible explanation behind the difference? 🧐 How large is the sample size of the segment? If we do take action, we design a follow-up test just for that segment with a registered hypothesis, new KPIs, and stricter thresholds. The first test raised the question. The next one proves (or disproves) it. That’s how you turn noise into signal and insight into action. You don’t need a win in every test. You need a program built on integrity. Real growth compounds. Made-up wins just keep you busy. CRO isn’t about looking smart. It’s about making sure what “works” actually works. If you’re a marketer stuck between gut feelings, opinion wars, and endless tests that don’t move the needle, you’re not alone. Chirpy's weekly newsletter is where I share what actually works when you’re under pressure to deliver. 👇 Get it in your inbox.