Questions tagged [ab-test]
A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.
52 questions
1 vote
0 answers
21 views
Is it valid to resolve A/B tests earlier using usage data?
I have a piece of software where N% of users convert to a paid plan after a 2 week free trial. I train the model (random forest) to use the first 2 days of usage data to predict whether the user ...
- 135
1 vote
0 answers
16 views
Seeking Fintech Datasets for Variance Reduction Research in A/B Testing
I’m a fourth-year Applied Math student currently writing my diploma thesis on variance reduction in online experiments. My goal is to apply different variance reduction techniques (e.g., ...
1 vote
0 answers
18 views
Which statistical approach is best for diverse conversion rates in a controlled experiment?
Our software startup builds chat bots for ecommerce websites. The chatbot talks to customers that open the chat bot, and has the goal of closing the sale with the store’s main product. We have about ...
- 111
0 votes
0 answers
46 views
Usefulness of a/b test if we already know we must implement the feature
The situation is as follows. Because of compliance reasons, we must implement a change to our registration flow. Party A says: We must run an a/b test so that we understand the effect of that change. ...
- 101
0 votes
1 answer
50 views
Evaluate a Recommender System based on the data between two months
currently my company's planning to use a new Recommender tool/library for a book website, and now we want to compare the result between these two tools (both of the tool use Universal Recommender ...
0 votes
1 answer
45 views
Why NHST based A/B testing is not allowed to continuously monitor?
I am reading a paper regarding A/B test continuous monitoring. https://alexdeng.github.io/public/files/continuousMonitoring.pdf In the abstract part, it states A crucial problem of Null Hypothesis ...
- 101
0 votes
2 answers
137 views
How to check the impact of predictive Model
Suppose that I build an email marketing campaign where the goal is to predict which members will make purchases or not and based on that change the marketing email campaigns. After we build the model, ...
- 11
1 vote
0 answers
16 views
Can I use multi armed bandits to optimize how much both algorithms are weighted when creating a composite score?
So, I'm aware that multi-armed bandits are great for evaluating multiple models and from what I understand, it is mainly used to pick a specific model. I would still like to evaluate two models but I ...
- 11
0 votes
1 answer
78 views
A/B test on model - split on results
I developed a predictive model that assigns the best product (P1, P2, P3) for each customer. I wanted to compare the conversion rate using this model VS the as-is deterministic assignment, so I ...
- 123
1 vote
0 answers
58 views
Causal Inference where the treatment assignment is randomized [closed]
I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
- 111
2 votes
1 answer
2k views
AB testing split algorithm
I want to understand what is the most effective algorithm for splitting. I have ids of users and I want to split them into 2 groups. Now I have 2 variants: Modulo approach - let's say we will place ...
- 123
0 votes
0 answers
243 views
Multivariate testing
I'm going to run a test with 4 different variants (3 variants and a control group), and we want to find the variant with the highest conversion. Are there any resources/methods in R/python to: ...
- 6,156
0 votes
1 answer
130 views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company. Beforehand we want to test this new price list in several countries. A/B test cannot be conducted here ...
1 vote
0 answers
113 views
Practical constraints in A/B testing
I saw an article about an A/B test that google had performed way back. They wanted to decide what shade of blue a button should be and how that affects click-through rate. They divided users randomly ...
- 187
0 votes
0 answers
103 views
What is the right approach to bucket users for algorithms with different coverage for A/B testing
I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide ...
- 571