Questions tagged [experiments]
An experiment is a procedure carried out to support, refute, or validate a hypothesis. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated.
39 questions
1 vote
0 answers
18 views
Which statistical approach is best for diverse conversion rates in a controlled experiment?
Our software startup builds chat bots for ecommerce websites. The chatbot talks to customers that open the chat bot, and has the goal of closing the sale with the store’s main product. We have about ...
- 111
1 vote
1 answer
77 views
Which test statistic do you recommend for checking the difference between a dynamic for some parameter in control and treatment group?
I have two groups of patients, a control (A) and a treatment (B), and data for patients' weight at the beginning and end of a treatment period in each group. I need to check if the dynamic of the mean ...
- 111
0 votes
1 answer
34 views
Blocks, factors and treatments in designed experiments
I have recently began studying a course on Designed Experiments and am having some trouble understanding some of the terminology. I've looked at some other answers on the site and I think that I am ...
- 101
3 votes
1 answer
135 views
Taxonomy of train-test split approaches
I am looking for as close as possible for a exhaustive taxonomy of each train-test split approach. For example, the 3 main splits that come to mind are: A non-time based problem - would lead you to a ...
- 445
1 vote
1 answer
480 views
Store preprocessing function along with model in mlflow.keras
The following is a simplified code snipet that is relevant to storing keras LSTM models in MLFlow. ...
- 135
1 vote
0 answers
28 views
Sample Size Calculation for Bayesian A/B Testing with Continuous variables
I'm doing some Bayesian A/B testing and I need to work out an appropriate sample size in order to detect an effect. Unfortunately, there doesn't really seem to be much information about this out there ...
- 11
1 vote
0 answers
16 views
Can I use multi armed bandits to optimize how much both algorithms are weighted when creating a composite score?
So, I'm aware that multi-armed bandits are great for evaluating multiple models and from what I understand, it is mainly used to pick a specific model. I would still like to evaluate two models but I ...
- 11
0 votes
1 answer
130 views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company. Beforehand we want to test this new price list in several countries. A/B test cannot be conducted here ...
2 votes
1 answer
535 views
Feature selection for two seperate datasets
Currently, I'm doing research with experimental data. The data comes from two experiments with two slightly different tasks, but with the same setup in a VR environment. Both experiments were done ...
- 21
1 vote
0 answers
113 views
Practical constraints in A/B testing
I saw an article about an A/B test that google had performed way back. They wanted to decide what shade of blue a button should be and how that affects click-through rate. They divided users randomly ...
- 187
0 votes
0 answers
103 views
What is the right approach to bucket users for algorithms with different coverage for A/B testing
I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide ...
- 571
1 vote
1 answer
58 views
paired t-test shows no difference between median and Wilcoxon test p value shows that there is a difference between median values ? How to interpret?
I have a dataset. I wanted to do paired t test on it. So I carried out normality test and it showed that it does not follow normal distribution. So I used Wilcoxon test in place of paired t test. The ...
- 151
5 votes
1 answer
137 views
experimental design in R project
I want to know of any repositories that contain complete experimental design in R covering basic test and analyses? I want to take a top-buttom approach to learn step by step through a real project ...
- 191
1 vote
0 answers
27 views
How to think about design of experiments in the context of more complex machine learning [closed]
I have experience with multifactor DoE, but in the context of optimizing treatment of a single or or a small number of populations. Are there any articles people recommend to help get my head around ...
- 11
0 votes
0 answers
210 views
Comparing the two feature sets
I am working on a classification of two feature sets derived from a dataset. We first obtain two feature matrices derived from two feature extraction methods. Now, I need to compare them. However, the ...
- 101