Do not perform simultaneous tests
Posted: Sun Feb 16, 2025 4:41 am
Having little patience when testing a hypothesis
Trying a variation for a short time is a hasty attitude: you will have to go back to the beginning and start all over again.
Do not rush into making a decision or change, as any misstep can result in wrong changes and future harm.
When given enough time, it is possible to perceive whether the differences found were truly consistent over time or just caused by an external event that altered the results.
Testing one variation for a week, locking it away, and testing another for teacher data 7 more days is a terrible idea. A lot can change from one testing period to the next, and the analysis performed would simply not be reliable.
It is best to run the experiments simultaneously. This way, all interactions occur in the same period and the scenario is the same for both variations.
Testing many variables at once
You might think that testing too many variables at once will increase your chances of optimizing your conversion rate or help you save time, but that's not a good idea.
If you change more than one element per test, how can you know what actually influenced the conversion? Instead, it's best to test one change at a time.
Trying a variation for a short time is a hasty attitude: you will have to go back to the beginning and start all over again.
Do not rush into making a decision or change, as any misstep can result in wrong changes and future harm.
When given enough time, it is possible to perceive whether the differences found were truly consistent over time or just caused by an external event that altered the results.
Testing one variation for a week, locking it away, and testing another for teacher data 7 more days is a terrible idea. A lot can change from one testing period to the next, and the analysis performed would simply not be reliable.
It is best to run the experiments simultaneously. This way, all interactions occur in the same period and the scenario is the same for both variations.
Testing many variables at once
You might think that testing too many variables at once will increase your chances of optimizing your conversion rate or help you save time, but that's not a good idea.
If you change more than one element per test, how can you know what actually influenced the conversion? Instead, it's best to test one change at a time.