How to do Product Development the right way, the Experimentation way.
As a new Experimentation Engineer at Miro, I discover a world where curiosity and data come together, revealing endless possibilities 💫. Sounds a bit cheesy, but bear 🐻 with me.
Here's an intro to Experimentation, and why I'm excited about it. This is supposed to spark your curiosity, and is by no means a comprehensive guide to this topic.
To follow our regular format, here are some problems we'll be discussing:
Problem 1
Your team is working together on how to make more users sign-up. One idea is to implement Sign Up with Twitter.
How do you decide if this is a good idea?
Solution
You run an A/B test. You implement the idea, and you show it to half of the users, and the other to the other half sees the current setup. Then, you compare the results.
Problem 2
Your team finished migrating from MongoDB to PostgreSQL. How do you safely release this change to production 🛟?
Solution
You use a Gradual Rollout. You slowly release the change to production, and you keep an eye 👀 on the metrics.
Problem 3
Your team is working on an integration that allows users to pay with PayPal. You hope that PayPal will not suddenly break the integration, but you also want to make sure that you can turn it off at any time if something goes wrong.
Solution
You use a Kill Switch.
Common ground
Now that we introduced some problems and briefly discussed the solutions, let's see what they have in common. They are all data-driven, control at runtime, democratised solutions. From now on, let's call all of these: feature flags or experiments.
What is a feature flag?
A feature flag is a way to validate an idea based on data, a way to slowly release changes to production while keeping an eye on the metrics, and a way to implement features that allows you to turn them on and off without redeploying.
A feature flag is in its simplest form, an if/else statement.
Some examples of feature flags:
- A kill switch is a feature flag that allows you to turn off a feature in production ☠️
- An experiment toggle is a feature flag that allows you to turn on a feature in production and measure its impact 📊
- A release toggle is a feature flag that allows you to turn on a feature in production after it was deployed 📤
- A permission toggle is a feature flag that allows you to change the experience certain users get 🔑.
- An ops toggle is a feature flag that gives you control over operational aspects of the system ⚙️.
Now, the more you use feature flags, the more you'll see that they require quite a few things to change within your company. You'll need to teach people how to use them. You'll need to teach them how to measure them. You'll need to teach them how to compare them.
You'll need to create a culture where people can safely fail.
You'll need to create an Experimentation Culture.
What is an experimentation culture?
An experimentation culture is a culture that allows everyone to run experiments, compare them in a fair way, and make decisions based on the results. It relies on readily available metrics, and a common way of implementing ideas.
A culture of experimentation allows you to create a safe environment where people can fail.
Embracing failure and learning from it is the key to progress 📈.
There are three pillars to an experimentation culture:
- Data-driven decisions. Decisions should be based on data, not on gut feeling. Data should be readily available, and everyone should have access to it.
- Easiness to run experiments. Everyone should be able to run experiments, and compare them in a fair way.
- Safe releases. Changes should be released gradually, and you should keep an eye on the metrics.
Let's talk a bit about these in order:
Data-driven decisions
In order to make the best decisions, you need to have some metrics. Metrics are a set of dynamic values that you want to keep an eye on 👀.
They are set by your business, and they are at the core of your business.
Now, usually there are 3 types of metrics:
- Decision metrics. These are directly linked to the goal of the experiment.
- Guardrail metrics. These are metrics that you want to keep an eye on, to make sure you're not breaking anything.
- Exploratory metrics. These are metrics that you're not sure if they're important, but you want to keep an eye on them.
It's important to have a unified way of measuring stuff. Not only that, but your platform needs to already offer a set of metrics that are at the core of the business ❤️.
With an Experimentation Culture, data is king! 👑
Easiness to run experiments
In order to run experiments, you need to have a way to implement them. Everyone in your organisation should be able to set up an experiment, access and read the metrics, and compare the results. This also means that there should be no friction with your engineers while implementing the experiments. This is not always the easiest thing to do, but it's worth it.
Safe releases
It is important to decouple the release of a feature from the deployment of a feature. This way, you're allowing your team to merge code to the main branch, and then release it when you're ready. Also, this way you also enable the team to release a feature gradually, and keep an eye on the metrics.
All of these will give your team the confidence to release features, and a safe environment to experiment.
Was this convincing?
Now, I know I talked a lot in this post, but there's even more to come 📚. We'll talk about different feature flags, implementation details, and event how to build an experimentation platform.
This post was meant to get you interested, or at least intrigued about the idea of an Experimentation Culture. 🤩
Experimentation Platforms as SaaS
Before I go, here are some platforms that might get you started if you can't wait for the next post:
- Unleash (this one is Open Source 👍)
- GrowthBook (also Open Source)
- Optimizely
- Split
- LaunchDarkly
Congrats 🎉
You made it yet through another article. Thanks for spending the time 🕰️ reading this, and hope it provided some sort of value, since time is a limited resource. Use it wisely. 🙏
Want more?
If you want to know more about this, or something doesn't make any sense, please let me know.
Also, if you have questions, you know where to find me… On the internet!
Be kind to each other! 🧡
Over and out
Time for something new! Here is an intro to Experimentation 🧪. What is it, and why I'm excited about it 🚀: https://catalincodes.com/posts/experimentation-this-is-the-way #Experimentation