Product and Tech
How we average 20-40 deployments a day. Why this helps us to build the product in a way that is both fast and stable. And why the key is empowered developers who merge their own code to master.
About the author: Oliver Stenbom is a DevOps Engineer and Team Lead in the DevOps Team.
The goal of Mentimeter’s Product Department is to ship a product that our users want in a fast and reliable way. The DevOps team (my team), exists to make it as easy as possible for us to do that! We strive to keep developers as close as possible to our users in production. That way we can quickly respond to feedback and move faster than our competitors. Let me explain!
In more “traditional” / old school software organizations, it was common for developers to work on big projects for long periods of time. You launch the thing. The thing might look like it works on an engineer's own laptop, but it often doesn’t work in production. The thing has a bunch of bugs, and requires a tonne more work to fix. These kinds of projects are often over-time and over-budget.
For a while now this has been seen as a super flawed way of working. Tech organizations realized some time ago that the better way to build a product is to integrate changes early and often - and to iterate as you go.
That’s exactly how we work at Mentimeter.
Our developers commit changes to master almost on a daily basis because it is important to make sure that your code is in sync with everyone else’s. With 60 people working in the same code base - there can be a lot of modifications from one day to another. With so much going on, you can’t work on something in isolation for a month and then expect it to immediately work with all of those changes. The ground you are working on will have shifted a lot, and the assumptions you made when you set out might no longer hold!
While some product organizations are happy to deploy once a week, or even once a month, we tend to deploy somewhere between 20-40 times a day. Working this way allows us to work with confidence and means we can consistently ship a stable product.
Because we do lots of smaller iterations on the code, rather than fewer big changes, it means that when something is shipped and breaks something - which happens! - the impact isn’t huge and can be easily fixed. Or we can just revert to a previous version. So smaller, more frequent iterations are more desirable. It allows us to move fast and move safely.
Then there is the question of who is going to do these 20-40 deployments a day? We must have a huge operations team who are rushed off their feet with that many deployments. Right? Well, no. We don’t even have an operations team. Instead, we have a department of entrusted and empowered developers who ship their own code to production.
And I haven’t even told you the best bit: the whole process takes 10 minutes and involves pushing a grand total of two buttons. Designing this process to be as easy and efficient as possible is how we in the DevOps team encourage developers to ship frequently, which is what helps us to build a better product.
Empowering developers to work in this way also means that when something does go wrong, usually the solution is much more readily available. When the code that you deployed breaks something in the product, the diagnostics are almost always more straightforward because you know your work better than anyone else.
There is a sense of ownership that we foster in our developers. A sense of self-leadership. You, the owner sitting on most of the context are best placed to see and solve any problems related to it. And that’s what we empower our developers to do.
It is the same with testing. We don’t have a QA team. Our developers build and run their own tests. This is all about designing a product organization that doesn’t hold developers back. You shouldn't be blocked by a QA team or an operations team. At Mentimeter you do it all yourself. And in this way you don’t just write code, you help to build a product. You own the success of your work at every step of the way. That’s what is so exciting.
Like every enabling team at Mentimeter - from the API team to the Design System team - we have to ask ourselves: Do we want to create an organizational setup that holds developers back, or do we want to create one that empowers developers to work quicker? To work better? To build a better product?
The work is never completely done. But we are always going in the right direction!