On Monday this week, the Prometheus authors have released version 1.0.0 of the central component of the Prometheus monitoring and alerting system, the Prometheus server. (Other components will follow suit over the next months.) This is a major milestone for the project. Read more about it on the Prometheus blog, and check out the announcement of the CNCF, which has recently accepted Prometheus as a hosted project.
In just a few years, Prometheus has gone a long way from its beginnings as a pet project of Matt T. Proud and Julius Volz to a widely adopted monitoring system with an amazingly active community of more than 300 contributors and a user-base spanning many different organizations, including companies like CoreOS, DigitalOcean, Docker, Ericsson, Percona, and Weaveworks. A pivotal midpoint in the history of the project was the public announcement in January 2015 in an earlier backstage blog post.
In this article, I would like to reflect a bit on the childhood of Prometheus at SoundCloud, and on the benefits a company can gain from developing projects in the open under a free software license.
SoundCloud has a very liberal open-source policy. We have published a large number of projects in our GitHub organization. Matt and Julius, who had joined SoundCloud as engineers back in 2012, knew from the beginning that Prometheus would have a much larger scope than any of those projects. When they suggested that SoundCloud should invest in the development of a completely new monitoring system, the resistance they had to overcome was not so much related to the open-source character of the project but to its scope. After all, why should a music-streaming company with a relatively small engineering force dare to fundamentally change the way open-source monitoring is done? It was a bold move, indeed. But it was also the time of #monitoringsucks, and no open-source effort in sight to make a real dent. So we took the torch and went on our Promethean mission, cautiously avoiding the eagles.
In hindsight, we know that it has paid off. For one, SoundCloud has just finished the quarter with the highest availability in the history of the company — which was also the quarter with the most features and the highest complexity the site ever had. This tremendous accomplishment would have been impossible without a novel way of monitoring our growing microservice architecture — the way enabled by Prometheus.
As the main source of early contributions to the project, SoundCloud was also the earliest adopter. A nice strategic technical advantage, which – naively thought – could have been so much larger if Prometheus had been developed in secret. However, that would ignore the overwhelming advantages of an open-source development. First and foremost, without the countless contributions from the community, Prometheus would not be what it is today, and the technical advantage would be much smaller. Kickstarting a successful open-source project that is then joined by many other parties is a good way for a small company to get something going it could never have accomplished on its own. The SoundCloud open-source policy states “free bug fixes from the community” as one of the key benefits of open-sourcing projects. In the case of Prometheus, that’s quite an understatement, to say the least.
Tech credibility and the resulting impact on recruiting is another important benefit. Excellent engineers like to work for companies with excellent projects, and they like to work on those projects in the open, where they benefit a whole community and provide visibility for the engineers, too. And then there is a perhaps surprising side-effect we can observe by now with Prometheus: If an open-source project is popular, some of your new-hires will already be familiar with it, resulting in a smoother and shorter onboarding.
Prometheus has grown up to a real titan. We are eagerly looking forward to the further evolution of the project and to what humankind will do with the fire that was given to them.
Björn is a production engineer at SoundCloud. He joined the company and the Prometheus project in 2013 and has been a Prometheus core developer ever since.