Home / Amsterdam / The status of our pipeline

The status of our pipeline

One thing we have always done since we have a CI server, is to show a dashboard (wallboard) of what it is doing on a television. We are doing this, because it is important to know if a build is broken or it is overloaded with jobs.

When we started showing it on a screen the team was still relatively small and we were not doing any fancy things… just compile test and create the artefacts we later needed to hand over to ops so that they can deploy it to production. The screen showing the dashboard was placed on a stand on the floor and almost every developer could see it.

… And then we started to grow.

Make it visible!

Less developers could see the screen and every now and then the build pipeline would become in a ‘broken’ state. Sometimes unnoticed for an entire day. Something definitely had to change! So it was time again to take a step back and ask ourself why we want to show the wallboard. Quickly we realised that we are not really interested in what it is doing, but the most important things was, that we wanted to see if the pipeline was broken!

So the wallboard was nice, but did not really show what was really interesting for us as developers. At the same time, another company named Dynatrace, of which we are using their APM tool, was creating something to visualise things like the performance of a production environment. The device is called UFO and consists of two rows of LED lights which you can change the colour of, or do some animations like rotation and/or blinking on. The nice thing is, that it has a REST interface (developers love this)! So we got ourselves one UFO and wrote a script that every now and then gets the status of the pipeline from Jenkins, and based on that do an API call to the UFO. And since we can hang it from the ceiling, all developers can see the status at all times.

This resulted in detecting a broken pipeline earlier and making people aware of what is going on with our pipeline. What we also did, was to make use of the animations and colours. In our case we can have different states/visualisations:

  • Green – all is fine
  • Blinking red – pipeline is broken
  • Yellow/green rotation – something is being built
  • Blinking purple – something is ready to go into production

Since it also has two rows of LED light, we used that to show the build status of two major parts of our application. So one row is used for the platform itself (the backend) and one for the new website (the frontend).

But… after a while and just talking about the influence the visibility of the status of the pipeline had on the developers we made a ‘joke’ that it would be awesome if the entire room (or ceiling) would change colour. This however did trigger that after a short while, there were some extra light bought from the IKEA and together with Philips Hue we were showing the same status at different locations on the work floor. These lights have no animation however, but still shows the most important information:

  • Green – all is fine
  • Red – pipeline is broken
  • Purple – ready to go to production

In the photo below you can see the UFO hanging no the metal rail. The top row is purple and the bottom one is green. This is also reflected on the hanging lamp (green) and the standing lamp (purple). There are more lights, but they are not shown on the photo. One of them is behind the photographer and another one on his right side. Down the little hallway on the right there also one located behind the closets.

Different lights to show the status of the build pipeline

Lessons learned

The most important thing we have learned was, that in order to make developers feel responsible for the build pipeline is to make it very visible. You do not want that a developer needs to go to a CI server to see if he is allowed to merge his changes (you should only do that on a pipeline that is not broken). Everybody should be able to see it… always.

Also an occasional visitor should see it and ask questions about it… this includes anyone coming over for a job interview of course (see our jobs section for current job openings).

Besides that, it brings a nice atmosphere to the work floor.

Interested joining BVA Auctions?

Take a look to our vacancies

About Marcel van den Brink

Marcel van den Brink is a software engineer with 20+ years of experience in web development. He is passionate about the development process, CI, CD, metrics and getting the most out of a team.

Leave a Reply

Your email address will not be published. Required fields are marked *