Paul, @pkiddie wrote a great blog about how we’ve made our customers happier – and engineers using New Relic. We were also given the opportunity to present some of these ideas at a recent New Relic User Group meetup in London.
Now with New Relic ticking along nicely, and dashboards all up next to our product teams so they can see the error rate, revenue and response times. We wanted to make sure we kept up to date with the latest features from New Relic. (Those updates have been arriving up to twice a month for the .Net APM agent we’re using.) As a cloud hosted platform, New Relic is continuously updated and improved, and we want to get the most out of our investment in it by keeping up to date with each release. It also makes our product teams happy when they get to play around with the latest new features.
A common question we were asked at the meetup was how we achieved this in a controlled manner.
We deploy our code using a mix of methods Thougthworks Go, Microsoft SCCM (System Centre Configuration Manager), and Chef. So when it came to work out how we test and keep the New Relic APM .Net agent up to date it wasn’t obvious as the different product teams have different methods.
The approach we settled on is to let our web front end development team, known as Tango (Paul, who championed New Relic, is a member) trial the latest candidate New Relic APM agent. They have the benefit of already being a Continuous Delivery team; using Martin Fowler’s Blue\Green approach to offline deployment with fast switching -and rollback if required. They can deploy the new agent to the Blue Slice, then perform the automated load balancing switch while monitoring how the new New Relic APM agent performs. If there are issues it’s easy to switch back to the original Green Slice, with the older agent version on that we know works.
Once the Tango team is confident in the new New Relic agent, we combine it with work our server team does every month to update Windows with the regular patches. We use SCCM to deploy these patches, and after some debate we decided to use SCCM to do the same for New Relic. It makes sense to combine the New Relic agent update with the Windows patching as updating New Relic involves a web server, (for us IIS) application pool recycle, which is almost as labour intensive as the server reboot we have to do when applying the patches which have to be done out of office hours.
To me this all sounds straightforward – but I’m assured by Bill Lapcevic @billap from New Relic, that it’s a model of Best Practice; and something we should tell the world about. We’re happy to do just that!