If you are in software development or technical operations of any kind, you are probably familiar with the term “Roll Back”. For those of you who are not in either camp mentioned above, allow me to provide a brief definition:
Rollback: To return the system, software, or database to some previous state (hopefully the state prior to your deployment).
Back in August of 2010, our teams at Fellowship Technologies started on the long road to continuous delivery. Continuous delivery is the notion that you engineer your software to allow for a continuous deployment through all environments, up to and including Production (with little to no manual intervention). It is also the notion that your engineering practices are so solid that each change you make to the system is backed by a series of automated tests written by the development team. This of course provides confidence and empowers them to make changes to those creaky parts of the system that no one “used to” want to touch.
In the past, our releases were huge chunks of software that required multiple levels of manual testing which of course took an in-ordinate amount of time and actually distracted us from obtaining the real goal; getting our software from our developer’s machine to Production once it is deemed “done” by the team. Our releases were flat out painful and we knew that we had to change. During our long “mock deployment” meetings, inevitably, we would discuss the infamous roll-back strategy. This strategy exists simply due to the low confidence of your team in their ability to deliver software. That confidence could stem from their own engineering practices to the actual environment or platform that they were deploying to. At any rate, the team owns those challenges and our teams set out on a course to remove the reasons that it so painful to deliver software.
We practice Scrum as our framework for software development. Scrum’s job is to show our teams why we can’t obtain our goal mentioned above. Scrum was doing its job well and our team finally decided that it was time to focus on this problem. I think we could all agree as a team that we are no where near where we want to be, but that is what excites me about our teams. They have the drive to get better. While it never happens as quickly as we all would like, we learn each step how to do something better and apply that to the next challenge. Scrum is empirical in nature, meaning that we learn from observation and experience. Today (Aug 9) is our 1 year anniversary to the turning point in our engineering practices. This post is to celebrate all of the team’s accomplishments over the year. I am so grateful to be a part of a team that embraces continuous improvement and at the heart of our decisions has the desire to serve our customers as best we can.
The path has been no bed of roses; no pleasure cruise. We have had our challenges, dis-agreements, and flat out failures (learning opportunities). However, I can say that I believe our confidence factor has improved by a factor of 10. Our teams are not afraid of any challenges nor do they shy away from those large “learning opportunities”. We continually improve in our ability to work as a team and learn to navigate the mine field of doing things the right way and doing what is best for our customers. In the end, we have removed the term “roll-back” from our deployment vocabulary and have replaced it with the self-confidence of “roll-forward”. We are confident that our engineering practices will continue to improve, thus helping us catch more of those “gotchas” up-front in our sprint cycles than when we get to Production.
Congratulations to our teams, thank you so much for your attitude, aptitude, and drive to be the best development team on the planet. You deserve it, our organization deserves it, but most of all, our customers deserve our best. Roll forward!