I was supporting a major global deployment for a large multinational insurance company that supplies insurance, annuities, and employee benefits programs when they asked me to pull together a “War Room” for their multi-million dollar program. Why do you ask? It’s because the original deployment of it failed numerous times causing the company to spend additional tens of millions of dollars to try again. Failure was not an option for the executives running the project this time. This story probably sounds familiar to many of you that support testing, deployments, and rollouts of high profile programs at numerous Fortune 500 companies in the US. The lesson here is don’t be like Samsung, don’t skimp on testing and put the right processes, procedures, people, technology and infrastructure in place to be successful.
My initial role on the project was to be a Senior Project Advisor to assure the continued viability of the in-flight projects and that best practices were adhered to at all times. I also was to identify potential risks and issues on projects, both business and technical, and recommending mitigations as required. But it soon morphed into additional responsibilities because of the lack of leadership in certain areas of the program, one of them being the deployment and support of the application during the testing phases of the program.
It was a huge cloud deployment that had teams working in three different continents (the US, Europe, and Japan). The coordination effort was huge because of the time zones, language differences, culture clashes, and technical capabilities. No one in the company had the experience of deploying a solution in the cloud, let alone understanding what it took to test, manage or operate the solution. So they needed help and a lot of it.
Because of these issues and others that I don’t need to get into in this article, I recommended and put into place a War Room to create a heightened level of support for deployment and environments and to promote the real-time collaboration between all affected project teams (Global IT Development, Testing, Business, IT Organization) with the Deployment and Support team. I also thought it would allow us to effectively facilitate real-time communication activities to address and manage Issues / Incidents and to simultaneously centralize focus, increase awareness and aid in our decision-making. It did all that and more. It was a resounding success.
The company built a location for the War Room that would stay in place during system integration testing, user acceptance testing and through a go-live monitoring period. Because of the global nature of the project, it was 9-12 hours a day, 5 days a week (sometimes on the weekend) assignment. Shifts were set up in advance and we had infrastructure monitoring set up as well for 24-hour coverage. We used tools and templates to manage the action items and issue logs and posted progress and reports on the walls to track progress. We implemented daily checkpoint meetings and issued a daily deployment status report that was quite comprehensive. We even had contingencies for late night support in case of issues or deployments that were needed at the last minute. We had everything covered.
My plan for communications was to over communicate. We wanted everyone on the same page because problems will still occur even though you tell everyone that needs to know what is going on. All in all, we had over 17 different communication templates outlined in our War Room Plan. We left nothing to chance. We even built a RACI chart and processes for all program teams. We outlined the key information to publish; the layout of the War Room; key contact information; and created templates of all of the reports we were going to disseminate to the program team. The planning was quite comprehensive and we were very proud of our environment readiness diagrams.
One of the issues the company struggled with was the time to deploy a new build. When I arrived on the project, the team was taking over 5 days to rebuild a new environment (yes – that is not a typo). Their infrastructure team would take forever to set up their ETL and database instances in the cloud. I told them this process should take hours if not minutes using scripts and automated processes. The deployment and support teams laughed at me and the infrastructure team was not convinced, but the executives were, and they let me work on the problem. They had to let me try. There was no way in the world the program would have hit their key milestone dates if the redeployment of a new build took that long. This is a classic DevOps problem to solve, but the technical advisor on the project was not sold on DevOps and we did not have the time to build the infrastructure. So we had to come up with a different solution that involved manual and automated steps in bringing the time down to a more manageable level.
With most of the cloud solutions out there today, you can build an environment from scratch using pre-built scripts (creating subnets, servers, firewalls, applications, etc.). And that is exactly what we did. Our goal was to get the time down to seven hours and we shattered that. By the time I left the project, we were under four hours. That is pretty good when you consider that the team would not buy into DevOps and you still had four manual steps in the deployment activities (like kicking off the scripts to build a new release, manually configuring the applications and performing sanity and readiness checks). Assuming 10 hour days, we went from fifty hours to four, which is a 92% decrease in time.
One huge point to mention in all of this is that the politics of the company and related team members can derail the best-laid plans. No one wants bad news, and with a War Room setup, it can deliver some incredibly painful bad news very quickly. And if your organization structure puts your group into the folks you are reporting against, it will not work. Why? They have their own agenda, and it is to put themselves in the best light possible. During the project, I can’t tell you the number of times we had to rewrite a report before we posted it.
As I mentioned previously, setting up a War Room can be very beneficial for global deployments. Author Steven Shaker said in an article about War Rooms: “It is the people, their interactions and the total process, which is core to its character and attributes”. Here are some lessons learned from my project:
- Report to a group outside the program to avoid the political ramifications of the program/project
- With any “bad” news, make sure you get it verified by multiple sources before posting it
- If you have language barriers, make sure your team members can speak/write well in both languages to avoid confusion
- Take the time up front to set up a DevOps framework to streamline the deployment process
- Build the War Room with the capability to hold large meetings since people tend to flock to the room during important discussions
- Document the resources, processes, procedures, technology, reports, and communication plans in advance and in detail
- Move to a digital War Room format for convenience and necessity. Too much paper is wasted on a daily basis
- Keep trying to improve upon the processes and technology you use throughout the campaign
- Make sure your team understands the political ramifications of any communication or report
Don’t just take my word on this, check out the YouTube Video from Google: https://www.youtube.com/
Please post any comments or suggestions.