Lessons from upgrading browser software
When IT recommended to the business that a simple PC software upgrade was needed, nobody thought it would be a major problem.
Within six months the software would no longer be supported. Of course, it was a large upgrade – the Head office and over 400 high street stores; a total of 30,000 devices would be affected, but the IT department was confident that there would be no impact on operational systems. The initial plan was simply to send out the upgrade links to users explaining the necessity of the upgrade and asking each person to follow the link and upgrade. It was clear early on that this was simply not having the desired effect. Even when managers were engaged and asked to oversee the implementation, there was an estimate of over 50% of the company not reacting to the emails.
The business change manager appointed knew that a much more structured approach where the software was ‘pushed’ to the devices would be necessary. He was also aware that this approach would not be universally popular.
First find the PC!
An early problem was identifying all of the devices that needed the upgrade. While it was clear for the stores, in Head Office there were a whole variety of devices, from laptops to cell phones that might be affected. It was only a year earlier that the IT department had implemented a similar company-wide upgrade of the operating system – so it was with some disappointment that the change manager found that no accurate record was available of what devices existed and where. (Authors note – the lack of configuration control of IT devices has been noted in other stories – it just seems too difficult to do, resulting in the repeated exercise of ‘first find the PC’!).
“We thought once we knew the devices the upgrade would be easy. It turned out to be quite the reverse”, reported the change manager. One of the key operational systems just would not work with the new browser. This system was so critical, all changes to it were controlled centrally by a specialist team. The sad thing was that these implications had been known for some time. As the change manager suggested, “it was almost as if the IT technical people in the know were too afraid to bring forward the bad news – we should have known earlier about this problem.”
The decision lies in the hands of the business
Every one of the functions in the operational application would be potentially be affected by the upgrade, and there was no way, except through a screen-by-screen check of identifying the implications. “After eight weeks – we could report was that it should work but with insufficient time for full regression testing the only way to test was to get access to the model office environments.” As in most businesses, this testing environment is extremely hard to get access to, and the team could not yet estimate how long they would need in the model office.
In the end, it was decided to go to the project board. In this business, this was a very well structured group empowered to make and carry through decision across the organisation. The change manager prepared and presented two options:
- Option 1: Stay with the unsupported software until regression testing was complete. Support ended within four months, but it would take 12 months to do full regression testing
- Option 2: Go-live and fix on failure. Operational risks were assessed and presented. These were very real risks including loss of income and reputational risk.
The board selected option 2.
A fix on failure approach
The change manager worked with his team to define the approach. The 80% most common transaction screens were identified, and this was the focus of error checking before going live in a selected store. Once sufficient usage and fixing had occurred then a national roll-out to the remaining 400 offices would be instigated. In the meantime planning for Head Office implementation, which involved the more complex identification of devices, would start in parallel. For the initial store implementation, the upgrade would be applied to all but two devices. These two devices would then be available to staff to use should there be problems in operation due to the upgrade.
The change manager had chosen to lead the briefings and be onsite during the week of the initial implementation to ensure he was there to help fix things if they went wrong. The implementation “mostly worked,” but there were critical actions, such as being able to print till receipts, which just wouldn’t work. It took tenacity on the part of the change manager to persuade the IT team of the all the issues that were being faced by the staff. It took two months to satisfactorily complete the store test implementation with all of the fixes that were required and the to-ing and fro-ing between the project and the IT team.
During this time, the upgrade built a reputation (often unfounded) as the cause of any problems which might occur on any system. “Oh, that is probably caused by the new software upgrade.” The change manager recognised that for the wider roll-out to be successful and acceptable, it was essential that a well-controlled and managed store-by-store roll-out be established, and that confidence in the roll-out process was regained. He insisted the second store implementation should occur in a particular location so IT could be closer to the problems. “I needed IT to see this, understand the issues faced and react quickly.”
After the early slow implementation, store roll-outs could be expedited with 100 stores implemented in just a week. The total implementation of all stores and Head Office took six months.
In reflecting back on the project the change manager identified a number of learning points:
- Decision-making around this had to be firmly in the hands of the business. Only they could decide what risks they were prepared to expose the business to. This was done well in this business.
- Sometimes getting IT to react to problems was hard. “My contact in IT was hard to engage with, but my feeling is that there will always be difficult people on a project – senior and junior – it’s just about finding ways of getting things done.”
And with respect to the Head Office implementation:
- We found we could not rely on the organisation chart in Head Office to work out who reported to who and where they sat to help us build up a picture of who we needed to engage with. So we started at the top of the organisation, talking with PAs and support staff to identify the people in the top three layers of Head Office. We opened up communications and consulted and worked with this group to agree on the roll-outs. This worked pretty well, but it turned out there were influential and high energy stakeholders in the lower levels that we missed. This did prove to be an issue. Just because somebody is junior does not mean they cannot significantly impact the project – something to look out for on future projects!