ITIL: The Change Management Tax Shelters
… based on a true story at a Fortune 1000 company
From the wikipedia:
… a tax shelter is any organized program in which many individuals, rich or poor, participate to reduce their taxes due. However, a few individuals stretch the limits of legal interpretation of the income tax laws. While these actions may be within the boundary of legally accepted practice in physical form, these actions could be deemed to be conducted in bad faith. Tax shelters were intended to induce good behaviors from the masses, but at the same time caused a handful to act in the opposite manner. Tax shelters have therefore often shared an unsavory association with fraud.
In most organizations when you want to make a change you need to fill out a change request form. The change request form states what needs to be accomplished, but does not concern itself with how the change is to be carried out. It usually contains when the change should be made and to which machines in the infrastructure. The change requests then go to the change
Again from the wikipedia
Change requests typically originate from one of five sources: (i) problem reports that identify bugs that must be fixed, which forms the most common source, (ii) system enhancement requests from users, (iii) events in the development of other systems, (iv) changes in underlying structure and or standards (e.g. in software development this could be a new operating system) and (v) demands from senior management
Almost every IT administrator I have met hates to fill out requests for change. Several of them see this as a TAX that they have to fill in addition to working 24×7 to get the work done. Now most administrators are smart people and they either created or found their own tax shelter: the Emergency Change.
When a change is labeled as an emergency change most organizations allow all the procedures of filling out a change request and approval to be bypassed, hence avoiding the change management tax. Just like IRS’s tax shelter some of which are perfectly legal and have legitimate uses, the emergency change is required for the proper functioning of the infrastructure; but like several illegal tax shelters it can be used and abused.
Recently a large retail company had outsourced their e-commerce website to an managed service provider (like IBM SO, EDS, Verizon etc). The outsourced service provider had very strict change management procedure, which they had developed over the past several years to ensure highest availability and uptime on the revenue generating sites. But ofcourse there was the change management tax shelter: the emergency change. After about an year of signing the contract, one of the outsourcer’s executives reviewing the retailers account found that 70-80% of all changes presented by the team at the retailer were emergency changes with no documentation or approvals. This was clearly and abuse of the tax shelter.
He called up his counterpart at the retailer and explained his analysis and why their performance fell short of the SLA. The VP of e-commerce operations was a reasonable person and admitted that he was between a rock and a hard place. His development team had revolted that if they had to follow the process and pay the change management tax they don’t have enough time or resources to get he site up before the Christmas season and he had let his operations team use this tax shelter in the contract.
What should we do? They could not come up with a solution at that very moment. Now the person at the outsourcer wanted to help his customer the VP at the retailer. So he got his team together and said we have got to figure out a way to this problem, because we look bad in front of the customer by not meeting our SLA and we are loosing money on the account also.
His team worked hard and came up with requirements: What they really wanted was something that could let the customer just do the change and then pick up all the information for the change and create the necessary documentation. Thus from the customers perspective they got rid of the tax but from the outsourcers perspective they still got the documentation needed and if there were exceptions that would be flagged. If someone could figure out a way of doing that with the IRS we would all save tax and the government won’t go into deficit.
Once they knew what they wanted they began looking into the market for solutions. The first one they encountered was from Mercury (Kintana), now part of HP. This solution would fit well with the development system that the developers were using. But it only took care of changes from the development system; if someone used some other system it would not work.
Next they looked at Bladelogic & Opsware, both tools used for pushing out the changes. Now these tools could detect if a server had been changed and what the changes but they had no idea who made the emergency change, when it was made or how it was made. The integration with their change management system was also not easy. Both these systems could provide some of the documentation required for the process, but not all of it. Then they came across Tripwire. Tripwire could also be run periodically and it would detect what had changed, but it could not provide who made the emergency change (which was a problem, because it was important to show that the change had been made by the retailer) , also there was no SLA on when the change was made and when the documentation would be complete. The other problem they found with the above systems was that when they created tickets from change the volume of change was so high that the tickets created were meaningless.
Finally they looked at Solidcore. Solidcore could tell them what changes had been made, who (user) made them, what application was used to make the change, when the change was made. It could also connect back to the change ticketing system. Before the changes were put into the ticketing system, Solidcore clustered them to find units of change. This dramatically reduced the number of tickets created and also there was more meaning to each ticket.
The next morning the VP @ the retailer received an email: we have discovered a legal tax shelter for you: Solidcore and would like to discuss and mutually agree to discontinue the use of the illegal tax shelter: the emergency change.
Add comment January 18th, 2007
