Posts filed under 'Change Control'
If you are a large IT organization, most likely there is a central IT organization and some IT in each business unit. Usually the relationship between the two organizations is somewhat strained. Recently I was on a panel with a CIO from the DC area and he mentioned that one of the best things he had done was to have one of his reports be part of each business unit (have dotted line to the BU head of IT). This increased information flow considerably.
When I think about this problem it is similar to the GM and the Auto workers Union (UAW) settlement. GM was asking for some really tough concessions from the UAW, so they brought a UAW representative inside GM and gave him access to all the financial infromation. He was not allowed to communicate this information, but he could see how serious and real the situation was. This helped GM and UAW engage in meaningful dialog.
Another CIO I met mentioned that if Central IT and the Business Units could see what each other was doing that would go a long way to build trust. If there was some way to have autonomy but visibility that would make the organization really efficient.
November 10th, 2006
I was with a CIO for a large public utility, where several of the IT systems control real world infrastructure. He made a very interesting point — universally what people in IT are doing to reduce cost is to automate all the manual tasks. While this seems the correct way of doing things, one of the big dangers of this is cascading changes is causing cascading failures.
Pre-automation there were several manual steps which inherently created barriers for the failures to cascade and also in some sense partitioned the infrastructure. Several examples came out
- Active Directory/DNS: since the directory auto-replicates, if you make a mistake it propogates relatively quickly
- Production and Disaster Recovery: auto-sync between these two can bring down both
- Network: this is the classic because routing changes propogate quickly
- Any clustering solution
We had an interesting discussion about what to do in this case. Clearly you want automation, introducing a human in the loop is not an option in many cases. But how do you solve this problem?
One of our colleagues at a large minufacturing facility had solved this problem in an interesting way. He took one node in a cluster or active directory and used to keep it disconnected!!! and the manually connect it once in a while.
That was very interesting because he had figured out a way to technically enforce a change window which opened by him connecting and dis-connecting to the network.
November 9th, 2006
After hunting for vendor independent material (see Part I) it was time to look at the vendor material. So I typed in “change management” into google and scan through the sponsored links … the following seemed relevant
- IBM Rational: the link takes us to Clear Case which is a source code management system.
- Sunview Software: lets you define workflows for different change types and then users to use them
- www.seapine.com/SCM: development tool to streamline software development
- www.bmc.com leads to a form which leads to a link to a whitepaper again about defining the business process and keeping track of a change request/project as it completes various phases
- www.Pega.com seems to be process definition and automation
- Solidcore: track and enforce changes on systems
- Opsware: provisioning system to push out new software to endpoints
- MetricsStream: quality management process
I refined the search to IT Change Management and the following other companies showed up:
- nLayers (EMC): application discovery and mapping tool for cmdb
Okay, so if you were looking for change management you could buy a software development system, process automation/ticket system, provisioning system, change tracking and enforcement, discovery and mapping tool. There seems to be no relationship between them but their marketing messages seem identical: conrol change, follow process, increase performance, be compliant … blah blah blah
So which one makes sense? (to be continued …. in part III)
November 9th, 2006
I was at a large retailer with stores in Europe and US. All the stores connected to a central transaction processing system. The setup was similar to the one in the picture below (which is from AJB software — no relation to the retailer) So all the stores connect to a system like RTS
Well it turned out that one of the software on these machines had the capability to download and self-update. The retailer was not only unaware of this but certainly believed that no such thing could be happening…Their normal change procedure was to copy every single file and keep several versions of it before making changes.
But it was happening … and did cause downtime every now or then … which was in the past attributed to flaky behavior. Do you have software which can auto-update in production: anti-virus, windows update, management packs are some examples where this commonly occurs.
November 8th, 2006
There seems to be a lot of people buying or upgrading there change management system. I was discussing with some of them how exactly do they go about this. I decided why not try and go through the process myself and see where we would end up.
So as s a first step I decided to go search for change management, service desk etc and see what articles came up …. the following I thought were interesting (of the links that were not not blatantly vendor specific)
The above were great exposition of the problems associated with change management or why you needed change management. But except for the first presentation by the gentleman from TCS. It was not clear what do about a change management system.
What else could we search on? Any insights and additional links would be great. I am going to attack this next from reading the vendor material: Part II: Exploring more vendor specific content (will add a link once I make some headway).
November 7th, 2006
I visited one of the larger (top 5 by revenue) wineries in Napa. And heard a very interesting story. Apparently most wines these days are made from grapes from a couple of places if not more. And the law requires that you have to keep track of exactly what volume of crushed grapes has done into a particular barrel, and where did the grapes come from etc. This is done with “Grape Crush Management System”.
If the wineries don’t keep track or for some reason the data gets messed up or the system goes down, they can sell the wine as $2 generic wine in a low end store. So this is very important.
The grape crushing lasts only for 2-3 months of the year. So what a lot of wineries do during this period is first to take the system off the network … so no one can log into it. Second all login’s except like the COO are disabled. You can only access this from a room which is under lock and key.
All because they don’t want anyone to make any changes during this time period.
November 5th, 2006
There was a lot of hype and hope when six-sigma was the “thing”. Now ITIL seems to be the “thing” and there is a similar kind of hype associated with it. There was a study couple of years back (I can’t find the link — will post it when I do) which compared what organizations adopted six sigma well and where it did not go anywhere. Jack Welch also talks about the same theme in his book.
What they found that a lot of companies did they following: bring in consultants, and they will tell us our six-sigma strategy. The consultants who were knowledgeable in most cases came and prepared a plan. Then the top management spent a lot of money implementing this plan. Usually most of these implementations stalled and failed to realize benefit for the companies.
The other way of doing six-sigma was to understand the essence of it: how do we drive variability out of a repeatable process. And then ask managers if this was applicable to them. That is did they have repeatable processes and if so how could they reduce variability. This approach was way more effective. One it got buy in from the grass roots and created small success stories, which had direct impact to the top and bottom lines.
I see the same kind of struggle with ITIL. There is industry pressure to do something about ITIL in your companies. What is the essence of ITIL? How do you get buy-in from the grass roots in your company? How do you create the short wins?
Those are the critical questions.
November 4th, 2006
In Tipping Point, Malcom Gladwell talked about how they brought crime under control in New York City. They began arresting people who were jumping the ticket turnstiles and booking them on the spot. How would this help with anything … people would have thought? But it so happened that several of the people they arrested were responsible for other larger crimes in the city. And slowly but surely this improved law and order dramatically.
One of the things I see organizations struggle with implementing change management or ITIL in general is where to start. Usually organizations emark on large projects which take a long time to realize the gain.
One thing we can learn for the NYC example is that we should begin monitoring the small infractions (unauthorized changes) in the infrastructure and it will have large impact on service availability. As some of those small infractions lead to much bigger crimes!
November 4th, 2006
Oracle’s “unbreakable” campaign is mostly mis-understood. To get enterprise customers to commit to Linux, Oracle promised them that if any bugs came up they would fix them within the SLA provided. Oracle had a few thousand (speculation) programmers and the way it worked was that Oracle wrote the code, give it to customer and provide the fix to RedHat which pushed it out.
So Oracle providing support for RedHat is not something completely alien. Whether their motive is really to provide support or this is first step in a multi-move chess ending to buy RedHat only Larry knows?
November 3rd, 2006
This picture is off the whiteboard of a the person responsible for ITIL implementation at one of the largest utilities in the US. I found it to be a simple and succint explanation of how things worked and inter-operated with each other.
Click on the picture to see animation about how change approval, staging & test, release management, configuration management and CMDB work together
November 1st, 2006
Next Posts
Previous Posts
