Process is supposed to be a Good Thing (tm), and SEI CMM Level 5 is supposed to be next to God.

But when I read Peopleware, it says that as an organisation "matures" on the CMM scale, it is more and more likely to take less risk, which of course means that they're not likely to produce any "killer apps" in the future.

My company (we're a high-end web application development company) is working towards a CMM Level 3 certification in the next 6-9 months. I was wondering if anybody would like to share their experiences with getting a CMM certification, and the problems you faced, and whether you think that there IS such a thing as too much documentation.

-- Anonymous, April 01, 2001

Answers

At my previous job there was an initiative to get on the CMM process, with the eventual goal of CMM 5. As we researched the process, we discovered that while CMM sounds great on paper, that's really all it is, just paper with no substance. For example:

1) Companies that achieve some level of CMM success (Levels 2 - 4) almost always lose it when a chief technical or management sponsor changes jobs, changes his mind or leaves the company.

2) CMM requires long-term and consistent support from upper management, which is of course an oxymoron. At company meetings, the CEO would proudly list CMM as a corporate goal; 18 months later there was no mention of CMM at all.

3) DMM doesn't lend itself to fast-paced business conditions. Imagine that it's 1 PM on a Friday. Your biggest customer calls and demands a change to the software you're selling them, and they need it by Monday afternoon. Of course, you tell your boss that there's no way to do it in less than a week with CMM-compliance; however, if you work the weekend and skip the CMM procedures you might just make it. Which will your boss choose?

4) CMM advocates like to use the Space Shuttle flight software (SSFS) as a shining example of CMM at work. Bad example! First, the SSFS team doesn't use CMM, they have their own (but similar) system. Second, developing software for the Shuttle is far different than developing commercial software: How fast would that new spreadsheet get written if EVERY change to a line of code had to be reviewed by eight people, as they do with the SSFS? Third, the SSFS has very well- defined requirements in place and locked down for each mission months ahead of time. Fourth, the SSFS is used on a stable and very well- understood hardware platform, not the infinite variety of PCs we deal with.

Well, I'm getting carried away a bit! As you can see, I'm not a fan of CMM. I view it primarily as a means for high-priced consultants to hold seminars and sell books, not as a process to improve software. If you want to improve software, use basic, solid engineering principles: Conservative design & analysis, good documentation and thorough testing.

MN

-- Anonymous, April 05, 2001

I've looked at CMM but not done anything with it. Notes:

1. Last minute changes are bad. If CMM prevents them it's good.

2. I think CMM allows use of electronic archives not just paper.

3. Peopleware (2nd edition) is contradictory on CMM:

"Organizations become more and more risk averse as they mature. An organization under the gun to demonstrate increased CMM compliance is not going to go looking for real challenge." (p. 190)

"The more strongly you believe in the meaningfulness of the [CMM] assessment the more inclined you are to take on ever more difficult work." (p. 191)

Maybe some empirical *data* could clear up which assertion describes the actual case.

-- Anonymous, April 07, 2001

In my experience, CMM is not a total waste of time. I think that there are a lot of misconceptions about the overhead generated by following CMM and the involvement required to make it successful. I have worked in groups with ratings from -1 to 3, and I much prefered the 3.

The first thing to remember is that CMM is about developing and using processes that are applicable to you and your work. If you are operating in fast paced situations, your process should be adapted to suit those conditions. If you are building life or mission critical products, the process will most likely be different.

If you are successful in defining your processes, you will find that it may take some degree of extra effort up front, but it will reduce rework and confusion substantially as well as ensuring that vital steps are not overlooked in the rush to get something out the door.

If you wish your company to maintain a high CMM rating, it is imperative that management support your efforts. However, it is still possible to achieve a high CMM rating within a group or subset of the company if that is something that the team values.

On the matter or risk and maturity, I think it could be argued that while a company that is more mature may take fewer risks than a less mature company, it doesn't limit them in the development of a 'killer app'. A company with a well developed process limits the risk it takes by ensuring that the necessary process and management is in place to complete the task. By having these safeguards in place, a mature company can actually take greater technical risks while maintaining a lower overall risk than a less mature company

You can have too much documentation and you can have too much process. The secret is doing enough to ensure that the needs of all stakeholders are met and no more. This is a difficult balance, and if you are going to miss the mark, I would err on the side of over documentation.

In the end, the important thing about CMM is not the CMM certification itself, but the discipline you have to apply to software development to achieve the rating. If you keep your eye on the development and use of process, the certification will come. If you keep your eye on the certification, odds are that your certification will be short lived.

-- Anonymous, April 09, 2001

I use CMM like this:

It's a guide, to show where to go and where to prioritize software improvement. My general rule in software for projects and team sizes I'm typically involved in (3-6 techs), anything internally that involves paper will fail. Small chunks of process are automated through good tools. A good tool or process is one where it's easier to do right than wrong (thanks Pat Haszard for that great little nuggest of advice). So, what are examples? Test Track Pro, www.seapine.com is a sensational success here for defect management, I've never seen such a simple, and useful improvement to software process improvement. Another is TogetherJ, www.together.com for java projects, it delivers on what Microsoft and Rational have only talked about: visual modeling and truly useful round-trip engineering. Suuuperb.

Other process-automation tools are version control, automated builds and automated unit tests for regression testing. Here of course I'm talking about CVS, Ant, and JUnit. Though to be honest, CVS pisses me off, I've spent an afternoon on it under NT with WinCVS and I've lost patience. Went to Visual Source safe and had it all going in 15 minutes.

-- Anonymous, April 10, 2001

My answer to this important question is twofold. The first concerns the general question of process (any level), and the second concerns the "godly" levels of process (levels 4 and 5).

Concerning process: software development is a complex biz. I teach software development process courses where I work, and I ask my students the following question before I start: "which is more complex: your car, or your word processor?" Most students - engineers all - answer "the car". They're wrong. If you need proof, look at the size of the respective owners manuals. The car's owner's manual is 120 pages on paper that's barely 7 inches by 4. Look now at your word processor's owner's manual. They're so big, the vendors usually don't even ship them on paper any more! They come to user on CD - sometimes more than one.

On an engineered product this big, some process is necessary and good. A poor process is probably better than no process, and a decent undocumented process definitely beats a poor documented process. Engineered products need to be built and modified in a method that enables the development team to create pieces, integrate them, test them, ship them, and feedback customer comments so improvements and corrections can be made. At one time, anarchy was not only OK; it was expected. In truth, it was part of the mystique of software development.

Systems are too big now to allow anarchy to occur. There's too much complexity, and too much risk of integration failing. In a business sense, there's too much risk. Software development houses have to make a profit; if a product never makes it out of integration, it can never make it to market. Everyone loses.

Process, then, is important. It's clear enough. And it's not hard to define nominal, useful processes in environments where people have already been. Office products are pretty standard now. So what's the process to develop them? It's kinda simple, really, at least in an abstract sense: learn how office workers do their jobs, provide tools to help them extend and enhance their efforts without significantly changing the way they work, provide simple training, release the product, listen to the comments, then make appropriate improvements. It takes years, and there are metrics to take, but we know what to do because we've done it before. Getting to a level 3 is not only fairly easy for experienced development shacks, but it's pretty easy to KNOW when you're there because we've all been there at one time or another, if we've been successful at all a building something big before. Typically, it's a matter of scale.

This brings me to my second point: what about going where no one has gone before? How about level 4 and 5? There are software shacks operating and assessed this level - accredited by SEI themselves. Do they actually produce sustainable process across projects, and are they actually superior makers of software products? In my experience, not always. Why? I believe it's because the criteria for the Mighty 4 and 5 levels simply are naive, based on unverifiable assumptions. Truth be told, we really can't know what it TAKES to be a level 5 organization because no-one has really ever done it - even if they are accredited at this level.

Additionally, the few software shacks that can claim a level 5 typically have commercially unviable processes. The team that builds the man-rated space shuttle environmental control software (the inventors of the "clean room" process) is a level 5 house, they spend 87% of their development resources on testing. This can't be duplicated in the commercial world! It's too expensive, and it takes too long to get it to market. Level 5 seems to be an orthogonal rating by itself; for government systems with life-and-death implications, it's probably appropriate. For a spreadsheet? I hardly think it applies in this context.

In sumary, yes, I think process is valuable, but we have yet to completely and correctly provide critieria for the highest levels of accreditation, and these levels may be meaningless anyway within the context of general commercial product development.

Rick Thorne (rick.thorne@lmco.com)

-- Anonymous, April 12, 2001

We've all heard the aphorism that "quality is free", and that it is more cost-effective to do things right than not. However, let's pretend like we're engineers for a moment, and actually look at some numbers.

From this ( http://www.fastcompany.com/online/06/writestuff.html ) article, we learn some numbers from the most visible CMM level 5 group, the Space Shuttle software programmers. They have about 260 employees who are responsible for all aspects of the 420 klines of code, and they've been writing and maintaining it for ten years. It is unclear to me how much of the 420klines changes for each launch, I'm guessing not much.

Assuming this scales linearly (ha!), and ignoring the amount of time it takes to create code with this process (double-ha), that's 1615 lines of code per employee (not per programmer). Assuming the well- publicized number of 50 million lines of source in Windows 2000, Microsoft would have needed about 31 thousand employees just for this one product alone, about as many as the entire company employs today. So clearly, it does cost more to use this method than the (obviously non CMM level 5) methods that are used to produce Windows 2K and other popular software.

Clearly, it isn't cost-effective for Microsoft to produce software in this manner. Obviously, we are willing to pay more to produce safety- critical software, but are most organizations willing to pay more for non-safety critical applications? I doubt it.

The open question is how to cost-effectively produce high-quality software. Just because the space shuttle method isn't cost effective doesn't mean a different method doesn't exist. It would be interesting to get similar numbers from the OpenBSD project, another group committed to delivering high-quality software. I think that their open-source nature makes it hard to get quantitative numbers about "cost", but I'm sure that they are nowhere near 1615 lines of code per auditor.

-- Anonymous, April 16, 2001

Concerning process: software development is a complex biz. I teach software development process courses where I work, and I ask my students the following question before I start: "which is more complex: your car, or your word processor?" Most students - engineers all - answer "the car". They're wrong.

No, they're right. A modern car has dozens of systems, thousands of components, and a multitude of complex engineering decisions behind it, many of which directly relate to safe vehicle operation and occupant safety. Consider the total engineering (design/safety/quality/manufacturing/support) hours for a new car design versus a new word processor. Furthermore, consider the worst- case consequences of a wrong design decision: For a word processor, the program crashes; for a car, someone dies.

My point here is that too many people in the software biz think that software development is a unique process, far more complex than any other engineering discipline, and therefore we need a bookcase full of procedures, software process-automation tools and the latest programming methodolgy to create quality software.

Yes, there are very complex SW projects and there are simple toasters, but until SW becomes a true engineering discipline all the CMM processes, TQM and the rest won't result in consistently high quality software.

-- Anonymous, April 17, 2001

"Yes, there are very complex SW projects and there are simple toasters, but until SW becomes a true engineering discipline all the CMM processes, TQM and the rest won't result in consistently high quality software."

The concept of CMM is that you document and develop the discipline that will be required for your group to deliver high quality software (remembering that quality is in the eye of the beholder). One step in developing SW as a true engineering discipline is providing the guidelines that need to be met by developers.

While software may not be more complex than other engineering disciplines, it is perhaps the most pervasive. Software applications are in part responsible for the design, control and monitoring of the products of other disciplines. Where these disciplines have their standards and bookcases full of information to govern their development work, should not the software discipline also have governing processes and standards?

This isn't to say that the process applied to a word processor should be or needs to be equivalent to the process applied to vehicle software, just as the same process is not applied for the design of pressure vessels versus storage vessels.

Since software remains a nascent discipline, movements such as CMM are a starting point to address the processes required to produce the various levels of software. As with any other discipline, these guidelines must exist to protect the public from software deficiencies.

-- Anonymous, April 17, 2001

Well there are processes, there are processes... I have been an Assessment Team Member in a Level 5 assessment and our organisation has been at Level 4 for 3 years before going for Level 5. I have managed many projects with and without mature processes.

The CMM is only a road map; a general pointer to the areas that need attention and improvement. Being CMM 4 or 5 doesn't by itself guarantee a quality product. Finally it's good project management, software engineering and good people.

But having a process is a very good CYA. It greatly reduces the project manager's knot in the stomach...

The chief advantage of a process is in the predictability; it is really true that I can predict my effort, defects and schedule if I use a process... Process capability is a much misunderstood term.

The whole key to CMM is in 2 KPA's; ISM and QPM. I can set the process for my projects and also understand my project progress in numbers. The value of a level 4 process comes in running an XMR chart on the plan effort vs actuals at a very small milestone level (say 10- 12 hrs /task) and tracking the outliers. Earned value status repoting and statistically managing progress is ccertainly recommended. With a CMM compliant process we dont need supermen for PM's; just good old plodding and details.

I am a big fan of the CMM; I think it delivers value and very good value at that. How else can I estimate my projects and expect to finish on time?

Cheers B

-- Anonymous, July 27, 2001

At first one may say that CMM is just a waste of time. They are not used to.. in doing.. being controled. But the end result of CMM is total productivity and improvement in processes. The transition is the most difficult part (waste of time), but the payoff is good!

-- Anonymous, August 01, 2001