Software Process - Maturity, Formality, & Ceremony

Blogger: Kirk Knoernschild

The degree to which the software development process is defined, managed, measured, and controlled defines an organization’s process maturity.  Increased process maturity is a good thing because it’s an indication of an organization’s ability to develop and deliver software. Unfortunately, in focusing on process maturity, many teams fail to consider two other important software process classification schemes - formality and ceremony.

Process formality represents the discipline of the team in adhering to the practices defined by the software development process employed. Process ceremony represents the weight and rigidity of the software development process employed by the development team. Increased process ceremony corresponds to tighter controls placed on the development team. The difference between process formality and ceremony is subtle, but distinct.

For instance, a team may be very disciplined in capturing system requirements and documenting them by writing use cases. However, one team captures these requirements on hand-written notecards and reviews them with customers during weekly system demonstrations. Another team uses a standard use case template and requires customer sign-off on the requirements before development commences. Both teams are formal in that they capture and document requirements through use cases, but only the process used by the latter has a high degree of ceremony

Waterfall methods tend to be formal and ceremonial, whereas agile methods tend to favor formality with little ceremony. The Rational Unified Process (RUP) is a good example of an iterative method that wants to be formal, but often achieves that formality through ceremony. An unfortunate manifestation of RUP.

The unique challenges of individual software development projects vary widely. In general, process formality is a good thing while ceremony is wasteful and impedes progress. Only when teams lack the discipline to execute important lifecycle activities is ceremony necessary. Something to keep in mind as you look to increase software development process maturity.

Comments

[Barry Baker]

Could you please expand on your definitions a bit? In particular, please explain your distinction between formality and ceremony. To me, "ceremony" is a loaded term often used by "agile" proponents bludgeon processes that don not meet their "agility" standards. Sometimes people classify work byproducts of a development process that do not get seen/used by the end user as "ceremony", while I've also heard others talk about the steps taken to do hand-offs between groups of people/teams as the "ceremony".

A formal process is a mature process composed of proven development practices. However, there are no artificial process gates in place that strictly govern the process. Formal processes rely on the professionalism and discipline of the team to ensure the right thing gets done at the right time. A ceremonial process is a formal process that strictly governs what is done and when it is done. Ceremonial processes are typically marked by milestones (ie. the artifical process gates) such as requirements complete, architecture complete, design complete, etc. Usually, each of these milestones is reached when some artifact is created and signed.

In the original post, the ceremonial aspect of the example is that customer sign-off is required before development can commence. The two samples you cite (work products not seen by the end user and steps taken to hand-off) aren't necessarily ceremonial unless done only to comply with process.

[NLS]

Would suggest that blanket dismissal of ceremony may be too critcal an approach. While the informality of agile processes is admirable, and works well in many environments, it has drawbacks in large projects, and sometimes in those for which software correctness and reliability are primary requirements.
In the latter case, formally mandated checkpoints and formal documentation of accomplishment of specific DT&E objectives may be essential to meet safety, reliability, and legal requirements.
For high-reliability and life-critical projects, the additional formal documentation of specific reviews, checks, and double-checks, while it doesn't guarantee completeness, can go far towards ensuring that processes intended to capture and correct the inevitable human errors and oversights are completed.
For large projects, the likelihood, and the absolute quantity, of defects is greater, due to the greater variety of skill levels associated with a larger workforce and the greater variety of novel problems and unforeseen interactions. In these cases, no one manager, or single small team of managers, has the span of awareness and authority to informally assure a disciplined DT&E process. 'Ceremony' may be essential in verifying that essential team interactions and processes have been accomplished.

Agreed. In some cases ceremony may be necessary, including if the team lacks the discipline.

I saw a definition of ceremony that I thought summed it up well, I believe from Alistair Cockburn. "Dogmatic application of SDLC practices without thinking through whether a specific practice makes sense."

One last note about your most recent comment...I'd argue that agile done right is very formal, but avoids ceremony.

[Barry Baker]

...I'd argue, because I've seen it, that other processes, include waterfall, done right can be very formal, but avoid ceremony...

How is the completion of a time-boxed iteration not a milestone, especially if you are letting end users get their hands on it? How is the completion of time-boxed iteration not "a formal process that strictly governs what is done and when it is done"? This sounds like ceremony to me as you describe it.