I've been thinking about process a lot lately. I sort of think of process like I think about the police. I really like having the police around, as long as they are not around me. My old saw has been "when a company starts worrying about process it is because it no longer trusts its people." I'll stand by that, for the most part. I've been known to say that a process committee is a sign of a human resources issue. This is all tongue in cheek, of course, but...
All of that said, a repeatable, optimized process is a crucial thing to have (like the police). In a utopia you wouldn't need the police. Can you think of a company you've worked at that could be called a utopia? Exactly.
I've done XP (and actually done it, all the elements) and I've done N(ot)XP (some of the steps, and call it XP), something calling itself RUP, something calling itself Spiral, and like everyone else in the industry lots of waterfall. XP is interesting as the early poster child (and now flame target) of the agile movement. I like a lot of the verbiage coming out of the agile movement. It is the kind of thing saying "don't make an eight lane superhighway and expect people to go 45 because the signs say that, duh" to software development (and use police to enforce the 45).
If process has a place, and in fact is needed, how the heck do you do this? I admit, I am deeply scarred by having been involved with CMM certification. CMM is... a nice opportunity for management consultants to make some more money. You trade predictability for productivity in a directly inverse proportional relationship. The idea is sound, but I think the design is wrong (okay you SEI guys, start the flamethrowers). No, I don't have a wholly better one, but people like Mary Poppendieck at least make sense.
I've been with product companies most of my career, and most recently with an ASP model. Typical feature request turnaround time was a couple of days there. It was pretty cool. Prior to that I was with a big waterfall process company, with year+ development efforts for a single version. Typical feature turnaround there was closer to the limit of 1/n as n approaches 0.
These two most recent product companies provide a stark contrast with each other, which colors my view a great deal. Iterative design and development works. Keeping iterations small, automating the heck out of everything (making that automation important), and delaying decisions works. In my experience anyway.
So, I am a consultant now, and things change. It's really interesting how my outlook changed. Previously the company's culture, team makeups, etc, were known qualities. I trusted my coworkers' judgement and ability because I knew it. Now I fully trust the abilities of my coworkers, but I cannot know the abilities of the people who will be maintaining code I work on, as they won't be my coworkers. This kind of changes my perceptions, a lot. My natural tendancy is to presume most people are smarter and more capable than me. For the most part, this is true. But, it is also scary as the long term viability of a product you are instrumental in designing and developing is sure as heck going to reflect on you (and by you, I mean me, and the consultancy I work for), so I am sure as heck motivated to make it unbreakable.
Oracle flirted with "unbreakable." I wish their installer was unbreakable (or at least allowed for specifying a bloody ip/hostname instead of trying to guess it when the machine is dhcp'd, or has multiple interfaces (and en1 is active, not en0) -- yes, I will update /etc/hosts when my ip changes, but I want oracle on my laptop, darn it). I digress.
So I start worrying about process from a defensive posture. All of a sudden I don't trust the people, because I have no clue who "the people" are, and cannot know. This makes me feel really, really bad. I suspect "the people" are smarter and more capable than I am, but I don't know, and cannot know. I think this is why bigger organizations worry more about process. It is easy to trust "the people" when there are only twenty of them slinging code, but when there are 500 it is impossible to know them.
So, no answers, but am poking around, and a whole bunch of folks are poking around at this stuff in a big way, and seem way brighter and more capable than me =)