This answer is so good it should be broken out of the Quora signup-wall.
Answer by Michael O. Church:
- The processes that result in quality software are almost impossible to control or manage. There's an expectancy-variance tradeoff in creative work, including software. If you want greatness, you hire excellent people and give them full autonomy, with the understanding that you won't always get the features you want on the schedule you expect, but that everything that is actually important will be done by a very smart, competent person in time. (Think of it as akin to Eventual consistency (wikipedia.org). It's not always what you want, but if you can afford to wait, you get good results.) With this approach, you get excellent average-case performance (expectancy) but a lot of variability, especially in the short term. On the other hand, if you want reliable mediocrity, you hire a lower quality of engineer in larger numbers, and micromanage. Your expectancy drops, but you're minimizing variance. The first produces an R&D environment that produces great work in the long term, but has variable output and can be politically messy (people envy the engineers for having better jobs in their R&D environments, and eventually it becomes An Issue). The second is much more tenable in the typical corporate environment. You can define "deliverables" and know who to fire if the schedule slips.
- Enterprise projects tend naturally toward Bigness, and Big Software is rarely of high quality. (See: Java Shop Politics (wordpress.com)). Competent software engineers like modularity, small programs, and simple interfaces. This is the essence of the Unix philosophy: large problems should be solved with systems and given the respect that systems require, and not with massive single programs with poorly specified and undocumented communication protocols. Good engineers make the software environment better by writing small programs– executables, utilities, libraries– and only create Big Things when absolutely necessary. However, this type of effort is hard to track, because one programmer might work on 20 different programs in a given year. Management has no idea what its best people are doing. Managerial dinosaurs feel more secure when the programmer:project relationship is reversed– many-to-one instead of one-to-many– because they can control the allocation of efforts when they get together and fight over "headcount". Unfortunately, Big Programs tend to collapse under the weight of their complexity, especially when they change hands.
- Birdshit syndrome. This is related to #2. Imagine a square-mile canvas, laid down in a park, that you intend to paint on. Before you're done, it's overwhelmingly likely that it will be ruined by birdshit, on account of the massive size of the canvas. Enterprise software projects need to constantly market themselves within the organization, and the increasing visibility (surface area) also drives up the risk of attracting nonsensical requirements (birdshit) from on high. If the project seems to be at risk of becoming actually important, everyone who has the power to slow the project down expects a hand-out. These requirements are akin to the never-ending and often escalating outflow of bribes and extortions that are required to do business in a politically-undeveloped country. If there are 18 people who have the power to stop the software project, then there are 18 directions from which nonsensical feature expectations and requirements can come, usually before the project is established or mature enough for its managers to say "No". After a few years, the conceptual integrity of the project's original design is gone.
- Lack of career incentives. Big Software projects are good for the manager's career, because he can put on his CV that he managed a large team. Having 20 reports is more impressive than having 4, regardless of whether the team of 4 engineers accomplished more. These projects are bad for engineers' careers, because per-person productivity is so low, and because you don't really learn much if you're not in a leadership or architectural role on such an effort. On Big Software projects, 10 to 20 lines of code per day is about average, 80 percent of the work is maintenance even before the thing launches, and it's not uncommon to go two months without checking in a line of code. Worse yet, the engineers gain none of the visibility that they might have if they, for example, maintained or started an open source project. This is a good environment for maxed-out, bored engineers who want to milk the clock for 5 years (long enough to make a manager-level external promotion inevitable) and not get fired, but the better and more ambitious engineers tend to realize when a project won't do anything for their careers, and they start planning their exits.
- "Uncanny Valley." Good software engineers either want to be directly in the line of business in high-impact roles (startup founder, trading algorithms, data scientist, CTO) or they want to be far away from it in a blissful R&D land with full autonomy over how they spend their time. The first provides the way to higher social status and credibility, better compensation, and most importantly, the chance to have an impact. Work in the direct line-of-business is often unpleasant– even (if not especially) for high-status players– but the upshot is knowing that your work actually matters. The second category of environment (R&D) gives the engineer the right to direct his efforts and optimize for learning, career growth, and long-term impact. Most IT organizations live in an uncanny valley between these two states, where the engineers' lives and quality-of-work is affected by the line of business, but in which they're second-string players, treated as cost centers rather than profit generators. Good engineers, if they're going to accept the compromises that come with being in the line of business, also want the "key player" status and credibility that can come with it. Most IT organizations have the compromise but not the status.
- Talent bleed. This is related to #4 and #5. The best engineers don't want to work on projects that have lost their conceptual integrity, because projects that get to that point are bad for an engineer's career. About 50% of those over-ambitious enterprise projects fail completely, and the other 40% are held up by a "too big to fail" sunk-cost mentality, but become disparaged legacy systems that not only harm the reputations of those who worked on them, but also erode the reputation of IT in general, creating the impression that "our engineers can never do anything right". (Weirdly, people who manage bad projects always seem to be OK. They "took one for the team" in attempting them, even though they delegated all the crap work.) When good engineers end up on these projects, they either negotiate for more autonomy (which often cannot be afforded) or more compensation (which is hard to get when the manager already has to fight hard for headcount). The only way a typical enterprise effort can keep its most talented people is to give them "special projects" (see #5) outside of the line of business, but very few IT managers have the "headcount" to afford that. Occasionally, however, one of these "special projects" will turn out to be really cool and take on a life of its own. Then, it needs additional support in order to meet the standards imposed by the business. Then it has to fight for headcount… the Cycle begins anew.
- A vicious cycle of low status leading to crappy results. Because most companies can't keep talent in their IT organizations, the result is that these highly ambitious Big Software projects tend overwhelmingly toward mediocrity. They build all-or-nothing systems that just barely work, but people don't have the right not to use them (as would occur on the market) because that would be politically messy. In the long term, this elephant graveyard of crappy Big Software brings down the status of IT, and reduces its autonomy within the organization even further.
