Agile development: one of the worst concepts ever to exist in software

This article is a first draft. I will polish it up later.

To begin, some context: in software development (and project management as a whole), Agile is a development structure. It can be seen at https://agilemanifesto.org/, but in a nutshell it says this:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

With no hindsight whatsoever, it is easy to see that this was a terrible, terrible idea from the start. Anyone who has ever hacked together a project under pressure can see this: the entire concept of Agile is focused on pleasing the customer in the short term, at any price in the long term.

This has caused no end of damage in most software companies, because most software is not a short-term project; cutting corners and failing to document what you’re doing ends in disaster. Imagine building a 70-storey skyscraper floor by floor, and each time you build the next floor you hack it together as fast as possible and cut every corner; you end up with a building that costs billions to repair to keep it from collapsing every day, and which is far more expensive than if you had just built it correctly from the start.

This is an ubiqituous problem in software, and it is known as technical debt. To summarise the definition of technical debt from Wikipedia:

Technical debt is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.

I should note that technical debt doesn’t only include the cost of reworking existing components: adding new features to software with high technical debt often takes orders of magnitude more time and money than it should, since the code is often tangled up with dependencies that shouldn’t exist all over the place, legacy code that isn’t documented and only one person vaguely understands, and inefficient code nobody wants to put up the time and money to fix.

The reason why Agile is so bad in this regard is that the entire Agile Manifesto works to increase technical debt as fast as possible, and is oriented at short-term thinking only. Why is this? Let us examine each line of the manifesto in turn.

Manifesto line 1: Individuals and interactions over processes and tools

It should be clear, logically, that this is bad in the long term. Processes and tools are documented, don’t rely on being in your memory, and when followed correctly have the same outcome no matter who performs them.

Individuals, on the other hand, will often leave the company and work elsewhere; interactions will be forgotten. When you have to ask Jane what John said about Mark’s personal rules for committing the code, there is a huge amount of potential for miscommunication and error. If Mark decides tomorrow to change how the code should be committed, who will know? What if he leaves the company? How do you ensure that a consistent process is followed?

If you have a process, you don’t have this problem; this is why all competent, modern software companies use processes and tools, such as continuous integration, version control, continuous delivery/deployment, dependency management, and more. By using processes and tools, we can eliminate more human error and speed up development at the same time by reducing tedious work.

Manifesto line 2: Working software over comprehensive documentation

This is the ultimate “short term tunnel vision mindset” concept. Software is not like a piece of furniture you need to complete, ship, and forget about once it’s shipped: it is going to be constantly reworked, maintained, and extended.

When you combine prioritising “working software” with commercial time pressures, the inevitable result is poor quality code that has been rushed to meet customer deadlines. The attributes of such poor quality code have a multitude of terms, but here are a few of them:

  • Spaghetti code — poorly organised code that will need refactoring in future, and so long as it isn’t refactored all modification and extension of this code will take longer than it should
  • Dependency hell — when you don’t have time to update your project to work with newer versions of third-party libraries, this is often the result.
  • Bugs — errors in the code, ranging from minor issues to core failure.
  • Magic Numbers — doesn’t necessarily have to be related to numbers, but is a reference to including constants with unexplained values in the middle of code with no comments. Leads to nightmares later when nobody can remember what the significance of the constant is.

When you try to make changes or extend this code later, it takes much longer than it should, making your company uncompetitive and your software expensive to maintain. Documentation is one of, if not the most, important aspects of software in commercial development: since staff members change and documentation remains the same, it is the one way you can ensure that processes are followed, and that new hires can understand the code without relying on having enough old hires to train them (and lose their development time in the process).

Manifesto line 3: Customer collaboration over contract negotiation

This line has the same root problem as the other lines: mindlessly bowing to customer demands. While in theory, it doesn’t sound bad to ‘collaborate’ with your customer, in the commercial world this frequently translates to agreeing to anything your customer asks for no matter how ridiculous it is.

Here is a very good comedy (or painful, if you work in this field) video which requires no software knowledge, showing how blindly agreeing to customer demands is a terrible idea. It also depicts the rather depressingly accurate reality of how commercial meetings of this nature can sometimes go.

I’ve seen this happen quite a few times myself: I was working in a company where the higher-ups of another team agreed to a huge project contract to write software for another company. Managers, against the very clearly stated objections of the technical leads and senior developers in charge of the project’s implementation, agreed to the customer’s unrealistic 18-month deadline; the result was that the deadline was missed, and the company ended up suffering heavy contractual penalties for not keeping our end of the contract.

There are a few reasons why this happens:

  • Customers are, in general, very poorly placed to know what is reasonable in a software project timeline. How is a non-technical customer meant to know what a reasonable timeframe is for a given feature, or how a certain feature should be implemented?
  • It can be difficult to pin the blame of technical debt on a manager who caused it — many reasons can be given for projects running slowly, but managers are partially measured in performance on how pleased their customers are. As a result, they have an incentive not to care too much about informing the customer of technical consequences of their decisions (or discouraging bad ones), as by the time it becomes a problem it will be impossible to prove they were at fault.
  • Companies that charge on a time-and-materials basis (which is many of them) have no incentive to clean up their code, since if they have to charge the customer for 200 days of work instead of 100 days, the inefficiency of the code doesn’t matter; you’re still getting paid a constant amount per your staffs’ time. While this makes the company less competitive, this generally isn’t seen as important by managers in many industries because of vendor lock-in. Artists might well be familiar with this as a result of Adobe Photoshop; Apple is also a good example of a company well known to most people, for which vendor lock-in is one of their primary strategies. There are endless other examples.

Manifesto line 4: Responding to change over following a plan

There is a good reason people make plans: because planning leads to better performance. If you know what might happen, and prepare for it, you are better placed to react to it. If you react only when things happen, you have no time to consider your options and no time to properly form a strategy, and often end up implementing the fastest, easiest solution in the short term (which in software, frequently leads to horrendous problems in the long term).

‘Responding to change’ is simply a lazy way of dealing with change, instead of having a plan that accounts for changes. In most software projects, a certain amount of contingency time is added to the total estimate of how long the project will take, to account for the fact that things don’t always go to plan. More importantly, a proper plan already accounts for most plausible changes: if the customer asks for a very big change that wasn’t part of the plan, you take time to revise the plan. The customer should not be allowed to throw away the plan and rewrite it in the middle of a project on a whim (which is one of the many reasons for the existence of contracts).

Other development paradigms

One of the most ironic attributes of Agile was that it was originally created as a response to the Waterfall system, in which each part of a project is completed in stages (making a “waterfall”). It was often criticised for being slow, and yet it is far better than Agile for producing good results, and is often faster (due to the technical debt that Agile encourages).

This is not to say that Waterfall doesn’t produce technical debt per se, but it does not encourage it in the manner that Agile does.

Computer scientist. https://twitter.com/antsstyle