Agile processes aim to develop a successful product. They work in iterations. After each iteration, customers can use the product in its current state. This gives the team feedback as to which features are popular and how the product can be further developed. Iterations also reduce risk: It is always obvious which features really work in production and which don’t. A minimal product can go live very quickly after project start, which further reduces the risk of failure. Based on the historic velocity, it is possible to estimate how much time the team will need for future features. The team is successful together. Not a single person is responsible for the project, but the entire team.
Agile methods thus reduce risk, lead to rapid success, enable realistic planning, and support team collaboration. With so many advantages, agility should be the obvious approach to software development and sell itself. But in practice agile transformations are unfortunately still a challenge.
Experiences
In one project in which the author was involved the advantages of agility became problems. Agility would have made it clear at an early stage that the project would take much longer than planned by putting the software into production at an early stage and estimating the effort based on this. Of course it would have been good if this risk had become apparent early on. But in this organization, the risk would probably have led to the termination of the project and probably also the team, because the team is not up to the task. Without agility, the project did not reach its original goals, but with a few political tricks it escaped termination and at least delivered software in production - and the team kept its job.
The problem is not the agile process, but how to deal with the feedback that agile processes provide. If problems cause teams to be laid off or projects to be cancelled, then it may unfortunately be that concealing risks and delays as well as political games are the only way to carry out a risky project and keeping your job. True agile organizations would analyse the problem and solve it.
Agility in this project would also have led to problems for the project managers' career. In order to progress their career, project managers must complete projects successfully. But in an agile project there is no project manager. The entire team is responsible for success or failure. This contradicts classical organizations where managers or project managers are responsible for the success of the teams and are rewarded or punished for it. This understanding must change fundamentally for agile processes. Agility advocates the responsibility of the team - and rightly so. But it is not so easy to implement in some organizations.
Even more Experiences
In another project, an agile project organization was combined with a classic product designers. The latter thought in quarterly releases, while the project organization was able to provide new software every fortnight. User feedback on the 14-day releases could not be included in product planning. Often it would have been possible to first implement a feature in a simple way and only later fully implement the feature. This would have allowed users to try out the feature at an early stage. If the feature hadn’t been successful, the investment in the feature could have been terminated. This is important because the literature shows that product designers predict the popularity of a feature correctly only in 30 percent of cases at most. Agility makes it possible to rely on user feedback instead of estimation. To do this, however, the product designers must be clear about their poor estimation rate and then incorporate feedback.
And finally, the motivations of those involved in a project are different. IT consultancies certainly want the project to be successful, but they also have to earn money, while clients want to save money. The opposite goals lead to potential conflicts. Engineers are often more interested in technologies than in project success. In the end, a project involves a large number of people who have their own motivations. The success of the project can be very important to them or rather irrelevant. And sometimes some participants are even interested in the failure of the project in order to advance their own career.
In contrast, agility assumes that all team members are interested in the success of the project. This makes sense, because if the those involved in the project do not really aim at its success, there is little you can do to lead the project to success.
Problems?
The problems mentioned above have their roots in culture and management. Organizations need to deal constructively with problems, rethink the role of project managers, agilize product design, and promote a positive and constructive culture. In other words, agile processes only work if the environment and culture support them. If these conditions are not present, you have to change them. But can software architects, developers or project managers change an organization so fundamentally? Changes in culture are difficult even for managers, because culture is fundamental to a company as a social structure. Companies are often well positioned in the market with their old process models and culture, so there is no pressure for change - let alone radical cultural change if it is „just“ a software development project that does not really work out.
The problem is not limited to agility. Continuous delivery supports frequent releases of software. Microservices support agility and self-organized teams. If agility has only a limited positive impact because of the culture, then this also applies to continuous delivery and microservices. And that’s why it’s so important to make agility a success and change the culture.
Solution?
If the companies do not fundamentally change the culture to support agility, the result will be a more or less half-hearted agile process. At least there seems to be no way to adapt agility to these cultures. Unfortunately, „real“ agility is the exception. In other words, agility does not work in the majority of cases, and the reason is the problem with the culture.
And too often agility is simply misunderstood. Sometimes agility has even become „more pressure“. This has nothing to do with a responsible team and a constructive approach to feedback.
Agile transition will remain a topic in the next twenty years and there will still be many „agile“ projects, which in reality are not agile.
Many thanks to my colleagues Björn Behrens, Martin Eigenbrodt, Lutz Hühnken, Martin Kühl, Andreas Krüger, Michael Schürig, Stefan Tilkov, Benjamin Wolf, and Oliver Wolf for their comments on an earlier version of the article.
tl;dr
Agile transformation has been an issue for twenty years. Because agility does not fit the culture of many companies, it will remain an issue. Most agile projects will still only be able to implement half-hearted agility.