Software development is not really about programming. Anybody can write a ten-line program. The true challenge are complex systems. If a system is so large that a single person alone cannot understand it and develop it further, concepts such as modularization are crucial. Modularization divides the system into small units, which a single person can handle. Then complexity becomes the main challenge. With this kind of complexity, single individuals can no longer implement projects, only teams. This leads to organizational challenges.
Conway’s Law
Conway’s Law is important in the context of organization and software development. It states that the architecture of a system represents the communication structures of the organization that implements the system. For each module in the software there is a organizational unit and for each communication relationship between organizational units there is a dependency between the modules in the software.
Conway’s paper of 1968, however, also describes something else: If an organization wants to develop a big system, a lot of people will need to work on the project. Since communication in a large team is difficult, it collapses at a certain team size. Since communication and architecture influence each other, poor communication leads to chaotic architecture and additional complexity.
But Conway goes further: Obviously, if at all possible, you should aim for an elegant solutions that a small team can implement. But a manager’s prestige depends on the size of the team and budget he or she is responsible for, says Conway. That’s why a manager will strive for as large a team and a large budget as possible.
That doesn’t seem to be a problem at first. If the project is done with too large a team, then some people will just be sitting around and do nothing. That costs money, but doesn’t jeopardize the project or the architecture. But Conway says that Parkinson’s Law will strike. Parkinson’s law explains why some administrations hire more employees, but still don’t get more work done. The law states that a task uses the time available to all employees completely. Even if the task can be processed easily and quickly, more and more people participate until everyone in the organization is busy. So in a software project, all team members will work on the project, regardless of whether this is necessary or not. Accordingly, the organization will grow, communication will collapse, and the architecture will become chaotic.
Conway’s insight, which is now 50 years old, is particularly interesting because it can explain why a large important project might have a bad architecture and is difficult to develop further even if it is a very important project.
The managers, that this paper describe, worship complexity without realizing it. They want as large a team as possible and thereby they make a problem complex because a large organization can cause the architecture to collapse.
What About Software Architects?
Not only managers but also software architects sometimes unconsciously worship complexity. This happens, for example, when we use patterns such as event sourcing, architectures with many layers or microservices without considering sufficiently the benefits in the specific context versus the complexity.
If you want to use the latest and shiniest technology, that can also lead to excessive complexity. After all, we all are looking for technical challenges and want to implement interesting projects. Modern approaches and in particular complex systems are well suited for this.
We also sometimes solve technical problems that do not exist. This might result in very generic or scalable solutions that are not required by the actual requirements and therefore generate too much complexity.
Complexity as an Excuse
A particularly blatant case of complexity worshiping is the statement “This doesn’t work for us. Our challenges are much greater than those of Amazon or Google.” I’ve heard that from employees of different companies. Such statements are surprising: companies like Amazon or Google have extremely complex IT systems. Their economic success depends directly on these IT systems. Not least because of these IT systems, they are among the most valuable companies in the world.
At first glance, the statements can be interpreted as defensive: Amazon and Google have a modern organization and a cloud infrastructure, but in the company’s much more complex environment, it’s obviously impossible to establish similar things. But perhaps this statement reflects pride. After all, you’re dealing with almost unprecedented challenges. Either way, the complexity of course has disadvantages, but also the advantage that you don’t have to consider certain approaches such as cloud, continuous delivery, or microservices, because they are impossible to implement anyway.
It is therefore questionable whether we really always avoid complexity. To concentrate only on techniques to make designs as simple and elegant as possible, is of no use if we unconsciously worship complexity. Therefore, it is important to realize these unconscious mechanisms. Of course, there are still many complex problems that are actually difficult to solve.
Many thanks to my colleagues Jens Bendisposto, Jochen Christ, Lutz Hühnken, Michael Vitz and Benjamin Wolf for their comments on an earlier version of the article.
tl;dr
Software development is all about handling complexity. It would be best to avoid complexity right away. But unfortunately, there are times when complexity is worshipped - consciously or unconsciously - leading to unnecessarily complex systems.