How to build a truly modular system (or organization)?
Everybody talks about the modular approach to building large systems. But what does it actually mean?
In today’s article, I will try to share modularity-related insights that might be useful not only for your development teams but also for your entire organization.
The first point I want to start with is that making software is hard. The best way to prove it is to fully answer a seemingly obvious question: how do computer science experts, solution architects, and tech leaders choose technologies they recommend to the client?
The general assumption is that the experts:
→ assess the full project scope and resources
→ compare all available technologies by their quality and project fit
→ minimize risk factors and overall client’s costs
But in reality, they rather:
→ choose technology that they already know (which is rarely the best technology out there)
→ choose technology that they want to learn (yes, that is correct)
→ discount risk factors and client’s costs
Before you come to the wrong conclusion, let me elaborate. It is not that experts have bad intentions or lack competence. Quite the contrary. They are just faced with a virtually impossible task. To explain this let’s compare software development to road construction:
Both domains have their experts. Both domains have projects that require careful planning, including assessment of client and supplier resources, time constraints, etc. What are the main differences between the two?
First, there are probably just a few technologies (techniques and materials) used to build roads. In programming, you have dozens of well-known languages, each of them having thousands or more libraries, with overlapping functionalities. On top of that, you have many parallel design patterns which you use to build and scale up the system. The number of possible ways in which you can build a system is gigantic.
Second, road construction is a quite repeatable task. You can compare one solution to another. In software, each application is unique, you never build the same system twice. You cannot directly compare different approaches, although some techniques such as prototyping allow you to revoke bad technological choices at an early stage (according to the fail-fast principle). But generally, building new software means dealing with an unknown.
Third, road construction techniques are very stable, they evolve slowly, the changes in the process are introduced rarely. In programming, the whole industry is in a state of constant change. Programming languages evolve, thousands of new software libraries are published every day. There is no top-down control over the community. Because of that, every programmer (no matter how proficient) needs to spend the majority of one’s time learning new things. You just simply never stop learning.
If mastering one programming language takes a few years (the same with design patterns), and mastering one technology or library takes a few months - can we expect that a single expert can analyze and compare every possible solution? We can’t.
Even the part where we let programmers experiment with new technologies is a justifiable win-win scenario. Otherwise, we would have narrow domain experts, who have zero knowledge in other technologies nor the possibility to learn ones. They would quickly burn out and their expertise would become outdated, the same as their preferred technologies.
To address the problem of dealing with an unknown, two guidelines have been established, the first process-wise and the second technology-wise:
1. Be agile (ready to change).
2. Reuse good practices.
Good practices (also called design principles) are replicable across different programming languages and technologies. Hence their success. Examples of such principles are microservices and plugin-module architecture.
But even practices are subject to trends. For example, microservices are a very powerful way to build scalable applications, but their utility in small and average systems is somewhat disputable. A similar problem of fashionable theory taking over the landscape of research has been explained by Sir Roger Penrose in his book Fashion, Faith, and Fantasy, where he describes the prevalent success of String Theory in the theoretical domain of physics, despite lack of substantial evidence confirming the theory so far. I recommend this book to anyone who likes to ask difficult questions.
Now, let’s go back to the main topic of the article - modularity.
To understand what the module is, please take a look at the general diagram of software architecture.
The general idea is that everything is a module. The software system can be built of horizontal slices (technical modules) and vertical slices (business modules). There are many definitions of modular architecture, but this is the one I use.
We can think of modules (especially technical ones) as compressing information and creating hierarchies of complexity. A good example of this would be TCP/IP network layers.
Each module provides an interface (which can be called a contract, abstraction, or generalization), which contains a compressed definition of how to use it. For example, when you shop online and pay for the goods, the PayPal service promises to pay to the store the transferred amount, without you having to worry about how it all works. It hides all the unnecessary details from you and promises that none of those details will ever affect your user experience. In a situation when it does, we call it a leaking abstraction.
When general contract depends on implementation details, we call it a leaking abstraction.
Now, let’s answer a very important question. What is the difference between standard approach and truly modular approach?
In traditional programming of technical modules (layers), we have had these long-established assumptions:
→ upper layers in hierarchy depend on (extend) lower layers
→ lower layers in hierarchy do not know anything about the upper layers
The truly modular approach requires that we bring these assumptions into the business domain.
The previous modules do not know anything about the next modules. The next modules require previous ones to work. All modules form a so-called dependency tree with branches. Incidentally, by using this definition one can discover the best way to test if something is really a module.
To test how modular something is,
check how difficult it is to remove it.
A true module can be removed using one click (I do not mean disabling functionality, which is easier, but physically removing the module). A false module will require a lot of changes to be fully removed, because of dependencies hard-coded within its parents.
Examples:
Lego has a truly modular structure. Most parts of the model can be removed in a single step. Lego piece does not know what other pieces are connected to it (the information about an extension is part of the extending piece, not an extended one).
Classic software application does not have a modular structure. The website’s home page knows all its sub-components. Removing any sub-component (child) requires modification of the home page (parent).
This way of thinking about modularity is useful beyond the field of computer science. For example, I believe all types of organizations should consider an idea that the best employee is the one that can leave the company without any loss for the company, or in other words:
the best employee is a modular employee.
Let me explain what I mean by this seemingly counter-intuitive statement :) Consider two example employees: Mark and John:
Mark is an example of an old-school expert. He has the best knowledge in the company about technology X. He knows the details that other people do not. When there are problems with technology X, everybody turns to Mark. Mark has created effective processes for years and everything depends on him. He has established a very high position at the company and he enjoys being indispensable.
Now, let’s consider John’s attitude towards work:
John is a different type of expert. He shares all the knowledge with his team members. He uses his experience to teach and guide others. He wants to create the next experts. He contributes to creating processes but makes final decisions together with his team. He has a boring habit of keeping documentation clear and up to date. John knows he is not indispensable. He can be hit by the bus (;-)) and the company will continue to operate on a normal basis.
Only one of those attitudes ends with a win-win result. Feel free to guess which it is.
Being indispensable makes you feel special. People struggle to find a moment with you. You can get a higher salary and have a decisive voice on certain matters. From an individual perspective, this seems like a valuable goal. But similarly to the prisoner dilemma, in iterated scenarios long-term strategies prevail (for details of the topic, please check our other blog posts by Marcin Woźniak). If you focus on your position within the company, soon other people will follow and processes will degenerate. You will be less free to leave, because of great cost to you or the company.
On the other hand, consider John’s position. He is only an addition (extension) to the team, not a necessary ingredient. He is wise enough to save his own and company’s resources at the same time. He is free to leave the company without hurting anyone in the process. For all those reasons, he should be highly appreciated. Unfortunately, there might be managers who will try to use him as a tool to increase the company’s efficiency, and then fire him, defaulting to a win-lose strategy (which equals lose-lose in the long term). Sadly, there is no way to prevent that because:
The best people are always the most vulnerable.
That is what makes them best.
The above paragraph is a result of my own analysis of team dynamics as a team member and team leader. It is also related to a discussion of centralized power vs building long-standing institutions. I hope you will find it interesting and relevant to your own work.
Going back to the main topic of the article: there are many ways you can implement a modular architecture, depending on the technology that you use. I will only mention a few general design principles that are reusable across different programming languages and technologies:
Think of a module as an addition to the existing code-base (for example new folder with the source code). Your architecture should be constructed in a way, that allows making any possible change by adding code only (you cannot remove or change old code).
If you work on a library module and you have only 1 use case of this library in your application - invent a second, different one. This can be an artificial use case or another client type. This will improve your abstraction.
Do not tie yourself to specific technologies, because they are currently best on the market (database, front-end, back-end). If you keep that in mind, your architecture will be more flexible.
Use dependency injection and inversion of control. Specify contract for extensions, but do not import the specific ones. It should be extensions (children) that decide in which house they want to be put (Gryffindor of course).
If your architecture allows you to add two modules at the same level, it is good architecture. Parallel modules are only possible when you have properly implemented extensions.
Make the code extensible. It will allow you to outsource work to many sub-teams and none of them will interfere with each other.
There are many more benefits of the modular approach and guidelines on how to implement it in specific technologies. In the future blog posts, I may be covering more technical details.
PS: I just reminded myself that the O in SOLID acronym stands for Software entities should be open for extension, but closed for modification. It is nice when your ideas are cross-validated by other, already approved practices.
Thank you for reading this article. Let’s all stay modular! :)
