Going back to old discussions with engineers, architects and CTOs, I noticed that practically all of them started their projects with passion and commitment but got trapped in the most crucial moments to be stuck with complexity, over-engineering and money-draining.

Most started with very few users, use cases and very low/average traffic. Most of them also began with very simplistic concepts and designs, but with time, layers of unconnected black boxes staffed with spaghetti became unmanageably complex.

From Simple to Chaos: A General Trajectory

When I asked them why they reached such a point, they mostly answered: « We thought that technology/concept/methodology X would help us to get over it, but we can not return any more.»

Yet, I usually justify the complexity in architecture using Gregor’s law :

Excessive complexity is nature’s punishment for organisations that are unable to make decisions.
– Gregor Hohpe

Furthermore, I concluded that over-engineering is caused by the lack of experience or the hype to use a new « toy ». And both to duct-tape a few parts of, most of the time, a not well-defined problem.

Retrospectively, most cases were due to lousy steering to handle the debts, any functional, architectural and technical debt. Just more duct tape (as a temporary solution) and rust (as degradation)

The pressure to get more cash and/or the will to overdo/overpromise blur their minds by trying solve urgently the core problems by creating more issues elsewhere, using fluffy goals like « quick wins », « best practices » and « higher KPI »

Yet, I was told quite often that I had trust issues with [software] engineers. And for being one of them by training, I do admit that specially in most of the over-engineering cases. We, engineers, often tend to have a narrow focus or we completly lack it, which drags us away from the problems by overthinking the rare edge cases.
We also suffer of very frequent symptoms of high ego, copying stubbornly others’ solutions/opinions, resistance to change, unable of understanding the market and the end-user, and finally, siloing teams and components.

Besides, I blame the architects also for diluting the coherence by driving it with rigid designs and hard coupled contracts. We, architects, frequently tend to be disconnected from reality and live in a realm full of abstract boxes and arrows. Without mentioning our tendency to overemphasise the latest technologies while forgetting the segmentation and/or the non-coherence of the solution according to the functional or non-functional requirements.

Overcome the Chaos

Now, you might be thinking, « How on Earth can I make sense of this chaos? »

Well, I usually base my gatherings and investiguation by identifying whys (purpose) and hows (actions).

Generally, I use for that : tools and methods

  • Golden Circle by Simon Sinek: The Golden Circle is a framework introduced by Simon Sinek that emphasizes starting with the “why” before addressing the “how” and “what” in business strategy and communication. It suggests that successful organizations inspire action by first communicating the purpose or belief (why), followed by the specific strategies or processes (how), and finally, the tangible products or services (what).

  • The Agile Triangle: The Agile Triangle is a concept within Agile project management that emphasizes the balance between three key constraints: value, quality, and constraints. Unlike the traditional project management triangle, which focuses on trade-offs between scope, time, and quality, the Agile Triangle acknowledges that scope may change dynamically throughout the project. Instead, Agile teams prioritize delivering value to customers by continuously adjusting the scope while managing debt and the constraints effectively. This flexible approach enables teams to respond quickly to changing requirements and market conditions while delivering high-quality products or services.

  • The Effort vs Impact Matrix: The Effort vs Impact Matrix is a visual tool used to prioritize tasks or initiatives based on their relative effort and potential impact. It consists of a grid with effort (or difficulty) plotted on one axis and impact (or value) plotted on the other axis. By plotting tasks on this matrix, teams can focus their efforts on high-impact activities while minimizing time and resources spent on low-impact tasks.

Based on that, I usually throw millions of stupid questions about:

  • Value: Understanding the value proposition is essential. What specific benefits will this feature or solution provide to the end-user? How does it align with the company’s goals and objectives?

  • Location: Placing the topic within the broader landscape helps determine its visibility and relevance. Is it a central component, prominently featured in the overall architecture, or does it serve a more auxiliary role, supporting other functionalities? Understanding its position within the system aids in making informed decisions about its design and implementation.

  • Confidence: Assessing the level of certainty surrounding the project is crucial for managing risk and uncertainty. How confident are we in our understanding of the requirements, technology choices, and potential challenges? By acknowledging and addressing uncertainties early on, we can implement strategies to mitigate risks and adapt to unforeseen circumstances effectively.

  • Organic Evolution: Considering the long-term trajectory of the project is essential for sustainable growth and scalability. What are our immediate goals and objectives? How do we envision the project evolving over the next few years? Understanding the strategic direction allows us to design flexible solutions that can adapt to changing needs and market dynamics.

  • Inertia: Recognizing the need for periodic reassessment and adaptation is vital for maintaining relevance and effectiveness. Do we anticipate the need for regular updates and improvements? How will we manage technical debt and ensure that the solution remains agile and responsive to evolving requirements?

  • Elasticity: Anticipating future usage patterns and scalability requirements helps ensure that the solution can accommodate growth without sacrificing performance or reliability. How will the system handle increased demand over time? Are there potential bottlenecks or scalability challenges that need to be addressed proactively?

  • Policies: Considering access control, regulatory compliance, and disaster recovery measures is essential for ensuring the security and integrity of the system. What policies and procedures need to be in place to protect sensitive data and ensure regulatory compliance? How will we recover from potential disruptions or security breaches?

  • People: Organizing teams effectively and fostering collaboration is critical for project success. How will teams be structured to maximize efficiency and productivity? What knowledge and skills are required, and how can they be acquired or developed within the team? Prioritizing simplicity and clear communication helps streamline workflows and facilitate seamless collaboration.

  • Cost: Evaluating the various costs associated with the project enables informed decision-making and resource allocation. What are the anticipated costs in terms of development effort, ongoing maintenance, and potential migration or upgrade expenses? Understanding the cost implications helps prioritize initiatives and allocate resources effectively.

Obvisouly, I think the more answers we get, the fewer grey zones we will have in the final picture. We can mitigate the risks, align the efforts with strategic objectives without having hallucinations about overpromised robust solutions.

Tackle the over-engineering

developers-moths

Well, to overcome over-engineering some blogs and articles detail more topics, like team topologies, clean code, training and practices … but, I’ll focus on :

  1. Forgetting about reinventing the wheel: It’s essential to leverage existing frameworks, libraries, and tools rather than reinventing solutions from scratch. There’s no need to develop something from scratch that has been out there for years. There are countless open-source libraries, frameworks, and APIs available, offering pre-built functionalities and components that have been thoroughly tested and optimized. I still believe that a developer is like a professional LEGO builder : he knows well what to use and how to make them fit together.

  2. Stop speculation on future needs: Avoid unnecessary complexity by focusing on current requirements rather than attempting to anticipate future needs. By prioritizing present challenges, teams can deliver solutions that address immediate concerns effectively without over-engineering for hypothetical scenarios. furtherlore, by addressing the current requirements and building a flexible, adaptable architecture/design that can accommodate future changes with minimal disruption. Adopting an agile development approach allows teams to respond quickly to evolving needs and incorporate feedback iteratively, ensuring that resources are allocated efficiently and effectively.

  3. Establish iterative approach: It’s not new that the iterative process fosters agility, responsiveness, and adaptability, ensuring that the final product meets user needs effectively while minimizing the risk of costly rework or misalignment with stakeholders’ expectations. The iterative approach will reduce the uncertainty and force the design to absorb more change with less impact.

  4. Continuous improvement to reduce debt: Technical debt accumulates when shortcuts or suboptimal solutions are implemented to meet immediate deadlines or address pressing concerns. Over time, this debt can impede productivity, hinder innovation, and increase the cost of maintenance and future development efforts. By prioritizing continuous improvement and debt reduction, teams can systematically address technical debt through refactoring, optimization, and automation. This proactive approach enhances code quality, improves system reliability, and fosters a culture of innovation and excellence within the development team.

  5. Stop the hype: While it’s tempting to embrace the latest technology trends and tools, it’s essential to evaluate their suitability for the specific problem at hand. Adopting new technologies solely for the sake of hype can introduce unnecessary complexity, increase development overhead, and lead to compatibility issues down the line.

Adress the architecture complexity

Similarly, other articles detail how to stop the architectural complexities, but I often call for the following architectural strategies:

  1. Understanding the complexity: If you didn’t hear about Cynefin, I invite you to dive into it asap.
    Cynefin is a sense-making framework that helps architects understand the nature of the problem they are dealing with. It categorizes problems into five domains: simple, complicated, complex, chaotic, and disorder. By applying the Cynefin framework, architects can determine whether the problem they are facing is predictable or unpredictable, enabling them to choose appropriate strategies and approaches to address it effectively.

  2. Identifying the domains and flows: Architectural design involves understanding the various domains within the system and how they interact with each other. This includes identifying different components, subsystems, and their relationships. Additionally, understanding the flows of data, information, and interactions between these domains is crucial for designing a coherent and efficient architecture. Architects must consider how data and processes flow through the system to ensure optimal performance and maintainability. Architects are the persons who understands and organises the flow of value, in whatever form that value is in.
    As a tip: using and mastering DDD modeling is very helpful for such an excercise.

  3. Gathering data and insights: Effective architectural decision-making requires a thorough understanding of the system’s requirements, constraints, and stakeholders’ needs. This involves gathering data and insights from various sources, including stakeholders, users, and existing systems. By collecting relevant information, architects can make informed decisions and design solutions that align with the organization’s goals and objectives. Those insights can be as simple as the storage size until the throughputs of of the components and their impact on the system’s resources and energy consumption

  4. Divide and conquer:

    The most fundamental problem in software development is complexity. There is only one basic way of dealing with complexity: divide and conquer.
    – Bjarne Stroustrup, C++ inventor

    Breaking down the architectural design into smaller, more manageable components is essential for managing complexity and promoting scalability. This involves decomposing the system into modular components, each responsible for specific functionality. By dividing the architecture into smaller pieces, architects can simplify development, testing, and maintenance, while also enabling parallel development and deployment.

  5. Continuous simplification: Architectural complexity tends to increase over time as systems evolve and grow. Continuous simplification involves regularly reviewing and refining the architecture to remove unnecessary complexity and streamline processes. This may include refactoring code, consolidating redundant components, or optimizing performance bottlenecks. By prioritizing simplicity and elegance in the architectural design, architects can improve system maintainability, scalability, and resilience.

Final thoughts

In navigating the complexities of the system, embracing simplicity, pragmatism, and continuous improvement is paramount. I’m a strong beliver that in prioritizing value, understanding complexity, and leveraging established methodologies, architects and developers can steer clear of over-engineering pitfalls and over complicated design headaches.

In closing, let’s champion simplicity, embrace clarity over complexity, iterate relentlessly, and remain steadfast in the commitment to deliver impactful solutions that resonate with users and drive sustainable growth.