Applying Evolution to Software Development: Iterative Planning in Software Projects

Introduction:

Software engineering is a field steeped in discipline and grounded in principles. Over the years, the field has developed a rich tapestry of methodologies: approaches, styles, and guidelines, all aimed at improving the way we write and deliver software. These methodologies emerge from the collective experiences of the field and are often seen as vital to the success of software implementation and delivery.

The evolution of the field itself reflects a journey of continuous learning and adaptation. Early, more simple, programming efforts laid the foundation for understanding the complexity and scale of software development. This progression from simplicity to complexity has been marked by an increasing sophistication in our tools and methods for coding and execution. Today, we are able to create and integrate software on a scale and with a complexity that marks a significant departure from our beginnings.

Modern software applications are immensely more complex and larger in scope compared to their early counterparts. When tackling important and valuable software projects, we typically lean on the wisdom of experienced professionals. Their years of navigating the intricacies of software architecture and project management are invaluable in guiding successful outcomes.

In this blog post, I'll focus on delivery, how we can frame it in terms of iterations, how we can draw from biology to help inform what those iterations should look like, and some things I've learned through recent experience.

Section 1: Acceptable States of Evolvable Deliverables

In a previous discussion, I delved into the concept of software evolution, drawing parallels between the development of software and biological evolution. I highlighted how changes in software mirror the gradual, incremental adaptations seen in living organisms.

A deeper appreciation emerges when we realize that evolution operates both at the micro and macro levels. Just as subtle shifts in DNA accumulate over generations to create significant changes in organisms, similar gradual transformations occur in software. Furthermore, it's crucial to recognize that each version or iteration of software represents a complete entity in itself. This understanding is vital for defining what I term as Acceptable States of Evolvable Deliverables.

Lets establish a foundational definition: An acceptable state for a successfully delivered software product is one that offers an improvement in value compared to its previous iteration.

From this definition, we can first infer that an 'acceptable' state must be fully functional. Real-world scenarios do not favor half-finished, partially functional prototypes as final products, especially not for paying customers (barring specific contexts like educational courses or experimental studies).

In essence, your product needs to be operational, it must offer improvements over previous versions, and it should be ready for customer use.

If we consider a greenfield product - i.e. something new - the first deliverable must achieve that minimal viable product (MVP) state. If we consider modifications to an existing product - i.e. a brownfield product - the first deliverable must be an improvement over the previous version. In both cases, the deliverable must be fully functional and ready for use by the customer.

This perspective can significantly shift the way we approach product delivery.

Section 2: Iterative Planning

When we consider the evolution of software in the context of product creation, it's useful to draw a parallel with the natural world.

Think of the development of animal species: they didn't appear overnight in their current forms, nor did they exist as incomplete fragments at any stage. Each time an animal is born, its a fully functional complete animal. This has applied to all evolutionary stages of life – from the primordial self-replicating genetic molecules, to the early worms and fish... to the the mighty dinosaurs and eventually humans with their technological achievements. Each stage in the evolutionary ladder was a functional, deliverable unit in its own right.

Let's refocus this perspective on product development.

The process of building and delivering products can be approached in at least two distinct ways.

The image above (borrowed from Henrik Knilberg) shows the two approaches I'd like to discuss. (Btw - I strongly recommend this post - its an excellent read!)

All or Nothing Approach (Top of image)

On the top, we have the assembly of a car - and only a car - where a wheel is created, then a chassis, then a body, and then the final car.

This approach, while ambitious, carries significant risk and offers limited immediate value. Its major drawback is the absence of a valuable product at any stage until completion - a process that can span months or years. This is inherently risky - what if a technical roadblock is encountered? What if you lose funding? What if your timeline blows out? What if the product is no longer relevant by the time it's finished? What if the product is not what the customer wants? There are a lot of ways this approach can fail.

This is the All or Nothing approach.

The Iterative Approach (Bottom of image)

On the bottom, we see an evolution of the product which finally becomes a car.

This approach involves developing product in such a way that each iteration results in a fully functional and deliverable state. For example, a fully functioning skateboard, then a scooter, and so on. This is achieved by defining the final product and then planning iterations that each contribute an incremental, evolutionary improvement in value. The challege of course is to be able to envision the final product and then plan the iterations that will lead to it, where each iteration is an acceptatable deliverable state.

This is the Iterative Planning approach.

Each of these approaches has its own set of advantages and disadvantages, requiring careful consideration of the context in which they are applied. For instance, small tactical tasks, bug fixes, or minor refactors might not necessitate iterations.

Section 3: The Advantages of Iterative Planning

In scenarios where iterations are beneficial - there are numerous advantages to be had.

Clear Stopping Points

When iterations produce fully viable incremental deliverables - engineering and product teams can collaboratively assess the value of proceeding to the next iteration at any time. Iterabtive planning incidently produces a series of clear stopping points. This is in contrast to the 'All or Nothing' approach where the only stopping point is the final product.

Early Customer Feedback

Early customer engagement and feedback is often vital to the success of a product because it allows teams to ensure the product develops in the right direction. As iterations are delivered, options like A/B testing between current and upcoming versions, or between two competing alternatives for the next iteration, become available.

Reduction of overall waste

A noticeable benefit of this method is the overall reduction in waste over the project's lifespan. Despite some elements being discarded during the evolution of the product or feature, each component contributes towards a functional product at the end of each iteration. There is lower potential for the development of unnecessary pieces or features that may not be used in the final product.

Composability of Work

Planned work should typically be decomposed into units of composable work in a way that doesn't encapsulate too much work in a single unit. There is an art to decomposing work into a units of appropriate granularity, however when done well, you're able to recompose your work in ways that make sense as you move through iterations. Having clear iterations give you a platform to re-evaluate and re-compose your work as you go.

Visibility of Work

Iterative development enhances the visibility of work progress. By breaking down the project into smaller, manageable iterations, it becomes easier to track and measure the flow rate of work. This visibility is crucial for project management as it allows for better forecasting, resource allocation, and identifying bottlenecks early in the process. With each iteration, the team can evaluate its pace, efficiency, and effectiveness, making necessary adjustments to improve subsequent iterations.

Enhanced Team Satisfaction

Adopting an iterative approach can also significantly boost team morale and satisfaction. When work is broken down into smaller, achievable goals, teams experience the satisfaction of completing tasks and ‘moving them across the board’ more frequently. This sense of progress is essential for maintaining high levels of motivation and engagement among team members. Furthermore, seeing tangible results at the end of each iteration validates their effort and fosters a sense of accomplishment.

In summary, iterative planning in software development offers a robust framework for delivering products that are not only high in quality but also aligned with customer needs and market demands. By focusing on creating deliverable and functional units at every stage, ensuring the composability of work, maintaining visibility of the workflow, and enhancing team satisfaction, this approach paves the way for a more efficient, flexible, and successful software development process.

Section 4: Knowing When to Apply Iterative Planning

Understanding when to employ iterative planning is as crucial as the methodology itself. It's essential to remember that no single approach is absolute or universally applicable. Sometimes, iterative planning may not be the most efficient path, especially considering that planning itself incurs overhead. Applying this method to tasks that are overly simple, for instance, can create unnecessary complexity and waste valuable resources.

Some good guidelines on when to apply iterative planning include:

Complex Projects: Iterative planning is particularly effective for complex projects where the end goal might evolve over time. It allows for flexibility and adaptability, essential in managing intricate or multifaceted tasks.

Projects with Uncertain Requirements: If you're dealing with a project where requirements are expected to change or are not fully defined from the start, iterative planning can help. It allows for regular reassessment and realignment with evolving needs.

Long-Term Projects: For projects that span a considerable duration, iterative planning helps in maintaining momentum and clarity. It breaks down the overwhelming scope into manageable parts, making progress more tangible and measurable.

High-Risk Projects: In scenarios where there's a high risk of failure or uncertainty, iterative planning allows for regular evaluation and course correction. This minimizes risk by avoiding long periods of work without review or testing.

Customer-Focused Projects: When a project's success is heavily dependent on user feedback or market trends, iterative planning is invaluable. It facilitates early and continuous user engagement, ensuring the product remains aligned with user needs and preferences.

Projects Requiring Team Collaboration: Iterative planning can enhance collaboration in teams, especially in cross-functional setups. It provides regular checkpoints for team members to sync up, review progress, and align on goals.

However, it’s important to be judicious in applying iterative planning. For straightforward tasks with clear objectives and known outcomes, a more linear approach might be more efficient. The key is to assess each project on its own merits and choose the methodology that optimizes time, resources, and effort while delivering the best possible outcome.

Section 5: Addressing Preconceptions and Biases from the start

Addressing preconceptions and biases you or the team have about the project is crucial in any project, especially when they may affect our perception of its potential success or failure. A personal experience of mine with a signal detection project highlights this point. Despite significant doubts about achieving the necessary sensitivity for detecting a benchmark signal, I realized my concerns were, to a large extent, personal preconceptions. My reservations, mostly unrecognized by others, influenced my contributions more than they should have.

It’s essential to distribute such concerns across the team. The team as a whole should actively engage in identifying potential obstacles, contemplating their implications, and devising strategies to mitigate them. By sharing these issues, they become collective challenges rather than individual burdens.

The real power in this approach lies in not having to ‘own’ problems alone. While individual team members can investigate and collaborate on potential issues, they are not solely responsible for the outcomes of these challenges. For instance, if there’s a concern about the project's feasibility – particularly for reasons beyond the team's control – it doesn't necessitate undue stress for any single member. Unforeseen issues, when they arise, can be addressed in a structured and collective manner.

This approach of distributing concerns and responsibilities fosters a more collaborative, less stressful environment. It encourages open communication and shared problem-solving, ensuring that challenges are approached with collective wisdom rather than isolated efforts.

Conclusion

The journey of software development, much like any creative or innovative process, is filled with complexities and nuances. This blog post has navigated through the essential facets of iterative planning, understanding its application, and addressing preconceptions within team dynamics. We've seen how iterative planning is not a one-size-fits-all solution but rather a strategic approach best suited for complex, long-term, and high-risk projects. It emphasizes the importance of adaptability, frequent reassessment, and the value of incremental progress.

Equally important is the understanding of when and how to apply iterative planning. By recognizing the specific contexts and projects where this approach yields the most benefit, teams can avoid unnecessary overhead and focus their energies where it matters most. This discernment is key to efficient and effective software development.

Lastly, we touched upon the significance of managing preconceptions and distributing responsibilities within a team. By fostering an environment where concerns and challenges are shared and addressed collectively, teams can leverage diverse perspectives and expertise, ultimately leading to more robust and innovative solutions.

The realm of software development is ever-evolving, and so are the methodologies we use to navigate its challenges. Iterative planning, with its focus on flexibility, incremental improvement, and collaborative problem-solving, offers a powerful framework for teams aiming to deliver high-quality software in an ever-changing technological landscape. As we continue to learn and adapt, these approaches will not only guide us in creating successful software solutions but also in fostering a more dynamic and resilient software development culture.