What’s Going Wrong with Your Software Project?

Certain hiccups in the course of my last consulting assignment had prompted me to write on my blog about what ails many, if not most software development projects.

I am cross-posting the article here, for input from a diverse and experienced audience. (Note: Original article can be found here.)

I will take a detour from my usual trend of writing purely about technology, and instead reflect on how and why so many software projects get into a terribly messy state of affairs, often bad enough to be terminated.

I’m trying to understand why, in spite of there being such a wealth of information, training, tools and techniques available on how to run a smooth ship, so little is actually implemented or consistently exercised in most places. I will limit my focus to a subset of problems for which I may be able to hint at a solution through a process of selective elimination of complexities and implementation overheads. Technology can obviously never replace a competent human for addressing personnel-related matters like conflicts, assessment, morale, etc. (although it can still be used to gather crucial data points that help measure individual and team performance), and I am not trying to shoehorn it into such a role.

Every project, small or big, starts out with a lot of energy, zeal and a headstrong optimism among the stakeholders regarding the successful and timely delivery of end products. Assuming there is no deliberate malice or vested interest in failure, no one in a decision-making role would set the ball rolling if they predicted near-certain collapse. How then, do so many projects spiral into an uncontrollable vortex of botched deadlines, leading to increased pressure and reduced morale, leading to more botched deadlines and ultimately, attrition, blame games, broken deliverables and even total shutdown? A bit of googling would provide a flood of information pointing to issues with personnel, techniques, tools and circumstance. I will focus on just a few, outlining how non-invasive, data-driven solutions can help in addressing some of them.

Poor Visibility

An often observed chain of events in many projects I have worked on, goes something like this:

1. Project Manager asks for overall status from team.
2. Team reports completion status and ETA from seat-of-the-pants guesstimates, often intentionally hiding flare spots and backlogs.
3. Project Manager gets an overall picture that’s rosier than what the hard ground realities would depict, and based on this information, assigns further tasks and deadlines to team, often too aggressive to be realistically achievable.
4. Team accepts tasks and deadlines without question, mostly for the same reason they concealed the bad news in the first place.
5. Deadlines fly by, most tasks are in a pitiful state of disarray, and no one can satisfactorily answer why, or what caused this mess in the first place.

Sound familiar? A point to note here is that this can happen even to the most competent and dedicated teams, since effort estimation and progress tracking over long periods of time are continuous, and non-trivial tasks, and not always diligently performed. The delay/failure in pushing out today‘s deliverables may have been caused by an insidious scope creep in the recent past from another project, eating into the time and resources of the project at hand. However, people can’t always connect the dots at a moment’s notice. The data is often just too thinly spread out or too deeply nested in several layers of loosely related or unrelated information, or just too old and foggy, and hence not easily visible/traceable.

Another well-known tendency in any chain of command, is for news to get more and more inaccurate and imprecise, the further it percolates up the org pyramid. This is especially true when the most granular data is hard to get at or comprehend, and people at higher levels rely instead on summary information prepared by those immediately below them. Typically, since no one wants to share bad news too willingly, it is only the good bits that are polished, often exaggerated and passed on for consumption up the corporate ladder, while the bad ones are quietly swept under the rug in the hope that no one will go sniffing about.

I have personally observed this phenomenon even in very small teams with greatly flattened hierarchies. Human psychology is obviously playing a very prominent role here, and one needs to allow for the fact that the mere presence of distorted data does not necessarily imply deliberate falsification.

So what can one do to improve the overall visibility and predictability in attaining project milestones? At the very least, we need to make atomic data accessible to all layers in the management stack. Raw data, when coupled with advanced analytical tools, can reduce or even totally eliminate the chances of error buildup. Reporting and analysis tools abound, but it all has to begin with collecting the raw data from deep down in the trenches.

However, when it comes to collecting data at the grassroots level, one cannot reasonably expect all developers, designers and testers to continually push out micro-updates on the progress of their work items. It is disruptive, inefficient and basically extremely annoying to have to do so on a regular basis. Processes and tools that work in the background to automate the collection of atomic progress data, therefore, are key to enabling optimum visibility across the organization for any software project.

Real and Perceived Overheads in Supervision / Management

I am strongly opposed to micro-management – exercising too much control over how a developer works has been proven to be counter-productive in many studies, and I can corroborate the same from my own experience. However, I don’t lobby for a total absence of management either. Rather, during the life of a running project, I think management needs to perform, with minimal intrusion and high precision, two jobs in particular:

1. Smartly allocate tasks to the resource pool based on availability and effective pairing with core competencies.
2. Identify tasks that are lagging behind schedule, and selectively focus on them to facilitate speedy completion.

Both these duties can be most effectively carried out when there is adequate, up-to-date data available in order to compute resource availability and progress of individual items. A manager’s task can be further assisted by automated planning software, using constraint programming algorithms to suggest time and resource allocation strategies. Driving this automation, again, is the raw data that needs to captured at every stage of the project life-cycle.

Real and Perceived Overheads in Using Project Tracking Tools

I‘ve often noticed, especially in small companies with small teams and few, if any rigidly enforced processes, a distinct tendency for issue trackers and sophisticated project tracking tools to be under-utilized. Very often, at the time of project kickoff, a brand new ’tracker’ is created on a spreadsheet, and a quick RTFM is disbursed to the team before it is let out into the wild.

This ‘tracker’ is diligently used and updated for perhaps the first 2-3 weeks. Following this initial phase of rigorous adherence is a marked decline in its usage. Limited usability, constant maintenance of fickle formulas, lack of scalability and lack of integration with external tools like issue trackers, version control systems and continuous integration systems are just a few reasons why most developers quickly learn to loathe a spreadsheet-based project tracker. For them it is no more than a gigantic, metric butt-load of (barely justifiable) double data entry. Data is already being constantly captured in various centralized as well as local development tools (VCS, CI, Bug Trackers, IDE, etc.). This data can and should be automatically fed into the project tracker so that it remains up-to-date without much manual input. Developers would much prefer to work on a project that tracks itself.

Another common ailment that many non-trivial projects suffer from is the reluctance of non-technical stakeholders, often business/end users, to formally enter their requirements, issues and feedback into an issue tracker, and the confusion that arises from the disconnect. Developers are bad at remembering the minute details of feedback and change requests. Email can work only for the simplest projects, with no more than 5-6 people involved in all. Input often comes in through phone calls, instant messages, SMS and direct conversation. As the project grows, it becomes cumbersome and error-prone to organize and track this barrage of information in an overly simplistic tracking tool.

One can always argue that the end users should not have to learn a new and complicated system. However, the core issue here is that most project and issue tracking systems lack a simple, intuitive user interface, geared towards simplifying the chore of submitting feedback. A system with such an interface in place would be easier to pitch to the layman, and thus enjoy much better adoption. Whenever possible, this input mechanism should be very tightly integrated with the feedback channels offered by the end product. However, an easy interface can only partially lower the barrier to adoption of tracker usage. The rest of the data must be captured, as far as possible, directly from their sources. Integrating a tracker with these channels can be a challenging task though.

Lack of Continous Integration and Automated Build Systems

Continuous integration systems and automated building and health monitoring systems are usually setup only for large projects with a sizable team, in medium to large enterprise scale operations. These tools ensure that mission-critical software goes through a good deal of testing and quality checks before it is shipped out. Smaller projects do not have such elaborate setups in place, mostly due to time and budget constraints, as well a (misguided) belief that they can manage without them.

Truth is, no project lasting longer than a few months and involving more than 2-3 people can be effectively monitored in the long run for new bugs or build breakage in borderline test cases without some process facilitated by a continuous build system. Small projects, in spite of their relative simplicity, are still just as mission-critical to a small company as a big project is to a big company. Hence, it is a mistake to overlook the importance of putting these systems in place.

Summary

Overall, I have not proposed any radically new idea or magic cure to heal an ailing project. I am merely driving at the point that although the answers are out there for solving most problems of project management, they often exist in mutual isolation. The need of the hour is to bring them all under one banner and provide an integrated solution that, while being comprehensive and powerful enough to suit the needs of large and complex projects, is also affordable and easy to setup and use, so that smaller projects can reap the benefits of adopting them as well.

I look forward to understanding what conclusions your individual experiences have led you to.