Shotcomings or Pitfalls of Traditional Waterfall approach

The essence of waterfall software development is that complex software systems can be built in a sequential, phase-wise manner where all of the requirements are gathered at the beginning, all of the design is completed next, and finally the master design is implemented into production quality software. This approach holds that complex systems can be built in a single pass, without going back and revisiting requirements or design ideas in light of changing business or technology conditions. It was first introduced in an article written by Winston Royce in 1970, primarily intended for use in government projects.
Waterfall equates software development to a production line conveyor belt.
“Requirements analysts” compile the system specifications until they pass the finished requirements specification document to “software designers” who plan the software system and create diagrams documenting how the code should be written. The design diagrams are then passed to the “developers” who implement the code from the design
Under the waterfall approach, traditional IT managers have made valiant efforts to craft and adhere to large-scale development plans. These plans are typically laid out in advance of development projects using Gantt or M.P.P (Microsoft Project Plan) or PERT charts to map detailed tasks and dependencies for each member of the development group months or years down the line. However, studies of past software projects show that only 9% to 16% are considered on-time and on-budget.

Up-front Requirements Analysis
What are requirements? From the stakeholder’s perspective, the requirements are the features and specifications of the system. Requirements define what developers are to build. For example, the system must have a web site with e-commerce capability that can handle 10,000 purchases per hour, or the system must be accessible 99.999% of the time.
One of the biggest problems with waterfall is that it assumes that all project requirements can be accurately gathered at the beginning of the project. In waterfall mode the first phase represents the requirements analysis phase of a software development project. Analysts slave for weeks or months compiling everything they can gleam about the proposed system into comprehensive
“Software Requirements Specification” (SRS) documents. Once finished, the SRS is sent over the fence to the designers while the requirements analysts go to work on the next project.
Imagine a scenario where you engage a software group to build a critical software system. Do you think you could provide every last detail the developers need to know right off the bat? I have yet to encounter such a customer and I am hard pressed to think I ever will. As a start, consider the areas that must be addressed: 

• Business rules and exceptions
• Scalability and concurrent user support
• Browser or OS support
• User roles and restrictions
• User interface standards. 

In fact, it is inevitable that attempts at up-front requirements specification will leave out some very important details simply because the stakeholders cannot tell developers everything about the system at the beginning of the project. This means that the requirements typically change outside of the requirements phase in the form of “change orders”, and in many waterfall projects this can be very costly. By virtue of a requirements change, the intricately planned design can be affected dramatically, which will in turn affect any implementation and test strategies. The cost of change in a waterfall project increases exponentially over time because the developer is forced to make any and all project decisions at the beginning of the project.
What if your business needs are still emerging and certain aspects of the system are rapidly changing or cannot be defined yet? Business climates and objectives often changrapidly, especially in today’s age of instant information. Can you afford to lock your business into a rigid long-term project where the cost of change grows exponentially? For example, a national test preparation organization commissioned my company to build a simulator for an upcoming standardized test. Since the test itself had not been released yet, the types of questions that would appear on the test were unknown when we started development. But the system had to be done shortly after the tests were released. Markets are forcing the software development community to respond with flexible development plans that flatten the cost of change.
People need to see and feel something before they really know what they want. The “I’ll Know it When I See It” (IKIWISI) law says that software development customers can better describe what they really want after seeing and trying working, functional software. I often use a “drawing” analogy to help explain this effect. Although I’m a terrible artist, when I draw a picture I need to see the drawing as I progress. If I tried to close my eyes and draw the same picture, it would prove far less successful. But this is what waterfall asks customers to do: specify the entire system without having a chance to periodically see the progress and make adjustments to the requirements as needed. Waterfall is an “over the fence” approach; the requirements are solicited from the user and some time later the finished product is presented to the user. This is entirely unnatural because customers find it difficult to specify software perfectly without seeing it evolve and progress.
The problem of undefined, changing, and emerging requirements presents a very large challenge to waterfall projects because by definition all requirements must be captured up-front at the risk of costly changes later.

The waterfall methodology assumes that up-front planning is enough to take into account all variables that could impact the development process. In fact, waterfall projects allocate copious effort detailing every possible risk, mitigation plan, and contingency. But is it possible to predict any and all variables that could possibly affect a software project? The empirical
answer is “no” considering the limited success of waterfall projects.

Waterfall therefore equates software development to an assembly line; defined processes can be established that, when used sequentially, result in a successful project each time. The first step is X, the second is Y, and the result is always Z.

Source: http://danube.com/docs/Intro_to_Agile.pdf

An Introduction to Agile Software Development

Incremental and Iterative Development

The simple ability to revisit the “phases” (Requirement collection, Analysis, Design, Development, Testing, Deply) of development dramatically improves project efficiency.

The idea of revisiting phases over and over is called “incremental and iterative development” (IID). The development lifecycle is cut up into increments or “iterations” and each iteration touches on each of the traditional “phases” of development. For example, with IID requirements is an ongoing process that is periodically revisited. As new requirements surface and as the scope changes, IID processes continually capture the requirements iteration after iteration Interestingly, 

Agile project management approaches balance the four variables in software development while keeping in mind the limits associated with new product development. In software development there are four broad control factors. These factors are interconnected, when one changes at least one other factor must also change.

• Cost – or Effort. Available money impacts the amount of effort put into the system.
• Schedule – A software project is impacted as the timeline is changed.
• Requirements – The scope of the work that needs to be done can be increased or decreased to affect the project.
• Quality – Cut corners by reducing quality.

Because software development is often considered a sequential, linear process, middle and upper management often assumes that all four of these factors could be dictated to the development team under the waterfall approach. However software development cannot be described by a simple linear process because it cannot be predicted accurately in advance. It is therefore unreasonable to assume that management can control all four of these factors. In reality, management can pick values for three of the four factors at most, and the development process dictates the fourth.this expectation of full control unrealistic.

Lean Thinking

Another effective way to analyze how Agile methods increase efficiencies is to apply

lean manufacturing

      Consider for example the “small batch” principle: things produced in smaller batches are of higher quality and efficiency because the feedback loop is short; controls can be adjusted more frequently, and resources are utilized efficiently to avoid “queuing” (see “queuing theory” and the theory of constraints).

     Second, Agile methods encourage delaying irreversible decisions until the last responsible moment. Many software development organizations that implement Agile software development are finding they get something they never expected: options. Rather than locking into decisions at the beginning of a project, organizations can reduce risks by leaving options open to decide at a better time when more accurate information is available.

Third, the concept of frequent or continuous integration keep software development teams synchronized. Teams can work independently for a while but the code base never diverges for long periods of time, thereby reducing the risks associated with large integrations at the tail end of projects.principles to software development. The highly complex and uncertain nature of software development makes

Agile Requirements: A Focus on Business ROI

               Agile projects avoid “up-front” requirements gathering for the reasons stated above:

customers cannot effectively produce all requirements in high enough detail for implementation to occur at the beginning of a project. Customers may not want to make decisions about the system until they have more information. Agile values a high visibility and customer involvement. The frequent demonstration and release of software common in Agile approaches gives customers a chance to “try software” periodically and provide feedback. Agile helps companies produce the “right product”. An iterative approach allows customers to delay decisions as well.

            One of the biggest advantages to IID is that work can begin before all of the requirements are known. Many organizations are not fully staffed with business analysts cranking out reams of requirements specs.

            IID is ideally suited then to take on bite-sized chunks of requirements that the customer can easily digest.

How do agile prioritize work?

This is largely because the unused features were specified in some upfront plan before the ratio of their cost to value was considered or even knowable. Focusing on high business value features first is therefore a critical component of efficient Agile development. Because we can change direction rapidly (every iteration) and the cost of change is low, there is a valuable opportunity for the customer to re-examine business factors at the

beginning of each iteration to select features for inclusion in the current iteration according to business ROI. Of course, the development team must bring technical risks to the customer, butin the end it is the customer that decides what the development team builds.

Source: http://danube.com/docs/Intro_to_Agile.pdf