Software Testing Fundamentals—Concepts, Roles, and Terminology

SOFTWARE TESTING—WHAT, WHY, AND WHO WHAT IS SOFTWARE TESTING?

1. Meets the business and technical requirements that guided its design and development, and
2. Works as expected.

Software testing also identifies important defects, flaws, or errors in the application code that must be fixed. The modifier "important" in the previous sentence is, well, important because defects must be categorized by severity (more on this later).

During test planning we decide what an important defect is by reviewing the requirements and design documents with an eye towards answering the question "Important to whom?" Generally speaking, an important defect is one that from the customer’s perspective affects the usability or functionality of the application. Using colors for a traffic lighting scheme in a desktop dashboard may be a no-brainer during requirements definition and easily implemented during development but in fact may not be entirely workable if during testing we discover that the primary business sponsor is color blind. Suddenly, it becomes an important defect. (About 8% of men and .4% of women have some form of color blindness.)

The quality assurance aspect of software development—documenting the degree to which the developers followed corporate standard processes or best practices—is not addressed in this paper because assuring quality is not a responsibility of the testing team. The testing team cannot improve quality; they can only measure it, although it can be argued that doing things like designing tests before coding begins will improve quality because the coders can then use that information while thinking about their designs and during coding and debugging.

Software testing has three main purposes: verification, validation, and defect finding.

♦ The verification process confirms that the software meets its technical specifications. A "specification" is a description of a function in terms of a measurable output value given a specific input value under specific preconditions. A simple specification may be along the line of "a SQL query retrieving data for a single account against the multi-month account-summary table must return these eight fields <list> ordered by month within 3 seconds of submission."

♦ The validation process confirms that the software meets the business requirements. A simple example of a business requirement is "After choosing a branch office name, information about the branch’s customer account managers will appear in a new window. The window will present manager identification and summary information about each manager’s customer base: <list of data elements>." Other requirements provide details on how the data will be summarized, formatted and displayed.

♦ A defect is a variance between the expected and actual result. The defect’s ultimate source may be traced to a fault introduced in the specification, design, or development (coding) phases.

WHY DO SOFTWARE TESTING?

"A clever person solves a problem. A wise person avoids it."

Why test software? "To find the bugs!" is the instinctive response and many people, developers and programmers included, think that that’s what debugging during development and code reviews is for, so formal testing is redundant at best. But a "bug" is really a problem in the code; software testing is focused on finding defects in the final product.

Here are some important defects that better testing would have found.

♦ In February 2003 the U.S. Treasury Department mailed 50,000 Social Security checks without a beneficiary name. A spokesperson said that the missing names were due to a software program maintenance error.

♦ In July 2001 a "serious flaw" was found in off-the-shelf software that had long been used in systems for tracking U.S. nuclear materials. The software had recently been donated to another country and scientists in that country discovered the problem and told U.S. officials about it.

♦ In October 1999 the $125 million NASA Mars Climate Orbiter—an interplanetary weather satellite—was lost in space due to a data conversion error. Investigators discovered that software on the spacecraft performed certain calculations in English units (yards) when it should have used metric units (meters).

♦ In June 1996 the first flight of the European Space Agency's Ariane 5 rocket failed shortly after launching, resulting in an uninsured loss of $500,000,000. The disaster was traced to the lack of exception handling for a floating-point error when a 64-bit integer was converted to a 16-bit signed integer.

Software testing answers questions that development testing and code reviews can’t.

♦ Does it really work as expected?

♦ Does it meet the users’ requirements?

♦ Is it what the users expect?

♦ Do the users like it?

♦ Is it compatible with our other systems?

♦ How does it perform?

♦ How does it scale when more users are added?

♦ Which areas need more work?

♦ Is it ready for release?

What can we do with the answers to these questions?

♦ Save time and money by identifying defects early

♦ Avoid or reduce development downtime

♦ Provide better customer service by building a better application

♦ Know that we’ve satisfied our users’ requirements

♦ Build a list of desired modifications and enhancements for later versions

♦ Identify and catalog reusable modules and components

♦ Identify areas where programmers and developers need training

WHAT DO WE TEST?

First, test what’s important. Focus on the "core functionality"—the parts that are critical or popular—before looking at the ‘nice to have’ features. Concentrate on the application’s capabilities in common usage situations before going on to unlikely situations. For example, if the application retrieves data and performance is important, test reasonable queries with a normal load on the server before going on to unlikely ones at peak usage times. It’s worth saying again: focus on what’s important. Good business requirements will tell you what’s important. Software testing is a process of verifying and validating that a software application or program

The value of software testing is that it goes far beyond testing the underlying code. It also examines the functional behavior of the application. Behavior is a function of the code, but it doesn’t always follow that if the behavior is "bad" then the code is bad. It’s entirely possible that the code is solid but the requirements were inaccurately or incompletely collected and communicated. It’s entirely possible that the application can be doing exactly what we’re telling it to do but we’re not telling it to do the right thing.

A comprehensive testing regime examines all components associated with the application. Even more, testing provides an opportunity to validate and verify things like the assumptions that went into the requirements, the appropriateness of the systems that the application is to run on, and the manuals and documentation that accompany the application. More likely though, unless your organization does true "software engineering" the focus will be on the functionality and reliability of application itself.

Testing can involve some or all of the following factors. The more, the better. ♦ Business requirements

♦ Functional design requirements

♦ Technical design requirements

♦ Regulatory requirements

♦ Programmer code

♦ Systems administration standards and restrictions

♦ Corporate standards

♦ Professional or trade association best practices

♦ Hardware configuration

♦ Cultural issues and language differences

WHO DOES THE TESTING?

Software testing is not a one person job. It takes a team, but the team may be larger or smaller depending on the size and complexity of the application being tested. The programmer(s) who wrote the application should have a reduced role in the testing if possible. The concern here is that they’re already so intimately involved with the product and "know" that it works that they may not be able to take an unbiased look at the results of their labors. 

Testers must be cautious, curious, critical but non-judgmental, and good communicators. One part of their job is to ask questions that the developers might find not be able to ask themselves or are awkward, irritating, insulting or even threatening to the developers. 

♦ How well does it work? 

♦ What does it mean to you that "it works"? 

♦ How do you know it works? What evidence do you have? 

♦ In what ways could it seem to work but still have something wrong? 

♦ In what ways could it seem to not work but really be working? 

♦ What might cause it to not to work well?

A good developer does not necessarily make a good tester and vice versa, but testers and developers do share at least one major trait—they itch to get their hands on the keyboard. As laudable as this may be, being in a hurry to start can cause important design work to be glossed over and so special, subtle situations might be missed that would otherwise be identified in planning. Like code reviews, test design reviews are a good sanity check and well worth the time and effort. 

Testers are the only IT people who will use the system as heavily an expert user on the business side. User testing almost invariably recruits too many novice business users because they’re available and the application must be usable by them. The problem is that novices don’t have the business experience that the expert users have and might not recognize that something is wrong. Testers from IT must find the defects that only the expert users will find because the experts may not report problems if they’ve learned that it's not worth their time or trouble.

Key Players and Their Roles
 

1. Business sponsor(s) and partners
                  ♦ Provides funding
                  ♦ Specifies requirements and deliverables
                  ♦ Approves changes and some test results
2. Project manager Plans and manages the project
3. Software developer(s)
                  ♦ Designs, codes, and builds the application
                  ♦ Participates in code reviews and testing
                  ♦ Fixes bugs, defects, and shortcomings
4. Testing Coordinator(s) Creates test plans and test specifications based on the requirements and functional, and technical documents 

Tester(s) Executes the tests and documents results  

Iron Man 2 (Three-Disc Blu-ray/DVD Combo + Digital Copy) 

The Adventures of Sherlock Holmes 

Robin Hood: Unrated Director's Cut (Blu-ray/DVD Combo + Digital Copy) 

Shotcomings or Pitfalls of Traditional Waterfall approach

The essence of waterfall software development is that complex software systems can be built in a sequential, phase-wise manner where all of the requirements are gathered at the beginning, all of the design is completed next, and finally the master design is implemented into production quality software. This approach holds that complex systems can be built in a single pass, without going back and revisiting requirements or design ideas in light of changing business or technology conditions. It was first introduced in an article written by Winston Royce in 1970, primarily intended for use in government projects.
Waterfall equates software development to a production line conveyor belt.
“Requirements analysts” compile the system specifications until they pass the finished requirements specification document to “software designers” who plan the software system and create diagrams documenting how the code should be written. The design diagrams are then passed to the “developers” who implement the code from the design
Under the waterfall approach, traditional IT managers have made valiant efforts to craft and adhere to large-scale development plans. These plans are typically laid out in advance of development projects using Gantt or M.P.P (Microsoft Project Plan) or PERT charts to map detailed tasks and dependencies for each member of the development group months or years down the line. However, studies of past software projects show that only 9% to 16% are considered on-time and on-budget.

Up-front Requirements Analysis
What are requirements? From the stakeholder’s perspective, the requirements are the features and specifications of the system. Requirements define what developers are to build. For example, the system must have a web site with e-commerce capability that can handle 10,000 purchases per hour, or the system must be accessible 99.999% of the time.
One of the biggest problems with waterfall is that it assumes that all project requirements can be accurately gathered at the beginning of the project. In waterfall mode the first phase represents the requirements analysis phase of a software development project. Analysts slave for weeks or months compiling everything they can gleam about the proposed system into comprehensive
“Software Requirements Specification” (SRS) documents. Once finished, the SRS is sent over the fence to the designers while the requirements analysts go to work on the next project.
Imagine a scenario where you engage a software group to build a critical software system. Do you think you could provide every last detail the developers need to know right off the bat? I have yet to encounter such a customer and I am hard pressed to think I ever will. As a start, consider the areas that must be addressed: 

• Business rules and exceptions
• Scalability and concurrent user support
• Browser or OS support
• User roles and restrictions
• User interface standards. 

In fact, it is inevitable that attempts at up-front requirements specification will leave out some very important details simply because the stakeholders cannot tell developers everything about the system at the beginning of the project. This means that the requirements typically change outside of the requirements phase in the form of “change orders”, and in many waterfall projects this can be very costly. By virtue of a requirements change, the intricately planned design can be affected dramatically, which will in turn affect any implementation and test strategies. The cost of change in a waterfall project increases exponentially over time because the developer is forced to make any and all project decisions at the beginning of the project.
What if your business needs are still emerging and certain aspects of the system are rapidly changing or cannot be defined yet? Business climates and objectives often changrapidly, especially in today’s age of instant information. Can you afford to lock your business into a rigid long-term project where the cost of change grows exponentially? For example, a national test preparation organization commissioned my company to build a simulator for an upcoming standardized test. Since the test itself had not been released yet, the types of questions that would appear on the test were unknown when we started development. But the system had to be done shortly after the tests were released. Markets are forcing the software development community to respond with flexible development plans that flatten the cost of change.
People need to see and feel something before they really know what they want. The “I’ll Know it When I See It” (IKIWISI) law says that software development customers can better describe what they really want after seeing and trying working, functional software. I often use a “drawing” analogy to help explain this effect. Although I’m a terrible artist, when I draw a picture I need to see the drawing as I progress. If I tried to close my eyes and draw the same picture, it would prove far less successful. But this is what waterfall asks customers to do: specify the entire system without having a chance to periodically see the progress and make adjustments to the requirements as needed. Waterfall is an “over the fence” approach; the requirements are solicited from the user and some time later the finished product is presented to the user. This is entirely unnatural because customers find it difficult to specify software perfectly without seeing it evolve and progress.
The problem of undefined, changing, and emerging requirements presents a very large challenge to waterfall projects because by definition all requirements must be captured up-front at the risk of costly changes later.

The waterfall methodology assumes that up-front planning is enough to take into account all variables that could impact the development process. In fact, waterfall projects allocate copious effort detailing every possible risk, mitigation plan, and contingency. But is it possible to predict any and all variables that could possibly affect a software project? The empirical
answer is “no” considering the limited success of waterfall projects.

Waterfall therefore equates software development to an assembly line; defined processes can be established that, when used sequentially, result in a successful project each time. The first step is X, the second is Y, and the result is always Z.

Source: http://danube.com/docs/Intro_to_Agile.pdf

SQL Server Programming Guidelines

SQL Server Programming Guidelines

1.0 Introduction
This section provides guidelines and best practices for SQL Server programming. Guidelines and best practices should be followed as a general rule, but it is understood that exception situations may exist. Developers must be prepared to provide a justification for any exceptions.
2.0 Code Readability and Format
• Write comments in your stored procedures, triggers and SQL batches generously, whenever something is not very obvious. This helps other programmers understand your code. Don't worry about the length of the comments, as it won't impact the performance, unlike interpreted languages (e.g., ASP 2.0).
• Always use case consistently in your code. On a case insensitive server, your code might work fine, but it will fail on a case sensitive SQL Server if the code is not consistent in case. For example, if you create a table in SQL Server or a database that has a case-sensitive or binary sort order, all references to the
table must use the same case that was specified in the CREATE TABLE statement. If you name the table "MyTable" in the CREATE TABLE statement and use "mytable" in the SELECT statement, you get an "object not found" error.
• Do not use column numbers in the ORDER BY clause. In the following examples, note that the second query is more readable than the first. Example 1:
SELECT OrderID, OrderDate FROM Orders ORDER BY 2
Example 2:
SELECT OrderID, OrderDate FROM Orders ORDER BY OrderDate
• Use the more readable ANSI-Standard Join clauses instead of the old style joins. With ANSI joins, the WHERE clause is used only for filtering data. With older style joins, the WHERE clause handles both the join condition and filtering data. The first of the following two examples shows the old style join syntax, while the second one shows the new ANSI join syntax. Example 1:
SELECT a.au_id, t.title FROM titles t, authors a, titleauthor ta WHERE a.au_id = ta.au_id AND ta.title_id = t.title_id AND t.title LIKE '%Computer%'
Example 2:
SELECT a.au_id, t.title FROM authors a INNER JOIN titleauthor ta ON a.au_id = ta.au_id INNER JOIN titles t ON ta.title_id = t.title_id WHERE t.title LIKE '%Computer%'
• To make SQL statements more readable, start each clause on a new line and indent when needed. E.g.:
SELECT title_id, title FROM titles WHERE title LIKE '%Computer%' AND title LIKE '%cook%'
• As is true with any other programming language, do not use GOTO, or use it sparingly. Excessive usage of GOTO can lead to hard-to-read-and-understand code.
3.0 Datatypes
• Use User Defined Datatypes if a particular column repeats in multiple tables so that the datatype of that column is consistent across all your tables.
• Use the CHAR data type for a column only when the column is non-nullable. If a CHAR column is nullable, it is treated as a fixed length column in SQL Server 7.0+. So, a CHAR(100), when NULL, will eat up 100 bytes, resulting in space wastage. Use VARCHAR(100) in this situation. Of course, variable length columns do have a very little processing overhead over fixed length columns. Carefully choose between CHAR and VARCHAR depending upon the length of the data you are going to store.
• Use Unicode datatypes, like NCHAR, NVARCHAR, or NTEXT, if your database is going to store not just plain English characters but a variety of characters used all over the world. Use these datatypes only when they are absolutely needed as they use twice as much space as non-Unicode datatypes.
• Try not to use TEXT or NTEXT datatypes for storing large blocks of textual data. The TEXT datatype has some inherent problems associated with it. For example, you cannot directly write or update text data using the INSERT or UPDATE statements. Instead, you have to use special statements like READTEXT, WRITETEXT and UPDATETEXT. There are also a lot of bugs associated with replicating tables containing text columns. So, if you don't have to store more than 8KB of text, use CHAR(8000) or VARCHAR(8000) datatypes instead.
4.0 Stored Procedures
• Always add an @Debug parameter to your stored procedures. This can be a BIT data type. When a '1' is passed for this parameter, print all the intermediate results, variable contents using SELECT or PRINT statements. When '0' is passed do not print anything. This helps in quickly debugging stored procedures as you don't have to add and remove these PRINT/SELECT statements before and after troubleshooting problems.
• Do not call functions repeatedly within your stored procedures, triggers, functions and batches. For example, you might need the length of a string variable in many places of your procedure, but don't call the LEN function whenever it's needed. Instead, call the LEN function once, and store the result in a variable for later use.
• Make sure your stored procedures always return a value indicating their status. Standardize on the return values of stored procedures for success and failures. The RETURN statement is meant for returning the execution status only, but not data. If you need to return data, use OUTPUT parameters.
• If your stored procedure always returns a single row resultset, consider returning the resultset using OUTPUT parameters instead of a SELECT statement, as ADO handles output parameters faster than resultsets returned by SELECT statements.
• Do not prefix your stored procedure names with 'sp_'. The prefix 'sp_' is reserved for system stored procedure that ship with SQL Server. Whenever SQL Server encounters a procedure name starting with 'sp_', it first tries to locate the procedure in the master database, then it looks for any qualifiers (database, owner) provided, then it tries dbo as the owner. You can save time in locating the stored procedure by avoiding the 'sp_' prefix.
• Do not let your front-end applications query/manipulate the data directly using SELECT or INSERT/UPDATE/DELETE statements. Instead, create stored procedures and let your applications access these stored procedures. This keeps the data access clean and consistent across all the modules of your application, while at the same time centralizing the business logic within the database.
5.0 Performance Considerations
• While designing your database, keep performance in mind. You can't really tune performance later when your database is in production as it involves rebuilding tables and indexes, re-writing queries, etc. Use the graphical execution plan in Query Analyzer or SHOWPLAN_TEXT or SHOWPLAN_ALL commands to analyze your queries. Make sure your queries do an "Index seek" instead of an "Index scan" or a "Table scan." A table scan or an index scan should be avoided where possible. Choose the right indexes on the right columns.
• Initially, your data should be normalized at least to the third normal form. If you then need to denormalize some of the data to improve performance, you may do so. There should be a documented rationale for all denormalization activities.
• Do not use 'SELECT *' in your queries. Always write the required column names after the SELECT statement, as in the following example:
SELECT CustomerID, CustomerFirstName, City
This technique results in reduced disk I/O and better performance.
• Avoid the creation of temporary tables while processing data as much as possible, as creating a temporary table means more disk I/O. Consider using advanced SQL, views, SQL Server 2000 table variable, or derived tables instead of temporary tables.
• Try to avoid wildcard characters at the beginning of a word while searching using the LIKE keyword as that results in a full table scan, which defeats the purpose of an index. The first example below results in an index scan, while the second example results in an index seek.
Example 1:
SELECT LocationID FROM Locations WHERE Specialties LIKE '%pples'
Example 2:
SELECT LocationID FROM Locations WHERE Specialties LIKE 'A%s'

The use of functions in SELECT statements will not take advantage of indexing.
• Also avoid searching using not equals operators (<> and NOT) as they result in table and index scans.
• Use derived tables wherever possible, as they perform better. Consider the following query to find the second highest salary from the Employees table:

SELECT MIN(Salary) FROM Employees WHERE EmpID IN (SELECT TOP 2 EmpID FROM Employees ORDER BY Salary Desc)

The same query can be re-written using a derived table, as shown below, and it performs twice as fast as the above query:

SELECT MIN(Salary) FROM (SELECT TOP 2 Salary FROM Employees ORDER BY Salary Desc) AS A

This is just an example and your results might differ in different scenarios depending on the database design, indexes, volume of data, etc. So, test all the possible ways a query could be written and go with the most efficient one.
• Use SET NOCOUNT ON at the beginning of your SQL batches, stored procedures and triggers in production environments, as this suppresses messages like '(1 row(s) affected)' after executing INSERT, UPDATE, DELETE and SELECT statements. This improves the performance of stored procedures by reducing network traffic.

Perform all your referential integrity checks and data validations using constraints (foreign key and check constraints) instead of triggers, as they are faster. Use triggers only for auditing, custom tasks and validations that cannot be performed using constraints. Constraints save you time as well, as you don't have to write code for these validations, allowing the RDBMS to do all the work for you.
6.0 Miscellaneous Topics
• Try to avoid server side cursors as much as possible. Always stick to a "set-based approach" instead of a "procedural approach" for accessing and manipulating data. Cursors can often be avoided by using SELECT statements instead. If a cursor is unavoidable, use a WHILE loop instead. A WHILE loop is always faster than a cursor. For a WHILE loop to replace a cursor you need a column (primary key or unique key) to identify each row uniquely. Every table must have a primary or unique key in any case.
• Views are generally used to show specific data to specific users based on their interest. Views are also used to restrict access to the base tables by granting permission only on views. Yet another significant use of views is that they simplify your queries. Incorporate your frequently required, complicated joins and calculations into a view so that you don't have to repeat those joins/calculations in all your queries. Instead, just select from the view.
• If you have a choice, do not store binary or image files (Binary Large Objects or BLOBs) inside the database. Instead, store the path to the binary or image file in the database and use that as a pointer to the actual binary file stored elsewhere on a server. Retrieving and manipulating these large binary files is better performed outside the database. Keep in mind that a database is not meant for storing files.
• Avoid dynamic SQL statements as much as possible. Dynamic SQL tends to be slower than static SQL, as SQL Server must generate an execution plan every time at runtime. IF and CASE statements come in handy to avoid dynamic SQL. Another major disadvantage of using dynamic SQL is that it requires users to have direct access permissions on all accessed objects, like tables and views. Generally, users are given access to the stored procedures which reference the tables, but not directly on the tables. In this case, dynamic SQL will not work.
• Consider the following drawbacks before using the IDENTITY property for generating primary keys. IDENTITY is very much SQL Server specific, and you will have problems porting your database application to some other RDBMS. IDENTITY columns have other inherent problems. For example, IDENTITY columns can run out of numbers at some point, depending on the data type selected; numbers can't be reused automatically, after deleting rows; and problems may arise if you are using replication. So, come up with an algorithm to generate a primary key in the front-end or from within the inserting stored procedure. There still could be issues with generating your own primary keys too, like concurrency while generating the
key, or running out of values. So, consider both options and choose the one that is most appropriate for your circumstances.
• Minimize the use of NULLs, as they often confuse the front-end applications, unless the applications are coded intelligently to eliminate NULLs or convert the NULLs into some other form. Any expression that deals with NULL results in a NULL output. ISNULL and COALESCE functions are helpful in dealing with NULL values.
• Always use a column list in your INSERT statements. This helps in avoiding problems when the table structure changes (like adding or dropping a column).
• Always access tables in the same order in all your stored procedures and triggers consistently. This helps in avoiding deadlocks. Other things to keep in mind to avoid deadlocks are:
• Keep your transactions as short as possible.
• Touch the least amount of data possible during a transaction.
• Do not wait for user input in the middle of a transaction.
• Do not use higher level locking hints or restrictive isolation levels unless they are absolutely needed.
• Make your front-end applications deadlock-intelligent, that is, these applications should be able to resubmit the transaction in case the previous transaction fails with error 1205.
• In your applications, process all the results returned by SQL Server immediately so that the locks on the processed rows are released, hence no blocking.
• Offload tasks, like string manipulations, concatenations, row numbering, case conversions, type conversions etc., to the front-end applications if these operations are going to consume more CPU cycles on the database server. Also try to do basic validations in the front-end itself during data entry. This saves unnecessary network roundtrips.
• Always check the global variable @@ERROR immediately after executing a data manipulation statement (like INSERT/UPDATE/DELETE), so that you can rollback the transaction in case of an error (@@ERROR will be greater than 0 in case of an error). This is important because, by default, SQL Server will not rollback all the previous changes within a transaction if a particular statement fails. This behavior can be changed by executing SET XACT_ABORT ON. The @@ROWCOUNT variable also plays an important role in determining how many rows were affected by a previous data manipulation (also, retrieval) statement, and based on that you could choose to commit or rollback a particular transaction.
• Always store 4 digit years instead of 2 digit years in dates (especially when using CHAR or INT datatype columns) to avoid any confusion and problems. This is not a problem with DATETIME columns, as the century is stored even if you specify a 2 digit year. But it's always a good practice to specify 4 digit years even with DATETIME datatype columns

• Do not forget to enforce unique constraints on your alternate keys.