Why Test Software?

As Drupal developers and users we share our software with a community of 630,000+ other users and developers (http://drupal.org/about). We have entered into a great social contract. We trust each other that our core and contributed code will work as they are supposed to. With the scale of the community and the scale of the code base there will be some exception. The introduction of a testing framework into Drupal has significantly helped reduce those exceptions and provide the community and clients with confidence in the quality of Drupal.

Drupal 7 has seen major benefits from embracing the testing framework. [Testing] has significantly enhanced the way core is developed and made it possible to make major API enhancements with confidence.

– http://drupal.org/simpletest

Software testing in Drupal benefits developers, client stakeholders, and end-users. This short series of posts is written for these audiences. In this series we will do some basic exploration together into the why, what, and how of software testing as it relates to Drupal web development. Throughout the series the reader will find links to resources for further review. This first post in the series examines the question of, “Why test software?”

A Definition

Let’s begin with what software testing is.

Software testing may be defined using the definition Cem Kaner gave it in his 2006 presentation titled, “Exploratory Testing”:

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test.

– http://www.kaner.com/pdfs/ETatQAI.pdf

In the excellent Wikipedia article, “Software Testing”, the process of software testing is described as:

Validating and verifying that a computer program/application/product:

  • Meets the requirements that guided its design and development
  • Works as expected
  • Can be implemented with the same characteristics
  • Satisfies the needs of stakeholders

– http://en.wikipedia.org/wiki/Software_testing

To summarize, software testing makes sure software does what it is supposed to do and meets the needs it is supposed to meet.

It is important to note that many of the practices involved in software testing focus on the technical challenge of answering whether code does what it is supposed to do, while answering whether the software satisfies the needs of stakeholders can often be missed or buried during the development process. In some cases, testing for satisfaction becomes relegated to the last moments of the development process during a relatively minuscule period of time we call “acceptance testing”.
Fortunately, with the popularization and adoption rate of Agile development practices, there is renewed up-front and continual focus on satisfying the needs of stakeholders through the use of user stories and conditions of satisfaction (acceptance criteria).

See:

  • http://en.wikipedia.org/wiki/Agile_software_development

  • http://www.mountaingoatsoftware.com/topics/user-stories

  • http://www.mountaingoatsoftware.com/uploads/presentations/Introduction-User-Stories-Agile-Development-Practices-2009.pdf

A good rule of thumb during testing is to remember we test to make sure code works the way it is supposed to in order to satisfy the greater objective; software must satisfy the needs of the stakeholders.

Why Test Software?

Satisfying the needs of stakeholders is our first and primary answer as to why we test software.

Further examination of the question includes a consideration of the balance between the following two points:

  1. Software testing will not always save time and money, at least not during software development cycles. In fact, software testing done well during a development project adds overhead and usually more cost.

  2. A good software testing implementation adds quality to what is produced and mitigates the risk of failure at any number of points in the life cycle of the software.

Software testing as a project requirement involves a balance between budget and time with quality and risk mitigation.

If testing increases time and budget it may be tempting to cut testing out of our plan because we are pretty sure the final product will meet the needs of its stakeholders. Before we decide not to test, let’s first consider two potentials. For you, your organization, or your business, the weight of importance in these two potentials may be greater than budget and time.

The potential for loss

Software, which has bugs and does not do what it is supposed to do, has the potential to cause loss of:

Money

  • Spent on fixing broken code
  • Not getting to market on time, losing market share
  • Getting to market with a bad product that turns customers away
  • Legal issues and cost due to repercussions of broken functionality
  • In house costs when internal software is not ready to meet business needs on time or correctly

Time

  • Users lose time with slow or buggy software
  • Project delays as schedules slip or time is diverted to fix issues
  • People not being able to use software at all to get their job done

Reputation

  • If customer service, production, quality, or other business essentials rely on software that does not work, then customer needs are not met, and business reputation is harmed
  • Everything a customer interacts with from an organization or business reflects upon the organization’s brand and reputation.
    • Buggy software provides a negative interaction.

Life

  • Though not likely with most Drupal development projects, software bugs have been the cause of industrial, government, and scientific safety hazards and accidents.
  • http://www.wired.com/software/coolapps/news/2005/11/69355
  • http://en.wikipedia.org/wiki/List_of_software_bugs

Reference

http://www.softwaretesting.com.au/Why_is_Software_Testing_important.php

The potential for gain

While the potential for loss provides significant reason to test software, there are gains to be considered as well.

Potential gains include:

A culture of quality

  • Training and experience with software testing practices often increases awareness of quality requirements and the measurement of quality improvement criteria

Better needs definition

  • The process of writing software tests often brings up issues missed during definition, while agreement upon test coverage contributes to agreement upon overall results
  • Software tests, when used as criteria for acceptance, gives us a definition of done

Proof in performance, quality, and delivery

  • Tests provide tangible metrics of success for internal and external clients
  • Passing test criteria provides a demonstration of meeting delivery criteria

Time

  • Detecting faults early and often saves development time
  • Debugging is often quicker with good test coverage
  • Together early problem detection and better debugging begin to make up for the initial overhead of adding a testing process

Risk Mitigation, as testing reduces:

  • The potentials for loss of money, time, reputation, and life
  • The risk of bad software making it to delivery
  • In the case with Drupal development, tests help prevent broken functionality from making it to a production site
  • Risks at critical points in the development process, such as during integration, scope change, and deployment

Reference

  1. http://www.gatsby.ucl.ac.uk/~turner/TeaTalks/unitTesting/unitTestingTalk.pdf
  2. http://dipperstove.com/development/software-testing-benefits.html
  3. http://mavericktester.com/archive/benefits-of-sofware-testing/

One Final Note

Drupal as a platform is web centric. The web and its technologies continue to rapidly evolve. Web-based software relies on interacting with a significant number of other software, APIs, and protocols. Managing the complexities of software on the web is always a challenge. The ability to test Drupal (at the core-level) in both isolation and integration has provided developer and client alike with a vital tool for managing these complexities. The next post in this series will explore what we test in Drupal. Then we will conclude our exploration with how to test Drupal.

Share or Comment via Twitter

dev ops, drupal, software testing