~/src/www.mokhan.ca/xlgmokha [main]
モ cat software-testing-guide.md
software-testing-guide.md 72464 bytes | 2008-01-01 00:00
symlink: /dev/eng/software-testing-guide.md

Software Testing & Quality Assurance Guide

A comprehensive guide to software testing practices, methodologies, and tools covering unit testing, TDD, BDD, integration testing, and JavaScript testing frameworks.

Table of Contents

  1. Testing Fundamentals
  2. Test-Driven Development (TDD)
  3. Behavior-Driven Development (BDD)
  4. Unit Testing Best Practices
  5. Testing Legacy Code
  6. JavaScript Testing
  7. Mocking and Stubbing
  8. Integration Testing
  9. Testing Tools and Frameworks
  10. Quality Assurance Practices

Testing Fundamentals

Types of Tests

Software testing can be categorized into different types based on scope, purpose, and implementation approach. Understanding these distinctions is crucial for building a comprehensive testing strategy.

Unit Tests

Unit tests are blocks of code that exercise very specific areas of a codebase to ensure it meets its single responsibility. They assert that a specific result or behavior is met and can be broken down into two main approaches:

State-Based Testing (Black Box)

[Test]
public void Should_be_able_to_lease_a_slip() 
{
  ICustomer customer = CreateSUT();
  ISlip slip = ObjectMother.Slip();
  ILeaseDuration duration = LeaseDurations.Monthly;

  customer.Lease(slip, duration);

  Assert.AreEqual(1, ListFactory.From(customer.Leases()).Count);
}

State-based tests focus on the result rather than the implementation. They:

  • Are easier to refactor
  • Provide confidence for significant code changes
  • Allow evolution of codebase architecture
  • Test from the client component’s perspective

Interaction-Based Testing (White Box)

[Test]
public void Should_leverage_task_to_retrieve_all_registered_boats_for_customer()
{
  long customerId = 23;
  IList<BoatRegistrationDTO> boats = new List<BoatRegistrationDTO>();

  using (_mockery.Record())
  {
    SetupResult.For(_mockRequest.ParsePayloadFor(PayloadKeys.CustomerId)).Return(customerId);
    Expect.Call(_mockTask.AllBoatsFor(customerId)).Return(boats);
  }

  using (_mockery.Playback())
  {
    CreateSUT().Initialize();
  }
}

Interaction-based tests verify that components work as expected with their dependencies:

  • Focus on component interaction rather than results
  • Simulate different environment conditions
  • Verify expectations under various scenarios
  • Enable testing without external dependencies

Integration Tests

Integration tests exercise the system from top to bottom, ensuring proper behavior in production-like environments:

[Test]
public void Should_process_complete_order_workflow()
{
  // Arrange: Set up real database, file system, etc.
  var order = CreateTestOrder();
  
  // Act: Exercise the full system
  var result = _orderProcessingService.ProcessOrder(order);
  
  // Assert: Verify end-to-end behavior
  Assert.IsTrue(result.Success);
  Assert.IsNotNull(GetOrderFromDatabase(order.Id));
}

Integration tests:

  • Exercise the full system stack
  • Hit actual third-party components
  • Take longer to run due to external dependencies
  • Help identify contract implementation gaps
  • Uncover environment-specific scenarios

Test-Driven Development (TDD)

The TDD Cycle

Test-Driven Development follows a simple but powerful cycle:

  1. Red: Write a failing test
  2. Green: Write the minimum code to make it pass
  3. Refactor: Improve the design while keeping tests green

TDD Fundamentals

Based on Kent Beck’s foundational work in “Test Driven Development: By Example”:

// Example: Money class development
public class Money {
    private int amount;
    private String currency;
    
    public Money(int amount, String currency) {
        this.amount = amount;
        this.currency = currency;
    }
    
    public Money times(int multiplier) {
        return new Money(amount * multiplier, currency);
    }
    
    public boolean equals(Object object) {
        Money money = (Money) object;
        return amount == money.amount && 
               currency.equals(money.currency);
    }
}

TDD Benefits

  • Design Emergence: Tests drive out clean, focused design
  • Documentation: Tests serve as living specifications
  • Confidence: Comprehensive test coverage enables refactoring
  • Feedback: Immediate feedback on design decisions

TDD Patterns

Test List Pattern

  • Write down all tests you think you need
  • Pick the simplest test to implement first
  • Cross off tests as you complete them

Fake It (‘Til You Make It)

  • Return constants until you have enough tests to force generalization
  • Gradually replace constants with variables

Triangulation

  • Only generalize code when you have two or more examples

Behavior-Driven Development (BDD)

BDD Syntax and Structure

BDD extends TDD with more expressive syntax focused on behavior specification:

public class behaves_like_save_changes_view_bound_to_presenter : concerns_for<SaveChangesView>
{
  context c = () => { presenter = an<ISaveChangesPresenter>(); };
  because b = () => subject.attach_to(presenter);

  static protected ISaveChangesPresenter presenter;
}

public class when_the_save_button_is_clicked : behaves_like_save_changes_view_bound_to_presenter
{
  because b = () => EventTrigger.trigger_event<Events.ControlEvents>(new EventArgs()), subject.ux_save_button);

  it should_forward_the_call_to_the_presenter = () => presenter.was_told_to(x => x.save());
}

BDD Structure Elements

  • Context: Set up the test scenario
  • Because: The action being performed
  • It: The expected behavior/outcome

BDD Benefits

  • More readable test specifications
  • Better communication with stakeholders
  • Focus on behavior rather than implementation
  • Natural language descriptions of system behavior

BDD vs TDD

Aspect TDD BDD
Focus Test first Behavior first
Language Technical Business-focused
Structure Arrange/Act/Assert Given/When/Then
Audience Developers Developers + Stakeholders

Unit Testing Best Practices

Test Organization

Test Class Structure

[TestFixture]
public class CustomerServiceTests
{
    private CustomerService _service;
    private Mock<ICustomerRepository> _mockRepository;
    
    [SetUp]
    public void SetUp()
    {
        _mockRepository = new Mock<ICustomerRepository>();
        _service = new CustomerService(_mockRepository.Object);
    }
    
    [Test]
    public void Should_create_customer_with_valid_data()
    {
        // Arrange
        var customerData = new CustomerData("John", "Doe", "john@example.com");
        
        // Act
        var result = _service.CreateCustomer(customerData);
        
        // Assert
        Assert.IsTrue(result.Success);
        Assert.IsNotNull(result.Customer);
    }
}

Test Naming Conventions

Descriptive Names

// Good: Describes behavior
Should_throw_exception_when_email_is_invalid()
Should_return_empty_list_when_no_customers_exist()
Should_calculate_discount_for_premium_members()

// Avoid: Generic or unclear
TestCustomer()
Test1()
CustomerTest()

AAA Pattern (Arrange, Act, Assert)

[Test]
public void Should_apply_discount_to_premium_customers()
{
    // Arrange
    var customer = new Customer { IsPremium = true };
    var order = new Order { Total = 100m };
    var discountService = new DiscountService();
    
    // Act
    var discountedTotal = discountService.ApplyDiscount(customer, order);
    
    // Assert
    Assert.AreEqual(90m, discountedTotal);
}

Test Data Builders (Object Mother Pattern)

public class CustomerBuilder
{
    private Customer _customer = new Customer();
    
    public static CustomerBuilder New()
    {
        return new CustomerBuilder();
    }
    
    public CustomerBuilder WithEmail(string email)
    {
        _customer.Email = email;
        return this;
    }
    
    public CustomerBuilder Premium()
    {
        _customer.IsPremium = true;
        return this;
    }
    
    public Customer Build()
    {
        return _customer;
    }
}

// Usage
var customer = CustomerBuilder.New()
    .WithEmail("test@example.com")
    .Premium()
    .Build();

Testing Legacy Code

The Legacy Code Change Algorithm

Working with legacy code requires a systematic approach:

  1. Identify change points - Where modifications need to be made
  2. Find test points - Where tests can be added
  3. Break dependencies - Isolate code under test
  4. Write tests - Create safety net
  5. Make changes and refactor - Implement changes safely

Strategies for Legacy Code

Characterization Tests

[Test]
public void Should_maintain_existing_behavior()
{
    // Document current behavior, even if it's wrong
    var result = LegacyMethod(input);
    Assert.AreEqual(unexpectedButCurrentResult, result);
    
    // Later, when understanding improves, change to:
    // Assert.AreEqual(correctResult, result);
}

Seam Identification

  • Find places where you can alter behavior without changing code
  • Use dependency injection to insert test doubles
  • Extract interfaces from concrete dependencies

Sprouting Methods

// Instead of modifying complex legacy method directly
public void ComplexLegacyMethod()
{
    // ... existing complex code ...
    
    // Sprout new, testable method
    var result = CalculateNewFeature(data);
    
    // ... continue with legacy code ...
}

// New method is easily testable
public decimal CalculateNewFeature(Data data)
{
    // New, clean, testable code
    return data.Value * 0.1m;
}

Key Principles

“Over time, tested areas of the codebase surface like islands rising out of the ocean.”

“Mock objects are fakes that perform assertions internally.”

“It is better to depend on interfaces or abstract classes than concrete classes. When you depend on less volatile things, you minimize the chance that particular changes will trigger massive recompilation.”

JavaScript Testing

Testing Frameworks Overview

Jasmine

describe("Calculator", function() {
    var calculator;
    
    beforeEach(function() {
        calculator = new Calculator();
    });
    
    it("should add two numbers correctly", function() {
        var result = calculator.add(2, 3);
        expect(result).toEqual(5);
    });
    
    it("should handle negative numbers", function() {
        var result = calculator.add(-1, 1);
        expect(result).toEqual(0);
    });
});

Jasmine Features:

  • BDD-style syntax
  • Built-in matchers
  • Spy functionality
  • No DOM dependency
  • Excellent for unit testing

QUnit

test("Calculator addition", function() {
    var calculator = new Calculator();
    var result = calculator.add(2, 3);
    equal(result, 5, "2 + 3 should equal 5");
});

module("Calculator Tests", {
    setup: function() {
        this.calculator = new Calculator();
    }
});

QUnit Features:

  • jQuery’s testing framework
  • Simple assertion methods
  • Asynchronous testing support
  • Good CI integration
  • Extensible assertion framework

Framework Comparison

Framework Syntax Async Support CI Integration Learning Curve
Jasmine BDD Yes Good Low
QUnit Traditional Yes Excellent Low
ScrewUnit BDD Limited Good Medium
JSSpec BDD Yes Fair Medium

JavaScript Testing Best Practices

Test Structure

// Good: Clear, focused test
describe("User Authentication", function() {
    describe("when credentials are valid", function() {
        it("should authenticate user", function() {
            // Test implementation
        });
    });
    
    describe("when credentials are invalid", function() {
        it("should reject authentication", function() {
            // Test implementation
        });
    });
});

Mocking and Spying

// Jasmine spies
describe("User Service", function() {
    it("should call API with correct parameters", function() {
        spyOn(apiService, 'post').and.returnValue(Promise.resolve({}));
        
        userService.createUser({name: "John"});
        
        expect(apiService.post).toHaveBeenCalledWith('/users', {name: "John"});
    });
});

Running Tests in CI

QUnit CI Integration

<!DOCTYPE html>
<html>
<head>
    <title>QUnit Test Runner</title>
    <link rel="stylesheet" href="qunit.css">
    <script src="qunit.js"></script>
    <script src="your-code.js"></script>
    <script src="your-tests.js"></script>
</head>
<body>
    <div id="qunit"></div>
    <div id="qunit-fixture"></div>
</body>
</html>

Headless Testing

  • Use PhantomJS for headless test execution
  • Integrate with build systems (NAnt, MSBuild, Grunt)
  • Generate XML reports for CI systems

Mocking and Stubbing

Mock Objects vs Stubs

Stubs: Provide predetermined responses to method calls

var stub = new Mock<IEmailService>();
stub.Setup(x => x.Send(It.IsAny<Email>())).Returns(true);

Mocks: Perform assertions about how they were called

var mock = new Mock<IEmailService>();
mock.Setup(x => x.Send(It.IsAny<Email>())).Returns(true);

// After execution
mock.Verify(x => x.Send(It.IsAny<Email>()), Times.Once);

Rhino Mocks Patterns

Record/Playback Pattern

[Test]
public void Should_process_order_correctly()
{
    // Arrange
    var mockery = new MockRepository();
    var paymentService = mockery.StrictMock<IPaymentService>();
    var order = new Order(100m);
    
    using (mockery.Record())
    {
        Expect.Call(paymentService.ProcessPayment(100m))
              .Return(new PaymentResult { Success = true });
    }
    
    using (mockery.Playback())
    {
        var processor = new OrderProcessor(paymentService);
        var result = processor.Process(order);
        
        Assert.IsTrue(result.Success);
    }
}

Stub vs Mock Usage

// Stub: When you need data
var stubRepository = mockery.Stub<ICustomerRepository>();
stubRepository.Stub(x => x.GetById(1))
              .Return(new Customer { Id = 1, Name = "John" });

// Mock: When you need to verify behavior
var mockEmailService = mockery.StrictMock<IEmailService>();
mockEmailService.Expect(x => x.SendWelcomeEmail(customer))
                .Return(true);

Mocking Best Practices

Don’t Mock Value Objects

// Bad: Mocking simple data structures
var mockAddress = new Mock<IAddress>();

// Good: Use real value objects
var address = new Address("123 Main St", "City", "State");

Mock Roles, Not Objects

// Good: Mock based on role/responsibility
var mockPaymentProcessor = new Mock<IPaymentProcessor>();
var mockNotificationService = new Mock<INotificationService>();

// Avoid: Mocking specific implementations
var mockCreditCardPayment = new Mock<CreditCardPayment>();

Integration Testing

Database Integration Tests

[TestFixture]
public class CustomerRepositoryIntegrationTests
{
    private IDbConnection _connection;
    private CustomerRepository _repository;
    
    [SetUp]
    public void SetUp()
    {
        _connection = new SqlConnection(TestConnectionString);
        _repository = new CustomerRepository(_connection);
        SetupTestData();
    }
    
    [TearDown]
    public void TearDown()
    {
        CleanupTestData();
        _connection.Dispose();
    }
    
    [Test]
    public void Should_save_and_retrieve_customer()
    {
        // Arrange
        var customer = new Customer("John", "Doe", "john@example.com");
        
        // Act
        var savedId = _repository.Save(customer);
        var retrieved = _repository.GetById(savedId);
        
        // Assert
        Assert.AreEqual(customer.FirstName, retrieved.FirstName);
        Assert.AreEqual(customer.Email, retrieved.Email);
    }
}

API Integration Tests

[TestFixture]
public class OrderApiIntegrationTests
{
    private TestServer _server;
    private HttpClient _client;
    
    [SetUp]
    public void SetUp()
    {
        _server = new TestServer(new WebHostBuilder()
            .UseStartup<TestStartup>());
        _client = _server.CreateClient();
    }
    
    [Test]
    public async Task Should_create_order_via_api()
    {
        // Arrange
        var orderData = new CreateOrderRequest
        {
            CustomerId = 1,
            Items = new[] { new OrderItem { ProductId = 1, Quantity = 2 } }
        };
        
        // Act
        var response = await _client.PostAsJsonAsync("/api/orders", orderData);
        
        // Assert
        Assert.AreEqual(HttpStatusCode.Created, response.StatusCode);
        var order = await response.Content.ReadAsAsync<Order>();
        Assert.IsNotNull(order.Id);
    }
}

Integration Testing Strategies

Test Database Management

  • Use separate test database
  • Run tests in transactions (rollback after each test)
  • Use database snapshots for faster reset
  • Consider in-memory databases for faster execution

External Service Testing

  • Use test doubles for external APIs
  • Create contract tests for API integrations
  • Use consumer-driven contracts when possible
  • Test failure scenarios and timeouts

Testing Tools and Frameworks

.NET Testing Frameworks

NUnit

[TestFixture]
public class CalculatorTests
{
    [Test]
    [TestCase(2, 3, 5)]
    [TestCase(-1, 1, 0)]
    [TestCase(0, 0, 0)]
    public void Should_add_numbers_correctly(int a, int b, int expected)
    {
        var calculator = new Calculator();
        var result = calculator.Add(a, b);
        Assert.AreEqual(expected, result);
    }
}

MbUnit (Now part of Gallio)

[TestFixture]
public class CalculatorTests
{
    [Test]
    [Row(2, 3, 5)]
    [Row(-1, 1, 0)]
    [Row(0, 0, 0)]
    public void Should_add_numbers_correctly(int a, int b, int expected)
    {
        var calculator = new Calculator();
        var result = calculator.Add(a, b);
        Assert.AreEqual(expected, result);
    }
}

NUnit vs MbUnit Comparison

Feature NUnit MbUnit
Parameterized Tests TestCase Row
Setup/Teardown SetUp/TearDown SetUp/TearDown
Categories Category Category
Assertions Classic Assert Enhanced Assertions
Extensibility Good Excellent
IDE Integration Excellent Good

Mocking Frameworks

Rhino Mocks

var repository = MockRepository.GenerateMock<IRepository>();
repository.Stub(x => x.Get(1)).Return(new Entity { Id = 1 });
repository.AssertWasCalled(x => x.Save(Arg<Entity>.Is.Anything));

Moq (Modern Alternative)

var repository = new Mock<IRepository>();
repository.Setup(x => x.Get(1)).Returns(new Entity { Id = 1 });
repository.Verify(x => x.Save(It.IsAny<Entity>()), Times.Once);

Test Runners

Console Runners

  • NUnit Console Runner
  • MSTest command line
  • dotnet test

IDE Integration

  • Visual Studio Test Explorer
  • ReSharper Test Runner
  • Continuous testing with NCrunch

Quality Assurance Practices

Code Quality Metrics

Test Coverage

  • Aim for high coverage but don’t obsess over 100%
  • Focus on critical paths and complex logic
  • Use coverage to find untested code, not as a quality metric

Cyclomatic Complexity

  • Keep methods simple and focused
  • High complexity indicates need for refactoring
  • Use complexity metrics to guide testing efforts

Continuous Integration

Automated Testing Pipeline

# Example CI pipeline
stages:
  - build
  - unit-tests
  - integration-tests
  - deploy

unit-tests:
  stage: unit-tests
  script:
    - dotnet test --configuration Release
    - dotnet test --collect:"XPlat Code Coverage"
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Code Review Practices

Testing-Focused Reviews

  • Review tests alongside production code
  • Ensure tests actually test what they claim to test
  • Verify test readability and maintainability
  • Check for proper test isolation

Quality Gates

Definition of Done

  • All tests pass
  • Code coverage meets threshold
  • No critical static analysis violations
  • Peer review completed
  • Documentation updated

Quality Assurance Mindset

Key Principles

“The way to really get better at identification is to read more. Read books about design patterns. More important, read other people’s code. Look at open-source projects, and just take some time to browse and see how other people do things.”

“Holding on to a lot of state mentally can be useful, but it doesn’t really make us better at decision-making. At this point in my career, I think I’m a much better programmer than I used to be, even though I know less about the details of each language I work in.”

Testing Philosophy

  • Tests as Documentation: Well-written tests explain system behavior
  • Fail Fast: Catch errors as early as possible in the development cycle
  • Test Pyramid: More unit tests, fewer integration tests, minimal UI tests
  • Continuous Feedback: Use tests to get immediate feedback on changes

Building a Testing Culture

Team Practices

  • Pair programming on complex test scenarios
  • Test-first mindset adoption
  • Regular refactoring of test code
  • Knowledge sharing through test reviews

Learning and Improvement

  • Study open-source projects’ testing strategies
  • Experiment with new testing frameworks and tools
  • Attend testing conferences and workshops
  • Maintain a testing learning library

This comprehensive guide represents accumulated testing knowledge and practices developed over years of software development experience. The principles and patterns covered here form the foundation of sustainable, high-quality software development practices.

This guide consolidates knowledge from multiple blog posts and real-world testing experience, covering the evolution of testing practices from traditional unit testing through modern BDD approaches.

Lines: 987 | Words: 4450 | Reading time: 23 min