Stop Shipping Tests That Pass While the Software Is Broken — Generate Suites That Prove Behavior

A good test name communicates:

• Scenario
• Action
• Expected result

Examples:

returns_forbidden_when_user_updates_another_account
creates_audit_event_after_successful_profile_update
does_not_retry_when_request_is_not_idempotent
rolls_back_order_when_payment_capture_fails

Avoid names such as:

test_method_1
works_correctly
handles_error

A common test structure is:

Arrange — Prepare inputs, dependencies, and expected values.

Act — Invoke the behavior.

Assert — Verify output and meaningful side effects.

Example:

[Fact]
public async Task ReturnsNotFoundWhenUserDoesNotExist()
{
    // Arrange
    repository
        .GetByIdAsync("missing-user")
        .Returns((User?)null);

    // Act
    var result = await service.GetUserAsync(
        "missing-user"
    );

    // Assert
    result.Should().BeNull();
}

Use the project’s established conventions.

Boundaries often expose defects.

For a valid range of 1 through 100, test:

• 0
• 1
• 2
• 99
• 100
• 101

For nullable values, test:

• Null
• Empty
• Whitespace
• Valid value
• Oversized value
• Invalid type where possible

Group input values expected to behave similarly.

Example for age:

• Below valid range
• Minimum valid value
• Typical valid value
• Maximum valid value
• Above valid range
• Non-numeric value
• Missing value

One or more representative tests can cover each partition.

Negative tests verify that unsafe or invalid behavior is rejected correctly.

Examples:

• Unauthenticated access
• Unauthorized access
• Invalid input
• Missing dependency
• Expired token
• Duplicate request
• Database conflict
• Timeout
• Unsupported format
• Malformed payload
• Invalid state transition

Tests should verify both rejection and side effects.

Example:

• Request returns 403
• Database remains unchanged
• No success audit event is created

Test:

• Exception type
• Error code
• Status code
• Error schema
• Message safety
• Retry behavior
• Logging behavior
• Cleanup
• Partial changes
• Caller-visible result

Do not assert exact internal error text unless it is part of a public contract.

Mock:

• Expensive external dependencies
• Unreliable external services
• Time
• Randomness
• Network calls
• File systems where isolation is required
• Notification systems
• Payment providers
• Cloud APIs

Do not mock:

• The function under test
• Business logic being verified
• Every internal method
• Framework behavior that should be validated through integration tests
• Database behavior when query correctness matters
• Serialization when schema compatibility matters

Over-mocking can create tests that pass even when the system fails.

Stub — Returns predefined data.

Fake — Provides a simplified working implementation, such as an in-memory repository.

Mock — Verifies interactions.

Choose the simplest test double that supports the test objective.

Good test data should be:

• Synthetic
• Minimal
• Explicit
• Reusable where appropriate
• Isolated
• Easy to understand
• Free of production secrets
• Free of unnecessary personal data

Avoid large test fixtures containing irrelevant data.

Use builders or factories when many tests require valid objects with small variations.

function buildUser(
  overrides: Partial<User> = {}
): User {
  return {
    id: 'user-001',
    email: 'user@example.invalid',
    displayName: 'Example User',
    status: 'Active',
    ...overrides,
  };
}

This keeps test intent visible.

Avoid relying on the actual system clock.

Use:

• Injected clock
• Fake timer
• Fixed timestamp
• Controllable scheduler

Weak:

await new Promise(resolve =>
  setTimeout(resolve, 5000));

Better:

clock.advanceBy(5000);

The exact mechanism depends on the framework.

Inject or control randomness.

Test:

• Deterministic seed
• Fixed identifier generator
• Collision behavior
• Boundary behavior

Do not make test results depend on uncontrolled random values.

Concurrency tests may verify:

• Lost update prevention
• Duplicate job prevention
• Locking
• Idempotency
• Optimistic concurrency
• Event ordering
• Thread safety
• Atomic operations

Concurrency tests should use synchronization signals rather than arbitrary sleeps.

For an idempotent operation, verify:

1. First request succeeds.
2. Second identical request does not duplicate side effects.
3. Response remains consistent.
4. Audit behavior is appropriate.
5. Partial failure does not break future retries.

Use one or more of:

• Transaction rollback
• Unique database schema
• Unique tenant
• Per-test database
• Containerized dependency
• Test namespace
• Generated identifiers
• Queue isolation
• Cleanup hooks

Parallel test execution should not cause collisions.

Verify:

• Real query behavior
• Mapping
• Constraints
• Transactions
• Rollback
• Null handling
• Index-sensitive behavior where necessary
• Migration compatibility
• Concurrency
• Data cleanup

An in-memory database may not reproduce production database behavior.

Use:

• Official sandbox
• Local emulator
• Contract test
• Recorded safe response
• Stub server
• Test tenant

Verify:

• Request format
• Authentication
• Response parsing
• Error mapping
• Timeouts
• Retries
• Rate limits
• Version compatibility

Test:

• Authentication
• Authorization
• Validation
• Headers
• Content type
• Status code
• Response schema
• Error schema
• Persistence
• Side effects
• Idempotency
• Pagination
• Caching
• Rate limiting

Example:

it('returns 403 when updating another user', async () => {
  const response = await authenticatedRequest(
    userA
  )
    .put(`/api/users/${userB.id}`)
    .send({
      displayName: 'Changed Name',
    });
  expect(response.status).toBe(403);
  const storedUser = await users.findById(
    userB.id
  );
  expect(storedUser.displayName).not.toBe(
    'Changed Name'
  );
});

Verify:

• Valid event
• Invalid schema
• Duplicate delivery
• Out-of-order event
• Retryable failure
• Non-retryable failure
• Dead-letter routing
• Idempotency
• Transaction behavior
• Audit logging
• Poison message

Verify important audit and security events.

Avoid tests that depend on incidental debug messages.

Test:

• Correct event type
• Correct actor
• Correct target
• Correct outcome
• Correlation identifier
• Sensitive data not included

Test supported:

• Runtime versions
• Operating systems
• Database versions
• API versions
• Browser versions
• Dependency versions
• Configuration modes

Compatibility claims should be tied to actual test execution.

Test:

• Upgrade from previous schema
• Existing data
• New application with old schema where relevant
• Old application with new schema where coexistence is required
• Rollback
• Default values
• Nullability
• Large data volume
• Failed migration recovery

Common causes of flaky tests include:

• Shared data
• Real clock
• Arbitrary sleeps
• Uncontrolled random values
• External services
• Execution order
• Resource contention
• Uncleaned state
• Asynchronous work not awaited
• Network timing
• Insufficient timeouts

A flaky test is a defect in the test suite.

Do not solve flakiness by automatically retrying every failed test without investigation.

Coverage can reveal unexecuted code but cannot prove correct behavior.

Review:

• Branch coverage
• Condition coverage
• Critical-path coverage
• Requirement coverage
• Failure-path coverage
• Permission coverage
• Data-boundary coverage

A high coverage percentage can coexist with poor assertions.

Mutation testing changes code intentionally to determine whether tests detect the change.

It can reveal:

• Weak assertions
• Untested conditions
• Tests that only execute code
• Missing boundary tests

Use selectively because it may be computationally expensive.

Property-based testing validates general rules over many generated inputs.

Examples:

• Sorting output remains ordered
• Serialization round-trips
• Total never becomes negative
• Repeated idempotent calls preserve state
• Encoding and decoding are reversible

Properties must be defined carefully.

Snapshot tests can be useful for:

• Stable rendered output
• Serialized structures
• Generated configuration
• API schemas

Risks include:

• Large unreadable snapshots
• Blind acceptance of changes
• Overly broad snapshots
• Accidental freezing of unstable fields

Review snapshot changes carefully.

A test failure should make clear:

• What scenario failed
• What input was used
• What was expected
• What actually occurred
• Which dependency state applied
• Which correlation ID applies

Avoid assertions that produce only:

Expected true, received false.

Prefer domain-specific assertions and messages.

Create a function that calculates an order total.

Rules:

• Subtotal must be zero or greater.
• Discount must be between 0 and 100.
• Tax rate must be between 0 and 1.
• Discount is applied before tax.
• Result is rounded to two decimal places.
• Invalid input should raise a validation error.

def calculate_total(subtotal, discount_percent, tax_rate):
    discounted = subtotal - (subtotal * discount_percent / 100)
    total = discounted + (discounted * tax_rate)
    return round(total, 2)

The implementation performs the intended calculation but does not validate input.

Important boundaries include:

• Subtotal = 0
• Negative subtotal
• Discount = 0
• Discount = 100
• Discount below 0
• Discount above 100
• Tax rate = 0
• Tax rate = 1
• Tax rate below 0
• Tax rate above 1
• Fractional currency
• Large values
• Invalid types

import pytest
from orders.pricing import calculate_total


def test_calculates_total_without_discount_or_tax():
    result = calculate_total(
        subtotal=100,
        discount_percent=0,
        tax_rate=0,
    )
    assert result == 100.00


def test_applies_discount_before_tax():
    result = calculate_total(
        subtotal=100,
        discount_percent=10,
        tax_rate=0.10,
    )
    assert result == 99.00


def test_allows_zero_subtotal():
    result = calculate_total(
        subtotal=0,
        discount_percent=25,
        tax_rate=0.10,
    )
    assert result == 0.00


def test_allows_full_discount():
    result = calculate_total(
        subtotal=100,
        discount_percent=100,
        tax_rate=0.10,
    )
    assert result == 0.00


@pytest.mark.parametrize(
    'subtotal',
    [-1, -0.01],
)
def test_rejects_negative_subtotal(subtotal):
    with pytest.raises(
        ValueError,
        match='subtotal must be zero or greater',
    ):
        calculate_total(
            subtotal=subtotal,
            discount_percent=0,
            tax_rate=0,
        )


@pytest.mark.parametrize(
    'discount_percent',
    [-1, 101],
)
def test_rejects_discount_outside_range(
    discount_percent,
):
    with pytest.raises(
        ValueError,
        match='discount_percent must be between 0 and 100',
    ):
        calculate_total(
            subtotal=100,
            discount_percent=discount_percent,
            tax_rate=0,
        )


@pytest.mark.parametrize(
    'tax_rate',
    [-0.01, 1.01],
)
def test_rejects_tax_rate_outside_range(tax_rate):
    with pytest.raises(
        ValueError,
        match='tax_rate must be between 0 and 1',
    ):
        calculate_total(
            subtotal=100,
            discount_percent=0,
            tax_rate=tax_rate,
        )


def test_rounds_result_to_two_decimal_places():
    result = calculate_total(
        subtotal=10.01,
        discount_percent=5,
        tax_rate=0.0825,
    )
    assert result == 10.30

These tests assume the function is updated to implement the documented validation behavior.

from numbers import Real


def calculate_total(
    subtotal: Real,
    discount_percent: Real,
    tax_rate: Real,
) -> float:
    values = {
        'subtotal': subtotal,
        'discount_percent': discount_percent,
        'tax_rate': tax_rate,
    }
    for name, value in values.items():
        if not isinstance(value, Real):
            raise TypeError(
                f'{name} must be numeric'
            )
    if subtotal < 0:
        raise ValueError(
            'subtotal must be zero or greater'
        )
    if not 0 <= discount_percent <= 100:
        raise ValueError(
            'discount_percent must be between 0 and 100'
        )
    if not 0 <= tax_rate <= 1:
        raise ValueError(
            'tax_rate must be between 0 and 1'
        )
    discounted = subtotal - (
        subtotal * discount_percent / 100
    )
    total = discounted + (
        discounted * tax_rate
    )
    return round(total, 2)

Currency-sensitive applications may require decimal arithmetic rather than binary floating-point values.