Check & Results

Overview

StepCheck compares actual results against expected data using deep object comparison. It is designed to run after the main execution steps and produce a structured report of what matched, what differed, and what was missing or unexpected.

StepCheck

StepCheck extends StepNormal with runOnError: true, which means it executes even if a previous step in the testcase has already failed. This ensures that result validation always runs regardless of earlier errors. When the comparison detects failures, StepCheck automatically marks the testcase status as ERROR.

Configuration

Configure StepCheck through its step data. The resultDir and dataDir fields set the base directories, and the checks array defines one or more comparison sets, each with a name and paths to the actual and expected data directories.

{
  "resultDir": "./results",
  "dataDir": "./expected",
  "checks": [
    {
      "name": "api-response",
      "actualDir": "actual/api",
      "expectedDir": "expected/api"
    }
  ]
}

Each entry in the checks array triggers an independent comparison. Files in actualDir are matched against files in expectedDir by name, and any mismatches are recorded in the output.

Comparison Directives

Not all fields can be compared with strict equality. BitDiver supports comparison directives that let you control how individual values are evaluated:

  • Time — Fuzzy time comparison that allows for a configurable tolerance window. Use this for timestamps that may differ slightly between runs.
  • Number — Numeric comparison with a configurable tolerance. Useful for floating-point values or metrics that may vary within an acceptable range.
  • Regex — Pattern matching against a regular expression. The actual value is tested against the provided pattern rather than compared for equality.
  • Contains — Substring matching. The check passes if the actual value contains the specified substring.

You can also define ignore paths to skip specific fields entirely during comparison. This is useful for fields like generated IDs, timestamps, or other non-deterministic values that should not affect the test outcome.

Output Files

After comparison, StepCheck writes structured output files to the result directory:

  • summary.json — Aggregate counts including total, passed, failed, missing (expected but not found), and unexpected (found but not expected).
  • mapping.json — Records which actual files were mapped to which expected files, making it easy to trace comparison pairs.
  • details.json — Per-file, attribute-level comparison results showing exactly which fields matched, which differed, and what the actual versus expected values were.

Custom Checks

For specialized validation logic you can extend StepCheck and override its comparison behavior, or use the runCheck function directly for programmatic access to the comparison engine.

import { StepCheck } from '@xhubio/bitdiver-runner'

class CustomCheck extends StepCheck {
  async run() {
    // Custom pre-processing
    await this.prepareActualData()

    // Run the built-in comparison
    await super.run()

    // Custom post-processing
    await this.generateReport()
  }
}

Alternatively, call runCheck directly when you need comparison results without the step lifecycle:

import { runCheck } from '@xhubio/bitdiver-runner'

const result = await runCheck({
  actualDir: './actual',
  expectedDir: './expected',
  ignorePaths: ['$.metadata.timestamp']
})
← Execution Logging →