Testing Bits

Habits provides a lightweight testing framework for bits that focuses on Input → Output validation without needing to mock entire servers or frameworks.

Overview

The @ha-bits/test-utils package provides:

• YAML-based tests (.test.yaml) - Simple, declarative test definitions
• TypeScript tests (.test.ts) - For complex mocking scenarios
• Fetch-level mocking - HTTP request interception
• Store mocking - For testing stateful bits
• Module mocking - For SDK-level mocking

Test Schema

The complete YAML schema for test files:

# Test Schema for bits testing - defines the structure of .test.yaml files
# Used by @ha-bits/test-utils to run automated bit tests
$schema: http://json-schema.org/draft-07/schema#

definitions:
  # Fetch mock definition - intercepts HTTP requests
  FetchMock:
    type: object
    description: Mock definition for intercepting HTTP fetch requests
    required:
      - url
      - response
    properties:
      url:
        type: string
        description: URL pattern to match (supports * wildcards). Use "*/endpoint" for partial match
        examples:
          - "*/chat/completions"
          - "https://api.example.com/v1/*"
          - "*"
      method:
        type: string
        description: HTTP method to match (GET, POST, PUT, DELETE, PATCH)
        enum: [GET, POST, PUT, DELETE, PATCH]
      status:
        type: integer
        description: HTTP status code to return (default 200)
        default: 200
      response:
        description: Response body to return (object for JSON, string for text)
      responseType:
        type: string
        description: Response content type
        enum: [json, text, arraybuffer]
        default: json
      assertRequest:
        type: object
        description: Assert request body contains these key-value pairs
        additionalProperties: true

  # Store mock definition - for testing stateful bits
  StoreMock:
    type: object
    description: Mock store with initial data for testing bits with state
    required:
      - initial
    properties:
      initial:
        type: object
        description: Initial store data (key-value pairs)
        additionalProperties: true

  # Module mock definition - for mocking npm modules
  ModuleMock:
    type: object
    description: Mock an entire npm module (for SDK-level mocking)
    required:
      - moduleName
      - exports
    properties:
      __type:
        type: string
        const: module_mock
        description: Type discriminator (auto-added if omitted)
      moduleName:
        type: string
        description: Name of the npm module to mock (e.g., "openai")
      exports:
        type: object
        description: Mock exports object
        additionalProperties: true
      clearSubmodules:
        type: boolean
        description: Clear all submodules from cache when applying mock
        default: true

  # Bit mock definition - for workflow testing
  BitMock:
    type: object
    description: Mock a specific bit's action response (for workflow testing)
    required:
      - bit
      - response
    properties:
      bit:
        type: string
        description: Bit name (e.g., "bit-openai")
      action:
        type: string
        description: Action to mock (optional, defaults to all actions)
      response:
        description: Mocked response - returned directly without executing the action
      assertInput:
        type: object
        description: Assert input before returning mock
        additionalProperties: true

  # Test mocks container
  TestMocks:
    type: object
    description: Container for all mock types used in a test
    properties:
      modules:
        oneOf:
          - type: array
            description: |
              Array format - for TypeScript tests with custom mock classes
              Example: [{ moduleName: 'openai', exports: { default: MockClass } }]
            items:
              $ref: "#/definitions/ModuleMock"
          - type: object
            description: |
              Object format - declarative mocking for YAML tests.
              Maps module names to method paths and return values.
              Example: { openai: { 'chat.completions.create': { choices: [...] } } }
            additionalProperties:
              type: object
              description: Methods to mock (dot notation paths to return values)
              additionalProperties: true
      fetch:
        type: array
        description: HTTP fetch mocks
        items:
          $ref: "#/definitions/FetchMock"
      store:
        $ref: "#/definitions/StoreMock"
      bits:
        type: array
        description: Bit mocks for workflow testing
        items:
          $ref: "#/definitions/BitMock"
      httpModules:
        type: array
        description: HTTP modules to replace with mock fetch (e.g., node-fetch, undici)
        items:
          type: string
      clearModules:
        type: array
        description: |
          Modules to clear from cache (but not replace).
          These will reload and pick up the mocked global fetch.
          Example: [openai, @anthropic-ai/sdk]
        items:
          type: string

  # Single test definition
  BitTestDefinition:
    type: object
    description: A single test case for a bit action
    required:
      - name
      - input
    properties:
      name:
        type: string
        description: Human-readable test name (shown in test output)
      action:
        type: string
        description: Action name to test (overrides file-level action)
      auth:
        type: object
        description: Authentication credentials for the bit
        additionalProperties: true
        examples:
          - apiKey: "sk-test-key"
          - apiKey: "sk-test"
            host: "api.openai.com"
      input:
        type: object
        description: Input parameters passed to the action (propsValue)
        additionalProperties: true
      mocks:
        $ref: "#/definitions/TestMocks"
      expect:
        description: |
          Expected output from the action.
          Can be a primitive (string, number) or an object to deep-compare partial matches.
      expectStore:
        type: object
        description: Expected store state after test execution
        additionalProperties: true
      expectError:
        type: string
        description: Expected error message (test passes if error contains this string)
      afterRun:
        description: |
          Callback function to run after the test (only in .test.ts files).
          Signature: (result: any, mocks: any) => void

  # Workflow test definition
  WorkflowTestDefinition:
    type: object
    description: A test case for an entire workflow/habit
    required:
      - name
    properties:
      name:
        type: string
        description: Human-readable test name
      trigger:
        type: object
        description: Trigger input data
        additionalProperties: true
      context:
        type: object
        description: Initial context/variables
        additionalProperties: true
      mocks:
        $ref: "#/definitions/TestMocks"
      expect:
        description: Expected final output from the workflow
      expectSteps:
        type: object
        description: Expected values at specific steps (step-id → expected value)
        additionalProperties: true
      expectError:
        type: string
        description: Expected error message

# Root schema - can be either bit tests or workflow tests
oneOf:
  # Bit test file (.test.yaml next to bit source)
  - type: object
    description: Bit test file - tests individual bit actions
    required:
      - bit
      - tests
    properties:
      bit:
        type: string
        description: Relative path to the bit source file (e.g., ./index.ts, ./send-prompt.ts)
      action:
        type: string
        description: Default action name for all tests (can be overridden per test)
      tests:
        type: array
        description: List of test cases
        items:
          $ref: "#/definitions/BitTestDefinition"
        minItems: 1

  # Workflow test file (.test.yaml for testing habits)
  - type: object
    description: Workflow test file - tests entire workflows/habits
    required:
      - workflow
      - tests
    properties:
      workflow:
        type: string
        description: Path to workflow YAML file
      tests:
        type: array
        description: List of workflow test cases
        items:
          $ref: "#/definitions/WorkflowTestDefinition"
        minItems: 1

Quick Start

1. Create a Test File

Place a .test.yaml file next to your bit source:

my-bit/
├── src/
│   ├── index.ts
│   └── index.test.yaml    # ← Test file here

2. Write Tests

bit: ./index.ts
tests:
  - name: "basic operation"
    action: myAction
    auth: { apiKey: "sk-test-key" }
    input:
      param1: "value1"
      param2: 42
    mocks:
      fetch:
        - url: "*/api/endpoint"
          response:
            result: "success"
    expect:
      output: "success"

3. Run Tests

# Run all bit tests
npx bits-test

# Run tests for specific bit
npx bits-test -f bit-openai

# Verbose output
npx bits-test -v

Test File Structure

Bit Test File (.test.yaml)

# Path to the bit source file
bit: ./index.ts

# Default action for all tests (optional)
action: myDefaultAction

# Test cases
tests:
  - name: "test name"
    action: specificAction      # Override default action
    auth:                       # Authentication credentials
      apiKey: "sk-test"
    input:                      # Input parameters (propsValue)
      prompt: "Hello"
    mocks:                      # Mocks configuration
      modules: {...}            # Module mocks (object or array format)
      fetch: [...]
      store: { initial: {} }
    expect: "expected output"   # Expected result
    expectStore: {}             # Expected store state after test
    expectError: "error msg"    # Expected error (for error tests)

Mocking

Module Mocking (Recommended for SDKs)

For bits that use SDKs like OpenAI, Anthropic, etc., mock the module directly using the object format:

mocks:
  modules:
    openai:
      chat.completions.create:
        choices:
          - message: { content: "Hello!", role: "assistant" }

This is cleaner than fetch mocking because:

• No need to match exact URLs
• Works regardless of how the SDK makes HTTP calls
• Automatically handles ES module interop

Method Paths

Specify the full method path using dot notation:

mocks:
  modules:
    openai:
      chat.completions.create: { ... }      # For chat
      images.generate: { ... }               # For images
      embeddings.create: { ... }             # For embeddings
    anthropic:
      messages.create: { ... }               # Anthropic SDK

Fetch Mocking

For bits that make direct HTTP calls (not through an SDK):

mocks:
  fetch:
    - url: "*/chat/completions"
      method: POST
      response:
        choices:
          - message: { content: "Hello!", role: "assistant" }
      status: 200

URL Patterns

• * - Match any URL
• */endpoint - Match URLs ending with /endpoint
• https://api.example.com/* - Match specific domain

Asserting Request Body

mocks:
  fetch:
    - url: "*/completions"
      assertRequest:
        model: "gpt-4"
        temperature: 0.7
      response: { ... }

Module Clearing

For SDKs that need to reload with mocked fetch:

mocks:
  clearModules: [openai, anthropic]
  fetch:
    - url: "*/completions"
      response: { ... }

This clears the module from Node's cache so it reloads and picks up the mocked global fetch.

Store Mocking

For bits that use persistent storage:

tests:
  - name: "with memory"
    input:
      prompt: "Remember: Alice"
      memoryKey: "session-1"
    mocks:
      store:
        initial: {}
    expect: "I'll remember that!"
    expectStore:
      "session-1":
        - role: user
          content: "Remember: Alice"
        - role: assistant
          content: "I'll remember that!"

TypeScript Tests (.test.ts)

For complex scenarios requiring custom mocks or logic:

// my-action.sdk.test.ts
class MockOpenAI {
  constructor(config: any) {}
  chat = {
    completions: {
      create: async () => ({
        choices: [{ message: { content: 'Response', role: 'assistant' } }],
      }),
    },
  };
}

export default [
  {
    name: 'with mocked SDK',
    action: 'myAction',
    mocks: {
      modules: [
        {
          moduleName: 'openai',
          exports: {
            default: MockOpenAI,
          },
        },
      ],
    },
    auth: { apiKey: 'sk-test' },
    input: { prompt: 'Hello' },
    expect: 'Response',
  },
];

Complete Example

Here's a full example testing an OpenAI-based bit with module mocking:

# bit-intersect/src/lib/actions/send-prompt.test.yaml
bit: ./send-prompt.ts
action: ask_chatgpt
tests:
  - name: "basic prompt"
    auth:
      apiKey: "sk-test"
      host: "api.openai.com"
    input:
      model: gpt-5
      prompt: "What is 2+2?"
      maxTokens: 100
    mocks:
      modules:
        openai:
          chat.completions.create:
            choices:
              - message: { content: "4", role: "assistant" }
    expect: "4"

  - name: "with memory key stores conversation"
    auth:
      apiKey: "sk-test"
      host: "api.openai.com"
    input:
      model: gpt-5
      prompt: "Remember my name is Alice"
      maxTokens: 100
      memoryKey: "session-1"
    mocks:
      modules:
        openai:
          chat.completions.create:
            choices:
              - message: { content: "I'll remember that!", role: "assistant" }
      store:
        initial: {}
    expect: "I'll remember that!"
    expectStore:
      "session-1":
        - role: user
          content: "Remember my name is Alice"
        - role: assistant
          content: "I'll remember that!"

Alternative: Fetch Mocking

For bits that don't use SDKs, you can mock at the fetch level:

# bit-openai/src/index.test.yaml
bit: ./index.ts
tests:
  - name: "chatCompletion"
    action: chatCompletion
    auth: { apiKey: "sk-test-key" }
    input:
      model: gpt-4o-mini
      userMessage: "Hello"
      systemPrompt: "You are helpful"
      temperature: 0.7
      maxTokens: 100
    mocks:
      clearModules: [openai]
      fetch:
        - url: "*/chat/completions"
          response:
            choices:
              - message: { content: "Hi there!", role: "assistant" }
                finish_reason: "stop"
            model: "gpt-4o-mini"
            usage: { prompt_tokens: 10, completion_tokens: 5 }
    expect:
      content: "Hi there!"
      finishReason: "stop"
      model: "gpt-4o-mini"

Testing Workflows

You can also test entire workflows/habits:

# workflow.test.yaml
workflow: ./my-workflow.yaml
tests:
  - name: "end-to-end flow"
    trigger:
      body: { message: "Hello" }
    mocks:
      bits:
        - bit: "bit-openai"
          action: "chatCompletion"
          response: { content: "Hi!" }
    expect:
      response: "Hi!"

CLI Reference

bits-test [options]

Options:
  -f, --file <pattern>   Run tests matching file pattern
  -a, --action <name>    Only run tests for specific action
  -v, --verbose          Show detailed output
  -h, --help             Show help

Examples:
  bits-test                           # Run all tests
  bits-test -f bit-openai             # Run bit-openai tests
  bits-test -f intersect -a send      # Run specific action tests
  bits-test -v                        # Verbose output

Best Practices

1. Test Happy Paths First

Focus on expected behavior before edge cases:

tests:
  - name: "returns valid response"
    input: { prompt: "Hello" }
    expect: { content: "Hi!" }

2. Use Descriptive Names

Test names should describe the scenario:

tests:
  - name: "with memory key stores conversation"
  - name: "handles empty input gracefully"
  - name: "returns error for invalid API key"

3. Mock at the Right Level

• Module mocking (object format) - For bits using SDKs (OpenAI, Anthropic, etc.) - declarative in YAML
• Module mocking (array format) - When full control over a module is needed - for TypeScript tests
• Fetch mocking - For bits making direct HTTP calls - use mocks.fetch
• Store mocking - For stateful operations - use mocks.store

4. Test Error Cases

tests:
  - name: "handles API error"
    input: { prompt: "Hello" }
    mocks:
      fetch:
        - url: "*/completions"
          status: 500
          response: { error: "Internal error" }
    expectError: "Internal error"

5. Keep Tests Independent

Each test should be self-contained with its own mocks:

tests:
  - name: "test 1"
    mocks:
      store: { initial: {} }
    # ...

  - name: "test 2"
    mocks:
      store: { initial: { key: "value" } }
    # ...

Troubleshooting

"No mock found for URL"

Ensure your URL pattern matches the actual request:

# Too specific - might not match
- url: "https://api.openai.com/v1/chat/completions"

# Better - uses wildcard
- url: "*/chat/completions"

Module Not Clearing

Add the module to clearModules:

mocks:
  clearModules: [openai, @anthropic-ai/sdk]

Store State Not Persisting

Ensure store.initial is defined:

mocks:
  store:
    initial: {}  # Required for store to work