Test Templates

Overview

Test templates allow you to create reusable test configurations that can be invoked multiple times with different parameters. Templates are perfect for complex, long-running tests that need consistent execution and structured output.

Creating a Template

const template = await bt.template.create({
  name: 'E-commerce Checkout Test',
  description: 'Test the complete checkout flow for an e-commerce site',
  instructions: `
    Navigate to the product page.
    Add the specified item to cart.
    Proceed to checkout.
    Fill in shipping information.
    Complete payment.
    Verify order confirmation.
  `,
  outputSchema: {
    type: 'object',
    properties: {
      checkoutSuccessful: {
        type: 'boolean',
        description: 'Whether checkout completed successfully'
      },
      orderNumber: {
        type: 'string',
        description: 'Order confirmation number'
      },
      totalAmount: {
        type: 'string',
        description: 'Total order amount'
      },
      items: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            name: { type: 'string' },
            quantity: { type: 'number' },
            price: { type: 'string' }
          }
        }
      },
      errors: {
        type: 'array',
        items: { type: 'string' },
        description: 'Any errors encountered'
      }
    }
  },
  tags: ['checkout', 'e-commerce', 'purchase']
});

console.log('Template created:', template.template.id);

Template Components

Instructions

Instructions can be detailed and step-by-step:

const complexTemplate = await bt.template.create({
  name: 'User Registration Flow',
  instructions: `
    1. Navigate to the registration page
    2. Fill out the registration form with provided data:
       - First Name: {{firstName}}
       - Last Name: {{lastName}}
       - Email: {{email}}
       - Password: {{password}}
    3. Agree to terms and conditions
    4. Submit the form
    5. Verify email confirmation is sent
    6. Check that user is redirected to welcome page
    7. Validate welcome message contains user's name
  `,
  // ... other config
});

Output Schema

Define structured output for consistent results:

const userTestSchema = {
  type: 'object',
  properties: {
    registrationSuccessful: { type: 'boolean' },
    userId: { type: 'string' },
    emailVerified: { type: 'boolean' },
    welcomeMessage: { type: 'string' },
    redirectUrl: { type: 'string' },
    formErrors: {
      type: 'array',
      items: { type: 'string' }
    },
    timeToComplete: { type: 'number' }
  },
  required: ['registrationSuccessful']
};

Tags and Metadata

const taggedTemplate = await bt.template.create({
  name: 'API Integration Test',
  description: 'Test third-party API integrations',
  tags: ['api', 'integration', 'external', 'critical'],
  instructions: `
    Test all external API integrations:
    - Payment gateway connectivity
    - Shipping calculator
    - Email service
    - Analytics tracking
  `,
  // ... schema and config
});

Invoking Templates

Basic Invocation

// Invoke with just a URL
const job = await bt.template.invoke(template.template.id, {
  url: 'https://example-shop.com/product/123'
});

console.log('Job created:', job.jobId);
console.log('Status:', job.status);

With Parameters

// Invoke with custom configuration
const job = await bt.template.invoke(template.template.id, {
  url: 'https://example-shop.com/checkout',
  config: {
    viewport: { width: 1920, height: 1080 },
    waitFor: 3000,
    timeout: 120000
  },
  parameters: {
    productId: '12345',
    quantity: 2,
    shippingMethod: 'express'
  }
});

Create and Invoke in One Step

const result = await bt.template.createAndInvoke({
  name: 'Quick Page Check',
  instructions: 'Navigate to page and verify it loads without errors',
  url: 'https://example.com'
});

console.log('Template created and executed:', result.results.success);

Managing Templates

List Templates

// Get all templates
const allTemplates = await bt.template.list();

// Search by name
const searchResults = await bt.template.list({
  search: 'checkout'
});

// Filter by tags
const taggedTemplates = await bt.template.list({
  tags: ['e-commerce', 'critical']
});

// Paginated results
const paginated = await bt.template.list({
  limit: 20,
  offset: 40
});

Update Templates

const updated = await bt.template.update(template.template.id, {
  name: 'Updated Checkout Test',
  description: 'Improved checkout testing template',
  instructions: `
    Enhanced checkout flow test with additional validations...
  `,
  tags: ['checkout', 'e-commerce', 'purchase', 'updated']
});

Delete Templates

await bt.template.delete(template.template.id);
console.log('Template deleted');

Async Job Management

Check Job Status

const status = await bt.testing.getStatus(job.jobId);
console.log('Job status:', status.job.status);

switch (status.job.status) {
  case 'pending':
    console.log('Job is queued for execution');
    break;
  case 'running':
    console.log('Job is currently running');
    break;
  case 'completed':
    console.log('Job completed successfully');
    console.log('Results:', status.job.results);
    break;
  case 'failed':
    console.log('Job failed:', status.job.error);
    break;
}

List Jobs

// Get all jobs
const allJobs = await bt.testing.list();

// Filter by status
const runningJobs = await bt.testing.list({
  status: 'running'
});

// Filter by template
const templateJobs = await bt.testing.list({
  templateId: template.template.id
});

// Recent jobs
const recentJobs = await bt.testing.list({
  limit: 10,
  status: 'completed'
});

Wait for Job Completion

async function waitForJob(jobId, maxWaitTime = 300000) { // 5 minutes
  const startTime = Date.now();

  while (Date.now() - startTime < maxWaitTime) {
    const status = await bt.testing.getStatus(jobId);

    if (status.job.status === 'completed') {
      return status.job.results;
    }

    if (status.job.status === 'failed') {
      throw new Error(`Job failed: ${status.job.error}`);
    }

    // Wait 5 seconds before checking again
    await new Promise(resolve => setTimeout(resolve, 5000));
  }

  throw new Error('Job timed out');
}

// Usage
const results = await waitForJob(job.jobId);
console.log('Job completed:', results);

Advanced Template Features

Parameterized Instructions

Use variables in your instructions:

const parameterizedTemplate = await bt.template.create({
  name: 'Custom Form Test',
  instructions: `
    Navigate to {{formUrl}}
    Fill out the form with:
    - Name: {{userName}}
    - Email: {{userEmail}}
    - Message: {{userMessage}}
    Submit the form
    Verify success message: "{{expectedMessage}}"
  `,
  outputSchema: {
    type: 'object',
    properties: {
      formSubmitted: { type: 'boolean' },
      successMessage: { type: 'string' },
      formValid: { type: 'boolean' }
    }
  }
});

// Invoke with parameters
const job = await bt.template.invoke(parameterizedTemplate.template.id, {
  url: 'https://example.com',
  parameters: {
    formUrl: '/contact',
    userName: 'John Doe',
    userEmail: 'john@example.com',
    userMessage: 'Test message',
    expectedMessage: 'Thank you for your message!'
  }
});

Conditional Logic in Instructions

const conditionalTemplate = await bt.template.create({
  name: 'Adaptive Test',
  instructions: `
    Navigate to the login page
    If login form is visible:
      - Fill in credentials
      - Submit form
      - Verify dashboard access
    If already logged in:
      - Verify dashboard is accessible
      - Check user menu

    Regardless of login state:
    - Test main navigation
    - Verify page content loads
  `,
  outputSchema: {
    type: 'object',
    properties: {
      loginRequired: { type: 'boolean' },
      loginSuccessful: { type: 'boolean' },
      navigationWorks: { type: 'boolean' },
      contentLoads: { type: 'boolean' }
    }
  }
});

Template Categories

E-commerce Templates

const checkoutTemplate = await bt.template.create({
  name: 'Complete Checkout Flow',
  tags: ['e-commerce', 'checkout', 'payment'],
  instructions: `
    1. Add items to cart
    2. Proceed to checkout
    3. Enter shipping information
    4. Select payment method
    5. Complete purchase
    6. Verify confirmation
  `,
  outputSchema: {
    // Comprehensive checkout validation schema
  }
});

const productSearchTemplate = await bt.template.create({
  name: 'Product Search & Filter',
  tags: ['e-commerce', 'search', 'filtering'],
  instructions: `
    1. Navigate to product catalog
    2. Test search functionality
    3. Apply various filters
    4. Verify results update correctly
    5. Test pagination
  `,
  // ... schema
});

User Authentication Templates

const authTemplate = await bt.template.create({
  name: 'Authentication Flow',
  tags: ['auth', 'login', 'security'],
  instructions: `
    1. Test login with valid credentials
    2. Test login with invalid credentials
    3. Test password reset flow
    4. Test session persistence
    5. Test logout functionality
  `,
  outputSchema: {
    type: 'object',
    properties: {
      validLoginWorks: { type: 'boolean' },
      invalidLoginRejected: { type: 'boolean' },
      passwordResetWorks: { type: 'boolean' },
      sessionPersists: { type: 'boolean' },
      logoutWorks: { type: 'boolean' }
    }
  }
});

API Integration Templates

const apiIntegrationTemplate = await bt.template.create({
  name: 'API Integration Test',
  tags: ['api', 'integration', 'external'],
  instructions: `
    1. Test payment gateway integration
    2. Verify shipping calculator
    3. Check email service connectivity
    4. Validate analytics tracking
    5. Test error handling for failed APIs
  `,
  outputSchema: {
    type: 'object',
    properties: {
      paymentGateway: { type: 'boolean' },
      shippingCalculator: { type: 'boolean' },
      emailService: { type: 'boolean' },
      analyticsTracking: { type: 'boolean' },
      errorHandling: { type: 'boolean' },
      degradedModeWorks: { type: 'boolean' }
    }
  }
});

Best Practices

Template Design

Modular instructions: Break complex tests into clear steps
Comprehensive schemas: Define detailed output structures
Error handling: Include error scenarios in instructions
Timeouts: Set appropriate timeouts for long-running tests
Parameterization: Use variables for flexible templates

Organization

// Template categories
const categories = {
  auth: [],
  eCommerce: [],
  api: [],
  ui: [],
  performance: []
};

// Tag consistently
const tags = {
  critical: 'mission-critical functionality',
  smoke: 'basic functionality tests',
  regression: 'prevent regression bugs',
  integration: 'external system integration'
};

Maintenance

// Version control for templates
const templateVersion = await bt.template.update(templateId, {
  name: `${templateName} v2.1`,
  instructions: updatedInstructions,
  tags: [...existingTags, 'updated']
});

// Archive old versions
const archived = await bt.template.update(oldTemplateId, {
  tags: [...existingTags, 'archived', 'deprecated']
});

Monitoring & Analytics

async function analyzeTemplatePerformance(templateId) {
  const jobs = await bt.testing.list({
    templateId,
    limit: 100,
    status: 'completed'
  });

  const stats = {
    totalJobs: jobs.jobs.length,
    successRate: jobs.jobs.filter(j => j.results.success).length / jobs.jobs.length,
    averageDuration: jobs.jobs.reduce((sum, j) => sum + j.results.meta.timeMs, 0) / jobs.jobs.length,
    commonErrors: {} // Analyze error patterns
  };

  return stats;
}

Troubleshooting

Common Template Issues

Timeout errors: Increase timeout or simplify instructions
Parameter not found: Check parameter names match template variables
Schema validation: Ensure output matches defined schema
Async job failures: Check job status and error messages

Debugging Templates

// Test template with debug output
const debugJob = await bt.template.invoke(templateId, {
  url: testUrl,
  config: {
    timeout: 120000, // Extra time for debugging
    debug: true // If supported
  }
});

// Monitor job progress
const status = await bt.testing.getStatus(debugJob.jobId);
console.log('Job progress:', status.job.progress);
console.log('Current step:', status.job.currentStep);
console.log('Logs:', status.job.logs);

Template Validation

async function validateTemplate(templateId) {
  try {
    // Test with minimal data
    const testJob = await bt.template.invoke(templateId, {
      url: 'https://httpbin.org/html', // Simple test page
      config: { timeout: 30000 }
    });

    const result = await waitForJob(testJob.jobId, 60000);

    return {
      valid: true,
      executionTime: result.meta.timeMs,
      outputValid: validateAgainstSchema(result.results.structuredData)
    };

  } catch (error) {
    return {
      valid: false,
      error: error.message
    };
  }
}

Getting Started

Core Features

Advanced Usage

Overview

Creating a Template

Template Components

Instructions

Output Schema

Tags and Metadata

Invoking Templates

Basic Invocation

With Parameters

Create and Invoke in One Step

Managing Templates

List Templates

Update Templates

Delete Templates

Async Job Management

Check Job Status

List Jobs

Wait for Job Completion

Advanced Template Features

Parameterized Instructions

Conditional Logic in Instructions

Template Categories

E-commerce Templates

User Authentication Templates

API Integration Templates

Best Practices

Template Design

Organization

Maintenance

Monitoring & Analytics

Troubleshooting

Common Template Issues

Debugging Templates

Template Validation

Getting Started

Core Features

Advanced Usage

​Overview

​Creating a Template

​Template Components

​Instructions

​Output Schema

​Tags and Metadata

​Invoking Templates

​Basic Invocation

​With Parameters

​Create and Invoke in One Step

​Managing Templates

​List Templates

​Update Templates

​Delete Templates

​Async Job Management

​Check Job Status

​List Jobs

​Wait for Job Completion

​Advanced Template Features

​Parameterized Instructions

​Conditional Logic in Instructions

​Template Categories

​E-commerce Templates

​User Authentication Templates

​API Integration Templates

​Best Practices

​Template Design

​Organization

​Maintenance

​Monitoring & Analytics

​Troubleshooting

​Common Template Issues

​Debugging Templates

​Template Validation

Overview

Creating a Template

Template Components

Instructions

Output Schema

Tags and Metadata

Invoking Templates

Basic Invocation

With Parameters

Create and Invoke in One Step

Managing Templates

List Templates

Update Templates

Delete Templates

Async Job Management

Check Job Status

List Jobs

Wait for Job Completion

Advanced Template Features

Parameterized Instructions

Conditional Logic in Instructions

Template Categories

E-commerce Templates

User Authentication Templates

API Integration Templates

Best Practices

Template Design

Organization

Maintenance

Monitoring & Analytics

Troubleshooting

Common Template Issues

Debugging Templates

Template Validation