TestingAPI - BrowserTest

Overview

The TestingAPI provides methods for running agentic tests with natural language instructions and managing test jobs.

Methods

execute()

Runs an agentic test with natural language instructions and optional structured output.

execute(options: TestOptions): Promise<TestResult>

Parameters

Parameter	Type	Required	Description
`options`	`TestOptions`	Yes	Test execution options

TestOptions

interface TestOptions {
  /** Natural language test instructions */
  instructions: string;
  /** URL to test */
  url: string;
  /** JSON schema for structured output (optional) */
  outputSchema?: object;
  /** Test configuration */
  config?: TestConfig;
}

TestConfig

interface TestConfig {
  /** Viewport width (default: 1280) */
  width?: number;
  /** Viewport height (default: 720) */
  height?: number;
  /** Wait time before starting test (default: 0) */
  waitFor?: number;
  /** Test timeout in milliseconds (default: 30000) */
  timeout?: number;
  /** Block ads and trackers (default: true) */
  blockAds?: boolean;
  /** User agent string (optional) */
  userAgent?: string;
}

Returns

Promise<TestResult> - Test execution result

TestResult

interface TestResult {
  success: boolean;
  results: {
    /** Agent actions performed during test */
    actions: Array<{
      type: string;
      description: string;
      timestamp: string;
    }>;
    /** Structured data if outputSchema was provided */
    structuredData?: any;
    /** Raw test output */
    output: string;
  };
  meta: {
    timeMs: number;
    url: string;
  };
}

Examples

// Basic test
const result = await bt.testing.execute({
  instructions: 'Navigate to the homepage and verify the main heading is visible',
  url: 'https://example.com'
});

console.log('Test successful:', result.success);

// Test with structured output
const loginTest = await bt.testing.execute({
  instructions: 'Test the login functionality with valid credentials',
  url: 'https://example.com/login',
  outputSchema: {
    type: 'object',
    properties: {
      loginSuccessful: { type: 'boolean' },
      userName: { type: 'string' },
      errorMessage: { type: 'string' }
    }
  }
});

console.log('Login result:', loginTest.results.structuredData);

// Test with custom configuration
const configuredTest = await bt.testing.execute({
  instructions: 'Test the checkout flow',
  url: 'https://shop.com',
  config: {
    width: 1920,
    height: 1080,
    waitFor: 2000,
    timeout: 60000
  }
});

create()

Creates an async test job without waiting for completion.

create(options: TestOptions): Promise<TestJob>

Parameters

Parameter	Type	Required	Description
`options`	`TestOptions`	Yes	Test execution options

Returns

Promise<TestJob> - Test job information

TestJob

interface TestJob {
  jobId: string;
  status: 'pending' | 'running' | 'completed' | 'failed';
  createdAt: string;
  url: string;
}

Example

const job = await bt.testing.create({
  instructions: 'Perform comprehensive site audit',
  url: 'https://example.com',
  config: { timeout: 300000 } // 5 minutes
});

console.log('Job created:', job.jobId, 'Status:', job.status);

getStatus()

Gets the status and results of a test job.

getStatus(jobId: string): Promise<TestJobStatus>

Parameters

Parameter	Type	Required	Description
`jobId`	`string`	Yes	Job ID to check

Returns

Promise<TestJobStatus> - Detailed job status

TestJobStatus

interface TestJobStatus {
  job: {
    id: string;
    status: 'pending' | 'running' | 'completed' | 'failed';
    createdAt: string;
    startedAt?: string;
    completedAt?: string;
    url: string;
    config?: TestConfig;
    results?: TestResult['results'];
    error?: string;
    progress?: string;
  };
}

Example

const status = await bt.testing.getStatus('job-123');

switch (status.job.status) {
  case 'pending':
    console.log('Job is queued');
    break;
  case 'running':
    console.log('Job is running...', status.job.progress);
    break;
  case 'completed':
    console.log('Job completed!', status.job.results);
    break;
  case 'failed':
    console.log('Job failed:', status.job.error);
    break;
}

list()

Lists test jobs with optional filtering.

list(options?: ListTestsOptions): Promise<TestJobsList>

Parameters

Parameter	Type	Required	Description
`options`	`ListTestsOptions`	No	Filtering and pagination options

ListTestsOptions

interface ListTestsOptions {
  /** Filter by status */
  status?: 'pending' | 'running' | 'completed' | 'failed';
  /** Filter by template ID */
  templateId?: string;
  /** Maximum number of results (default: 20) */
  limit?: number;
  /** Number of results to skip (default: 0) */
  offset?: number;
  /** Sort order (default: 'desc') */
  sortOrder?: 'asc' | 'desc';
}

Returns

Promise<TestJobsList> - List of test jobs

TestJobsList

interface TestJobsList {
  jobs: Array<{
    id: string;
    status: string;
    createdAt: string;
    completedAt?: string;
    url: string;
    templateId?: string;
    results?: TestResult['results'];
    error?: string;
  }>;
  total: number;
  hasMore: boolean;
}

Examples

// Get all jobs
const allJobs = await bt.testing.list();

// Get running jobs
const running = await bt.testing.list({ status: 'running' });

// Get jobs for specific template
const templateJobs = await bt.testing.list({
  templateId: 'template-123',
  limit: 10
});

// Paginated results
const page1 = await bt.testing.list({ limit: 20, offset: 0 });
const page2 = await bt.testing.list({ limit: 20, offset: 20 });

test()

Convenience method for simple tests.

test(instructions: string, url: string, options?: TestExecutionOptions): Promise<TestResult>

Parameters

Parameter	Type	Required	Description
`instructions`	`string`	Yes	Test instructions
`url`	`string`	Yes	URL to test
`options`	`TestExecutionOptions`	No	Additional options

TestExecutionOptions

interface TestExecutionOptions {
  /** Test configuration */
  config?: TestConfig;
  /** Output schema */
  outputSchema?: object;
}

Returns

Promise<TestResult> - Test result

Example

const result = await bt.testing.test(
  'Check if the homepage loads without errors',
  'https://example.com'
);

Convenience method for login testing.

login(url: string, credentials: LoginCredentials, options?: Partial<TestOptions>): Promise<TestResult>

Parameters

Parameter	Type	Required	Description
`url`	`string`	Yes	Login page URL
`credentials`	`LoginCredentials`	Yes	Login credentials
`options`	`Partial<TestOptions>`	No	Additional test options

LoginCredentials

interface LoginCredentials {
  email: string;
  password: string;
}

Returns

Promise<TestResult> - Login test result

Example

const loginResult = await bt.testing.login(
  'https://app.com/login',
  {
    email: 'user@example.com',
    password: 'password123'
  }
);

fillForm()

Convenience method for form filling tests.

fillForm(url: string, formData: object, options?: Partial<TestOptions>): Promise<TestResult>

Parameters

Parameter	Type	Required	Description
`url`	`string`	Yes	Form page URL
`formData`	`object`	Yes	Form field data
`options`	`Partial<TestOptions>`	No	Additional test options

Returns

Promise<TestResult> - Form filling test result

Example

const formResult = await bt.testing.fillForm(
  'https://example.com/contact',
  {
    name: 'John Doe',
    email: 'john@example.com',
    message: 'Hello, this is a test!'
  }
);

Advanced Usage

Structured Output Schemas

const eCommerceTest = await bt.testing.execute({
  instructions: `
    Navigate to the product page
    Add item to cart
    Proceed to checkout
    Fill shipping information
    Complete payment
  `,
  url: 'https://shop.com/product/123',
  outputSchema: {
    type: 'object',
    properties: {
      navigationSuccessful: { type: 'boolean' },
      itemAddedToCart: { type: 'boolean' },
      checkoutStarted: { type: 'boolean' },
      shippingCompleted: { type: 'boolean' },
      paymentSuccessful: { type: 'boolean' },
      orderNumber: { type: 'string' },
      totalAmount: { type: 'number' },
      errors: {
        type: 'array',
        items: { type: 'string' }
      }
    },
    required: ['navigationSuccessful', 'itemAddedToCart']
  }
});

Complex Test Scenarios

const multiStepTest = await bt.testing.execute({
  instructions: `
    1. Navigate to the dashboard
    2. Click "Create New Project"
    3. Fill in project details:
       - Name: "Test Automation Project"
       - Description: "Automated testing suite"
       - Category: "Quality Assurance"
    4. Upload a test file
    5. Save the project
    6. Verify the project appears in the list
    7. Open project details
    8. Run a test within the project
    9. Check test results
  `,
  url: 'https://testing-app.com/dashboard',
  config: {
    width: 1920,
    height: 1080,
    timeout: 120000 // 2 minutes for complex test
  },
  outputSchema: {
    type: 'object',
    properties: {
      projectCreated: { type: 'boolean' },
      projectName: { type: 'string' },
      fileUploaded: { type: 'boolean' },
      testExecuted: { type: 'boolean' },
      testResults: { type: 'object' },
      finalStatus: { type: 'string' }
    }
  }
});

Error Handling and Validation

async function robustTest(url, instructions, options = {}) {
  try {
    const result = await bt.testing.execute({
      instructions,
      url,
      config: {
        timeout: 60000,
        ...options.config
      },
      outputSchema: options.schema
    });

    return {
      success: true,
      data: result.results,
      meta: result.meta
    };

  } catch (error) {
    console.error('Test execution failed:', error.message);

    return {
      success: false,
      error: error.message,
      url,
      instructions
    };
  }
}

// Usage
const testCases = [
  {
    name: 'Homepage Load Test',
    instructions: 'Verify homepage loads and main elements are visible',
    url: 'https://example.com',
    schema: {
      type: 'object',
      properties: {
        pageLoaded: { type: 'boolean' },
        titleVisible: { type: 'boolean' },
        navWorking: { type: 'boolean' }
      }
    }
  },
  {
    name: 'Search Functionality',
    instructions: 'Test search feature with valid and invalid queries',
    url: 'https://example.com/search',
    schema: {
      type: 'object',
      properties: {
        validSearchWorks: { type: 'boolean' },
        invalidSearchHandled: { type: 'boolean' },
        resultsDisplayed: { type: 'boolean' }
      }
    }
  }
];

const results = await Promise.all(
  testCases.map(test => robustTest(test.url, test.instructions, {
    schema: test.schema
  }))
);

Job Monitoring and Reporting

class TestMonitor {
  constructor(bt) {
    this.bt = bt;
    this.activeJobs = new Map();
  }

  async startTest(testOptions) {
    const job = await this.bt.testing.create(testOptions);
    this.activeJobs.set(job.jobId, {
      ...job,
      startTime: Date.now()
    });

    return job.jobId;
  }

  async checkJob(jobId) {
    const status = await this.bt.testing.getStatus(jobId);
    const jobInfo = this.activeJobs.get(jobId);

    if (status.job.status === 'completed' || status.job.status === 'failed') {
      this.activeJobs.delete(jobId);

      return {
        jobId,
        status: status.job.status,
        duration: Date.now() - jobInfo.startTime,
        results: status.job.results,
        error: status.job.error
      };
    }

    return {
      jobId,
      status: status.job.status,
      progress: status.job.progress,
      duration: Date.now() - jobInfo.startTime
    };
  }

  async waitForAll() {
    const results = [];

    while (this.activeJobs.size > 0) {
      const jobIds = Array.from(this.activeJobs.keys());

      for (const jobId of jobIds) {
        const result = await this.checkJob(jobId);
        if (result.status === 'completed' || result.status === 'failed') {
          results.push(result);
        }
      }

      // Wait before checking again
      await new Promise(resolve => setTimeout(resolve, 2000));
    }

    return results;
  }

  getActiveCount() {
    return this.activeJobs.size;
  }
}

// Usage
const monitor = new TestMonitor(bt);

// Start multiple tests
const jobIds = await Promise.all([
  monitor.startTest({
    instructions: 'Test login flow',
    url: 'https://app.com/login'
  }),
  monitor.startTest({
    instructions: 'Test checkout process',
    url: 'https://shop.com/checkout'
  })
]);

// Monitor progress
setInterval(() => {
  console.log(`Active jobs: ${monitor.getActiveCount()}`);
}, 5000);

// Wait for completion
const results = await monitor.waitForAll();
console.log('All tests completed:', results);

Best Practices

Instruction Writing

Be specific: Use clear, actionable language
Include context: Describe what success looks like
Handle edge cases: Consider what might go wrong
Use sequences: Number steps for complex tests
Specify timeouts: For operations that might take time

Schema Design

// Good schema - comprehensive and typed
const goodSchema = {
  type: 'object',
  properties: {
    operationSuccessful: { type: 'boolean' },
    dataCollected: { type: 'object' },
    errors: {
      type: 'array',
      items: { type: 'string' }
    },
    metadata: {
      type: 'object',
      properties: {
        duration: { type: 'number' },
        steps: { type: 'number' }
      }
    }
  },
  required: ['operationSuccessful']
};

// Avoid - too generic
const badSchema = {
  type: 'object',
  properties: {
    result: { type: 'string' }
  }
};

Error Handling

class TestRunner {
  constructor(bt) {
    this.bt = bt;
  }

  async runTestWithRetry(testOptions, maxRetries = 3) {
    let lastError;

    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        console.log(`Test attempt ${attempt}/${maxRetries}`);

        const result = await this.bt.testing.execute({
          ...testOptions,
          config: {
            timeout: 45000, // Shorter for retries
            ...testOptions.config
          }
        });

        return {
          success: true,
          attempt,
          result
        };

      } catch (error) {
        lastError = error;
        console.warn(`Attempt ${attempt} failed:`, error.message);

        if (attempt < maxRetries) {
          // Exponential backoff
          const delay = Math.pow(2, attempt) * 1000;
          await new Promise(resolve => setTimeout(resolve, delay));
        }
      }
    }

    return {
      success: false,
      attempt: maxRetries,
      error: lastError.message
    };
  }

  async runTestSuite(tests) {
    const results = [];

    for (const test of tests) {
      console.log(`Running: ${test.name}`);
      const result = await this.runTestWithRetry(test.options);
      results.push({
        name: test.name,
        ...result
      });
    }

    const successful = results.filter(r => r.success).length;
    console.log(`Suite complete: ${successful}/${results.length} passed`);

    return results;
  }
}

Resource Management

// Queue tests to avoid overwhelming the API
class TestQueue {
  constructor(bt, concurrency = 3) {
    this.bt = bt;
    this.concurrency = concurrency;
    this.queue = [];
    this.running = 0;
    this.results = [];
  }

  add(testOptions) {
    return new Promise((resolve, reject) => {
      this.queue.push({
        options: testOptions,
        resolve,
        reject
      });
      this.process();
    });
  }

  async process() {
    if (this.running >= this.concurrency || this.queue.length === 0) {
      return;
    }

    this.running++;
    const { options, resolve, reject } = this.queue.shift();

    try {
      const result = await this.bt.testing.execute(options);
      this.results.push(result);
      resolve(result);
    } catch (error) {
      reject(error);
    } finally {
      this.running--;
      this.process();
    }
  }

  async waitForAll() {
    while (this.queue.length > 0 || this.running > 0) {
      await new Promise(resolve => setTimeout(resolve, 1000));
    }
    return this.results;
  }
}

// Usage
const queue = new TestQueue(bt, 2); // Max 2 concurrent tests

const testPromises = [
  queue.add({
    instructions: 'Test login',
    url: 'https://app.com/login'
  }),
  queue.add({
    instructions: 'Test registration',
    url: 'https://app.com/register'
  }),
  queue.add({
    instructions: 'Test password reset',
    url: 'https://app.com/forgot-password'
  })
];

const results = await Promise.all(testPromises);
console.log('All tests completed');

​Overview

​Methods

​execute()

​Parameters

​TestOptions

​TestConfig

​Returns

​TestResult

​Examples

​create()

​Parameters

​Returns

​TestJob

​Example

​getStatus()

​Parameters

​Returns

​TestJobStatus

​Example

​list()

​Parameters

​ListTestsOptions

​Returns

​TestJobsList

​Examples

​test()

​Parameters

​TestExecutionOptions

​Returns

​Example

​login()

​Parameters

​LoginCredentials

​Returns

​Example

​fillForm()

​Parameters

​Returns

​Example

​Advanced Usage

​Structured Output Schemas

​Complex Test Scenarios

​Error Handling and Validation

​Job Monitoring and Reporting

​Best Practices

​Instruction Writing

​Schema Design

​Error Handling

​Resource Management

Overview

Methods

execute()

Parameters

TestOptions

TestConfig

Returns

TestResult

Examples

create()

Parameters

Returns

TestJob

Example

getStatus()

Parameters

Returns

TestJobStatus

Example

list()

Parameters

ListTestsOptions

Returns

TestJobsList

Examples

test()

Parameters

TestExecutionOptions

Returns

Example

login()

Parameters

LoginCredentials

Returns

Example

fillForm()

Parameters

Returns

Example

Advanced Usage

Structured Output Schemas

Complex Test Scenarios

Error Handling and Validation

Job Monitoring and Reporting

Best Practices

Instruction Writing

Schema Design

Error Handling

Resource Management