Agentic Testing

Overview

Agentic testing allows you to run complex browser tests using natural language instructions. AI agents interpret your instructions and perform the necessary actions, making testing more intuitive and powerful.

Basic Agentic Test

The simplest way to run an agentic test:

const result = await bt.testing.execute({
  instructions: 'Navigate to the homepage and verify the main heading is visible',
  url: 'https://example.com'
});

console.log('Test completed:', result.results.success);
console.log('Agent actions:', result.results.actions);

Structured Output

Define expected output schemas for consistent results:

const loginTest = await bt.testing.execute({
  instructions: 'Test the login functionality with valid credentials',
  url: 'https://example.com/login',
  outputSchema: {
    type: 'object',
    properties: {
      loginSuccessful: {
        type: 'boolean',
        description: 'Whether login succeeded'
      },
      userName: {
        type: 'string',
        description: 'Logged in user name if available'
      },
      errorMessage: {
        type: 'string',
        description: 'Any error message encountered'
      },
      redirectUrl: {
        type: 'string',
        description: 'URL redirected to after login'
      }
    },
    required: ['loginSuccessful']
  }
});

console.log('Login result:', loginTest.results.structuredData);

Advanced Instructions

Write detailed, step-by-step instructions for complex scenarios:

const eCommerceTest = await bt.testing.execute({
  instructions: `
    Navigate to the product catalog page.
    Search for "wireless headphones".
    Click on the first product result.
    Add the item to cart.
    Proceed to checkout.
    Verify the checkout page loads correctly.
    Check that the product appears in the cart summary.
  `,
  url: 'https://example-shop.com',
  outputSchema: {
    type: 'object',
    properties: {
      searchSuccessful: { type: 'boolean' },
      productFound: { type: 'boolean' },
      addedToCart: { type: 'boolean' },
      checkoutAccessible: { type: 'boolean' },
      cartTotal: { type: 'string' },
      errors: { type: 'array', items: { type: 'string' } }
    }
  }
});

Convenience Methods

BrowserTest provides specialized methods for common testing patterns:

const loginResult = await bt.testing.login(
  'https://example.com/login',
  {
    email: 'user@example.com',
    password: 'password123'
  }
);

console.log('Login successful:', loginResult.results.structuredData.loginSuccessful);

Form Filling

const formResult = await bt.testing.fillForm(
  'https://example.com/contact',
  {
    name: 'John Doe',
    email: 'john@example.com',
    message: 'Hello, this is a test message!'
  }
);

console.log('Form submitted:', formResult.results.structuredData.submitted);

Simple Test Method

// Quick test without structured output
const quickTest = await bt.testing.test(
  'Check if the homepage loads without errors',
  'https://example.com'
);

console.log('Test passed:', quickTest.results.success);

Configuration Options

Customize test execution behavior:

const result = await bt.testing.execute({
  instructions: 'Test the checkout flow',
  url: 'https://example-shop.com',
  config: {
    viewport: { width: 1920, height: 1080 },
    waitFor: 3000, // Wait time before starting
    timeout: 60000, // Test timeout
    userAgent: 'custom-user-agent-string',
    blockAds: true
  },
  outputSchema: { /* schema */ }
});

Handling Dynamic Content

Test applications with dynamic content and user interactions:

const dynamicTest = await bt.testing.execute({
  instructions: `
    Wait for the page to load completely.
    Click on the "Load More" button.
    Wait for new content to appear.
    Verify that at least 10 items are now visible.
    Take a screenshot of the expanded content.
  `,
  url: 'https://example.com/products',
  outputSchema: {
    type: 'object',
    properties: {
      loadMoreClicked: { type: 'boolean' },
      contentLoaded: { type: 'boolean' },
      itemCount: { type: 'number' },
      screenshotTaken: { type: 'boolean' }
    }
  }
});

Error Handling & Retry Logic

Handle test failures gracefully:

async function robustTest(url, instructions, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await bt.testing.execute({
        instructions,
        url,
        config: { timeout: 30000 }
      });

      if (result.results.success) {
        return result;
      }

      console.log(`Test failed on attempt ${attempt}, retrying...`);

    } catch (error) {
      console.error(`Attempt ${attempt} failed:`, error.message);

      if (attempt === maxRetries) {
        throw new Error(`Test failed after ${maxRetries} attempts`);
      }

      // Wait before retry
      await new Promise(resolve => setTimeout(resolve, 2000));
    }
  }
}

Test Result Analysis

Understanding and analyzing test results:

const result = await bt.testing.execute({
  instructions: 'Complete the user registration flow',
  url: 'https://example.com/register'
});

// Check overall success
console.log('Test successful:', result.results.success);

// Analyze agent actions
console.log('Actions performed:');
result.results.actions.forEach((action, index) => {
  console.log(`${index + 1}. ${action.type}: ${action.description}`);
});

// Check structured data if schema was provided
if (result.results.structuredData) {
  console.log('Structured results:', result.results.structuredData);
}

// Check for errors or warnings
if (result.results.errors && result.results.errors.length > 0) {
  console.log('Errors encountered:', result.results.errors);
}

Advanced Scenarios

Multi-Step Workflows

const workflowTest = await bt.testing.execute({
  instructions: `
    1. Navigate to the dashboard
    2. Click "Create New Project"
    3. Fill in project details (name: "Test Project", description: "Automated test")
    4. Save the project
    5. Verify the project appears in the project list
    6. Open the project details
    7. Confirm all information is saved correctly
  `,
  url: 'https://project-manager.com/dashboard',
  outputSchema: {
    type: 'object',
    properties: {
      projectCreated: { type: 'boolean' },
      projectName: { type: 'string' },
      projectId: { type: 'string' },
      detailsVerified: { type: 'boolean' },
      errors: { type: 'array', items: { type: 'string' } }
    }
  }
});

Data Validation Testing

const validationTest = await bt.testing.execute({
  instructions: `
    Test form validation by:
    1. Leaving all fields empty and submitting
    2. Entering invalid email format
    3. Entering password that's too short
    4. Entering mismatched passwords
    5. Finally entering valid data and submitting successfully
  `,
  url: 'https://example.com/signup',
  outputSchema: {
    type: 'object',
    properties: {
      emptyFormValidation: {
        type: 'object',
        properties: {
          preventedSubmission: { type: 'boolean' },
          errorMessages: { type: 'array', items: { type: 'string' } }
        }
      },
      emailValidation: { type: 'boolean' },
      passwordValidation: { type: 'boolean' },
      finalSubmission: { type: 'boolean' }
    }
  }
});

Best Practices

Writing Effective Instructions

Be specific: Clearly describe what you want the agent to do
Use action verbs: “Click”, “Type”, “Navigate”, “Verify”, “Wait”
Include timing: Specify when to wait for elements or actions
Describe expectations: Explain what success looks like
Handle edge cases: Consider what might go wrong

Optimizing Performance

// Use appropriate timeouts
const optimizedTest = await bt.testing.execute({
  instructions: 'Quick page load check',
  url: 'https://example.com',
  config: {
    timeout: 15000, // Shorter for simple tests
    waitFor: 1000 // Less wait time
  }
});

Test Organization

class EcommerceTester {
  constructor(bt) {
    this.bt = bt;
  }

  async testProductSearch(query) {
    return this.bt.testing.execute({
      instructions: `Search for "${query}" and verify results`,
      url: 'https://shop.com',
      outputSchema: { /* search schema */ }
    });
  }

  async testCheckout() {
    return this.bt.testing.execute({
      instructions: 'Complete full checkout flow',
      url: 'https://shop.com/checkout',
      outputSchema: { /* checkout schema */ }
    });
  }
}

Monitoring & Reporting

async function runTestSuite(url, tests) {
  const results = [];

  for (const test of tests) {
    console.log(`Running: ${test.name}`);
    try {
      const result = await bt.testing.execute({
        instructions: test.instructions,
        url,
        outputSchema: test.schema
      });

      results.push({
        name: test.name,
        success: result.results.success,
        duration: result.meta.timeMs,
        data: result.results.structuredData
      });

    } catch (error) {
      results.push({
        name: test.name,
        success: false,
        error: error.message
      });
    }
  }

  return results;
}

Common Patterns

Authentication Testing

const authTest = await bt.testing.execute({
  instructions: `
    Try to access a protected page without authentication.
    Verify you're redirected to login.
    Log in with valid credentials.
    Verify you can now access the protected page.
    Log out.
    Verify you're redirected to login again.
  `,
  url: 'https://app.com/protected',
  outputSchema: {
    type: 'object',
    properties: {
      initialRedirect: { type: 'boolean' },
      loginSuccess: { type: 'boolean' },
      accessGranted: { type: 'boolean' },
      logoutSuccess: { type: 'boolean' },
      finalRedirect: { type: 'boolean' }
    }
  }
});

const navTest = await bt.testing.execute({
  instructions: `
    Test all main navigation links:
    - Click each menu item
    - Verify the correct page loads
    - Check that breadcrumbs work
    - Test back/forward browser buttons
  `,
  url: 'https://site.com',
  outputSchema: {
    type: 'object',
    properties: {
      navigationLinks: { type: 'number' },
      allLinksWork: { type: 'boolean' },
      breadcrumbsFunctional: { type: 'boolean' },
      browserNavigationWorks: { type: 'boolean' },
      brokenLinks: { type: 'array', items: { type: 'string' } }
    }
  }
});

Troubleshooting

Common Issues

Test timeouts: Increase timeout or simplify instructions
Element not found: Use more specific selectors or add wait times
Dynamic content: Add explicit waits for content to load
Complex interactions: Break down into smaller, sequential steps

Debugging Tips

// Add detailed logging
const debugTest = await bt.testing.execute({
  instructions: `
    Navigate to page
    Wait 2 seconds
    Look for login form
    Take a screenshot
    Fill in credentials
    Submit form
    Wait for redirect
  `,
  url: 'https://example.com/login',
  config: {
    timeout: 45000 // Extra time for debugging
  }
});

// Log all actions performed
console.log('Agent actions:');
debugTest.results.actions.forEach(action => {
  console.log(`- ${action.type}: ${action.description}`);
});

Getting Started

Core Features

Advanced Usage

Overview

Basic Agentic Test

Structured Output

Advanced Instructions

Convenience Methods

Form Filling

Simple Test Method

Configuration Options

Handling Dynamic Content

Error Handling & Retry Logic

Test Result Analysis

Advanced Scenarios

Multi-Step Workflows

Data Validation Testing

Best Practices

Writing Effective Instructions

Optimizing Performance

Test Organization

Monitoring & Reporting

Common Patterns

Authentication Testing

Navigation Testing

Troubleshooting

Common Issues

Debugging Tips

​Overview

​Basic Agentic Test

​Structured Output

​Advanced Instructions

​Convenience Methods

​Login Testing

​Form Filling

​Simple Test Method

​Configuration Options

​Handling Dynamic Content

​Error Handling & Retry Logic

​Test Result Analysis

​Advanced Scenarios

​Multi-Step Workflows

​Data Validation Testing

​Best Practices

​Writing Effective Instructions

​Optimizing Performance

​Test Organization

​Monitoring & Reporting

​Common Patterns

​Authentication Testing

​Navigation Testing

​Troubleshooting

​Common Issues

​Debugging Tips

Overview

Basic Agentic Test

Structured Output

Advanced Instructions

Convenience Methods

Login Testing

Form Filling

Simple Test Method

Configuration Options

Handling Dynamic Content

Error Handling & Retry Logic

Test Result Analysis

Advanced Scenarios

Multi-Step Workflows

Data Validation Testing

Best Practices

Writing Effective Instructions

Optimizing Performance

Test Organization

Monitoring & Reporting

Common Patterns

Authentication Testing

Navigation Testing

Troubleshooting

Common Issues

Debugging Tips