Navigation Engine Documentation
This section provides detailed documentation for the Advanced Web Scraper API’s Navigation Engine. The engine allows you to define complex, multi-step browser interactions to navigate websites, interact with elements, and extract data dynamically.
Table of Contents
- Navigation Step Types: Comprehensive reference guide detailing all available step types (
goto, click, input, extract, condition, forEachElement, mergeContext, etc.) and their configuration options. This is the core reference for building navigation flows.
- General Navigation Examples: Practical examples covering common scenarios like form filling, pagination handling, conditional logic, and data extraction patterns.
- Screenshot Feature: Learn how to enable and use the built-in screenshot capability for debugging and monitoring navigation flows.
Specific Examples
These documents provide in-depth walkthroughs for specific websites or advanced features:
- Google Trends Example: Demonstrates scraping dynamic content using
forEachElement to loop through items and mergeContext to combine extracted data.
- Rightmove Example: Shows how to handle a real-world property search site, including dealing with dynamic selectors, cookie consent, and debugging timeout issues.
Advanced / Internal
- System Prompt for AI Generation: (Internal) Details the system prompt used to guide Large Language Models (LLMs) in generating navigation configuration JSON.
Start with Navigation Step Types to understand the building blocks, then explore the General Navigation Examples for common patterns. Refer to the specific examples for more complex scenarios.