Home > Tasks > Web Operations > Selenium

Selenium Fetch Strategy

High-fidelity web content extraction using headless Chrome. Designed for JavaScript-heavy applications where standard HTTP clients fail to render the DOM.

Side-Effect: Safe JS-Enabled Auto-Fallback
⚙️ Fetch Configuration

{
  "method": "Selenium",
  "url": "https://app.example.com/dashboard",
  "wait_for_selector": ".main-content",
  "timeout": 30000
}
→
👁️ Rendered Output
✔ DOM Fully Rendered (142kb)
<div id="root">
  <nav class="sidebar">...</nav>
  <main class="main-content">
    <h1>Welcome, User</h1>
    ...
  </main>
</div>

Fetch Parameters

Parameter Type Description
url * String The target URL to navigate to and extract source from.
pool ExecutorService Thread pool used for managing the Selenium driver lifecycle.
webSearchDir File Directory for storing temporary artifacts or search context.
isSeleniumEnabled Boolean Global toggle. If a fetch fails, this is automatically set to false to trigger fallback.

Resource Usage

Token Consumption: Low (Retrieval only). Output length depends on target DOM density.

Compute: High (Requires local Chrome/Chromium instance).

Task Lifecycle

  1. Initialization: Checks if task.selenium is null. If so, instantiates Selenium2S3 with a fresh chromeDriver.
  2. Navigation: Commands the driver to navigate to the target url.
  3. Extraction: Retrieves the full page source via getPageSource().
  4. Cleanup: Executes quit() on the driver in a finally block to ensure no zombie processes remain.
  5. Error Handling: If an exception occurs, FetchConfig.isSeleniumEnabled is disabled and the system retries using the next available strategy (e.g., HttpClient).

Kotlin Implementation

Selenium.kt

class Selenium : FetchMethodFactory {
    override fun createStrategy(task: CrawlerAgentTask): FetchStrategy = object : FetchStrategy {
        override fun fetch(url: String, ...): String {
            return try {
                task.selenium = Selenium2S3(pool, driver = chromeDriver())
                task.selenium?.navigate(url)
                task.selenium?.getPageSource() ?: ""
            } finally {
                task.selenium?.quit()
            }
        }
    }
}