Cognotik | FetchMethod

Strategy Selection Kotlin / Enum


// Selecting the retrieval strategy
val method = FetchMethod.Selenium
val strategy = method.createStrategy(task)

strategy.fetch(
    url = "https://app.dynamic-site.com",
    webSearchDir = workspace,
    index = 1,
    pool = executor,
    orchestrationConfig = config
)

→

Execution Result CrawlerAgent Log

                    [INFO] Initializing Selenium WebDriver...

                    [INFO] Navigating to https://app.dynamic-site.com

                    [INFO] Waiting for DOM idle (3000ms)...

                    [SUCCESS] Content retrieved: 142kb

Dashboard - Dynamic App

<div id="root">
<nav>...</nav>
<main>Welcome, User!</main>
</div>

Available Fetch Methods

Method	Implementation	Best For
`Selenium`	Headless Chrome/Firefox	SPAs (React/Vue), sites requiring JS execution, or cookie handling.
`HttpClient`	Standard Java/Kotlin HTTP	Static documentation, APIs, and high-performance bulk crawling.

Global Configuration

Field	Type	Description
`isSeleniumEnabled`	Boolean	Global toggle in `FetchConfig` to enable/disable browser automation.

Retrieval Lifecycle

Factory Creation: The FetchMethod enum produces a FetchStrategy instance based on task requirements.
Environment Setup: For Selenium, the WebDriver is initialized; for HttpClient, the connection pool is verified.
Execution: The fetch() method is called with the target URL and orchestration context.
Persistence: Results are indexed and stored in the webSearchDir for future reference by the LLM.

Orchestration Setup

Add the following to your OrchestrationConfig to customize how the agent retrieves external data:


val config = OrchestrationConfig(
    // ... other settings
).apply {
    FetchConfig.isSeleniumEnabled = true // Enable browser support
}

// Usage in a CrawlerAgentTask
val fetcher = FetchMethod.HttpClient.createStrategy(this)