Interface: UrlIngestionCapability
@kortexya/reasoninglayer / UrlIngestionCapability
Interface: UrlIngestionCapability
Section titled “Interface: UrlIngestionCapability”Defined in: src/types/ingestion.ts:2098
The URL-ingestion capability and its limits.
Remarks
Section titled “Remarks”The limits bound the worst case of one crawl. A crawl sends at most
maxPages × (maxRedirects + 1) × (maxRedirects + 2) requests: each page
URL and each robots.txt fetch costs at most maxRedirects + 1 requests.
Each request waits at most
max(maxCrawlDelayMs, minRequestIntervalMs) + requestTimeoutMs
milliseconds. With the defaults (100 pages, 5 redirects, 10 s delay, 30 s
timeout) that bound is about 46.7 hours, so it is a ceiling to show, not a
client deadline to set.
Properties
Section titled “Properties”available
Section titled “available”available:
boolean
Defined in: src/types/ingestion.ts:2100
Always true: fetching needs no configuration.
maxCrawlDelayMs
Section titled “maxCrawlDelayMs”maxCrawlDelayMs:
number
Defined in: src/types/ingestion.ts:2106
The longest robots.txt Crawl-delay honoured, in milliseconds (10000).
A longer one is capped here. Two requests to one host leave
max(min(Crawl-delay, maxCrawlDelayMs), minRequestIntervalMs) apart.
maxDepthLimit
Section titled “maxDepthLimit”maxDepthLimit:
number
Defined in: src/types/ingestion.ts:2108
The largest maxDepth a request may ask for.
maxPagesLimit
Section titled “maxPagesLimit”maxPagesLimit:
number
Defined in: src/types/ingestion.ts:2110
The largest maxPages a request may ask for.
maxRedirects
Section titled “maxRedirects”maxRedirects:
number
Defined in: src/types/ingestion.ts:2115
Redirect hops followed per page URL and per robots.txt fetch (5). Either
costs at most maxRedirects + 1 requests.
minRequestIntervalMs
Section titled “minRequestIntervalMs”minRequestIntervalMs:
number
Defined in: src/types/ingestion.ts:2117
Minimum milliseconds between two requests to one host.
requestTimeoutMs
Section titled “requestTimeoutMs”requestTimeoutMs:
number
Defined in: src/types/ingestion.ts:2123
The longest one request may take, connection and body included, in
milliseconds (30000). A page whose request outlasts it is skipped as
fetch_failed.
robotsProductToken
Section titled “robotsProductToken”robotsProductToken:
string
Defined in: src/types/ingestion.ts:2125
The robots.txt product token the crawler sends and obeys.