Start AI training data collection
Define the dataset contract
Choose a permitted source and document the intended training use, rights, languages, fields, retention and exclusion rules. Begin with an official export or a licensed corpus when it covers the need.
Collect one reproducible slice
Generate country-level Residential Flex credentials, verify the exit, and request a small fixed URL set. Record response time, source URL, license reference, parser version and document hash.
Version the accepted corpus
Separate extraction from acceptance. Review a sample for language, duplicate content and sensitive data, then freeze the manifest and record the bytes billed before increasing the collection.
Three documents, two separate quality gates
A fictional collection run checks rights and content quality independently. These rows show decisions, not a model-quality benchmark.
Report bytes and documents that pass rights, schema and quality checks for each source. Keep excluded material and duplicate counts in the run ledger; more downloaded pages alone do not establish a better corpus.
What to measure: accepted bytes per source
Report bytes and documents that pass rights, schema and quality checks for each source. Keep excluded material and duplicate counts in the run ledger; more downloaded pages alone do not establish a better corpus.
Start with a source you can reuse
For an existing broad corpus, review Common Crawl before scheduling another crawl. For your own sources, prefer a publisher export, licensed feed or direct agreement. A proxy is useful when the approved collection needs a particular network location; it is not a substitute for a source agreement.
Write the intended model use into the source register. Keep training, evaluation and retrieval datasets distinguishable so a later rights review or deletion can identify the affected dataset versions. Link each collected document to its original URL and source decision.
Keep extraction and acceptance separate
A successful HTTP response still needs validation. Reject error templates, login pages, empty text and unsupported formats before a document reaches a training shard. Store the original language label and the detector output separately; preserve enough metadata to revisit a mistaken label.
Normalize text for duplicate detection while retaining the source hash. Split by source or document family before creating evaluation sets so mirrored pages do not leak between splits. Review personal or sensitive information through the dataset owner’s approved process, with access controls and deletion rules.
Size collection from a measured batch
Illustrative sizing: 10,000 approved documents averaging 200,000 response bytes produce 2 GB of response payload using decimal GB. Request traffic, failed responses and overhead can change billed usage. If 8,000 documents pass your acceptance checks, keep that count beside the whole run’s bytes rather than reporting only successful downloads.
Start with Residential Flex plans for country-level volume. Move to Premium Residential when the research question requires a finer location selector. Use the bandwidth calculator and measured usage to size the next order.
Operate a collection you can stop and revise
Set one per-source request budget across all workers and exits. Identify the crawler accurately, follow the applicable robots rules, and stop at a login wall, challenge or explicit refusal. Robots directives describe crawler access preferences; they do not grant training rights.
Keep a named owner, a stop switch and an auditable retention policy. If a license or source decision changes, the corpus manifest should identify every affected document and derived dataset. Databay’s Acceptable Use Policy applies to the collection.
When the result looks wrong
The observed country differs
Confirm the generated country selector and inspect a fresh connection with the IP diagnostic.
Next step: Keep the mismatch in the record and correct the route before the sample.
A response is incomplete
Compare the content type, required fields and source template with the extraction contract.
Next step: Quarantine the response and review the parser before adding it to the accepted dataset.
The source returns a refusal or rate limit
Inspect the response class and the source-wide request budget.
Next step: Pause the queue, honor Retry-After and resolve the access or budget issue before continuing.
Sources and further reading
Published by Databay Research Team. The worked example is an illustrative exercise. Documentation and existing test evidence support the technical guidance; their scope and dates remain attached to the relevant sections.