Responsible Web Data Collection: Authorization and Controls

Before collection, document the source owner, business purpose, exact URLs and fields, public or account-gated boundary, access method, terms review date, license or written permission, reuse rights, personal or protected data, retention, deletion process, and people responsible for review. Prefer an official API, export, bulk download, feed, or data partnership.

Technical accessibility is not permission. A proxy changes network origin; it does not grant a license, expand a quota, waive contract, or make copying lawful. This guide is operational information, not legal advice.

RFC 9309 standardizes the Robots Exclusion Protocol and explicitly says its rules are not access authorization. Fetch and cache the current file, apply the record for the crawler's actual product token, and retain the retrieval time and parser version.

An allow rule does not settle terms, copyright, privacy, authentication, or technical-access questions. A disallow rule is a strong reason to stop and seek an approved source unless the owner has provided specific written authorization for the workflow.

Do not invent a universal requests-per-second number from site size or brand. Use published quotas, Retry-After, API limits, owner guidance, written scope, and measured service health. Begin with the minimum needed, cache unchanged resources, deduplicate URLs, use conditional requests where supported, apply jittered exponential backoff for transient failures, and cap retries.

Enforce the budget per source across every worker, account, region, and proxy exit. Distributing traffic across IPs does not reduce aggregate load. Stop on persistent errors, latency degradation, 401, 403, 429, CAPTCHA, or another access control.

Never bypass login, paywall, CAPTCHA, purchase limit, queue, rate limit, IP block, or other technical control without the system owner's explicit written authorization for that exact test. Do not replace accounts, fingerprints, payment details, or proxy exits to continue after a refusal.

US computer-access law is fact-specific. Van Buren v. United States and the Ninth Circuit's hiQ opinion address particular CFAA questions; neither is blanket permission to scrape public pages or ignore other claims, jurisdictions, contracts, or later facts. Obtain counsel for consequential collection.

Public availability does not remove personal-data obligations. Under GDPR Article 6, processing needs a lawful basis; other duties can include transparency, purpose limitation, minimization, accuracy, retention limits, security, rights handling, and transfer controls. Sensitive, children's, location, biometric, employment, and account data can require stricter review.

Inventory fields before collection, remove unnecessary identifiers, restrict access, encrypt storage, define deletion, and maintain a way to correct or erase records when applicable. Consult qualified privacy counsel for the people and jurisdictions in scope.

Store source URL, retrieval time, response status and hash, relevant headers, parser and schema version, authorization reference, market and account state, extraction result, and validation status. Quarantine partial pages, challenges, unexpected templates, and outliers rather than treating them as valid data.

Measure freshness, missingness, duplicate rate, parse failures, match confidence, confirmed errors, and deletion compliance. Keep protected expression out of downstream products unless reuse is licensed; facts and expressive content require different analysis.

Where appropriate and accepted by the source, use an accurate product token and a monitored contact page or address. Do not impersonate a browser or person to conceal unauthorized automation. Keep credentials in a secret manager, minimize permissions, rotate them through the approved process, redact logs, and separate development from production.

Unknown public proxies are unsuitable for authenticated, personal, or private data. If regional routing is authorized, use accountable infrastructure and log the observed exit.

Maintain a source register with owner, purpose, data rights, fields, robots and terms review, privacy assessment, request budget, stop signals, retention, deletion, incident contact, and next review date. Require approval for new sources and for material changes in purpose, scale, fields, or access boundary.

Audit logs and samples regularly. Suspend a source when its rules change, authorization expires, quality drops, complaints arrive, or controls appear. A sustainable program can explain why every field was collected and remove it when the justification ends.