Start With the Data Source, Not the Proxy#
Use an official API, bulk download, licensed feed, or direct data partnership whenever available. Before requesting a page, document the source owner, exact fields and URLs, public or account-gated boundary, terms and robots review date, written authorization, personal or protected data, reuse rights, and retention.
A proxy changes network origin. It does not grant access, expand a quota, make collection lawful, or entitle a crawler to continue after a block.
Define One Source-Level Request Budget#
Set concurrency and request frequency for the destination as a whole, regardless of the number of workers or exit addresses. Cache unchanged responses; use conditional requests where supported; collect only required fields and assets; deduplicate URLs; and schedule around source guidance.
There is no universal safe requests-per-second figure. Begin below a documented quota or agreed limit, observe latency and errors, and reduce load when the source degrades. A pool must never multiply the traffic a source agreed to receive.
Choose a Network Origin#
Datacenter, residential, and mobile proxies provide hosting, ISP, and carrier network origins. Choose only when the approved workflow has a documented routing or regional-sampling requirement. Compare live location fulfillment, latency, usable-response rate, sourcing and consent, security, retention, support, and current cost.
No network type has a universal trust score or success rate. A destination can evaluate address history, protocol and browser data, cookies, account state, rate, timing, and behavior. Do not switch address classes to route around a control.
Sessions, Retries, and Stop Conditions#
Use a sticky session only when an authorized public flow requires short-term continuity. Connection reuse means a rotating gateway may keep one exit for several requests; log the observed address if it matters.
Retry an idempotent request only for a transient transport or server error, with capped exponential backoff and a global attempt budget. Treat 401, 403, 407 from the destination, 429, CAPTCHA, login, paywall, consent wall, cease-and-desist notice, or another access control as a stop or review signal. Never change identity to bypass it.
Data Quality and Provenance#
Store source URL, retrieval timestamp, apparent region, response status, content hash, parser version, and any missing or challenge state with each observation. Preserve raw evidence only as long as permitted, and separate facts from derived classifications.
Validate schemas, units, currency, tax and shipping treatment, product identity, duplicates, stale pages, and personalization. Confirm consequential findings with another authorized source or a human review. A successful HTTP response does not prove that the content is complete or representative.
A Minimal Technical Pattern#
Centralize authorization, caching, request budgets, and logging ahead of workers. A scheduler should emit only approved tasks; workers should respect the shared source limiter; a validator should quarantine incomplete or challenged responses; and a storage layer should retain provenance and deletion controls.
Keep credentials outside code, validate TLS certificates, restrict outbound destinations, minimize logs, redact secrets and personal data, and monitor source load and error classes. The proxy credential belongs in that controlled transport layer, not scattered through scraping code.
Review Before Launch#
Confirm: the approved source and access method; permitted fields and reuse; robots and terms review; privacy basis; request budget; cache and freshness plan; session need; stop and escalation conditions; destination allowlist; secret handling; provenance; retention and deletion; owner; and incident response.
Re-review whenever the source, method, data, law, purpose, volume, or recipient changes. If the workflow depends on looking like different people or preserving access after a refusal, replace it with an approved API, license, or written permission.