Web scraping proxies for usable data

Start with the source agreement and the network-origin variable the job actually requires. All Databay products are shared pools, bill by transferred traffic, support HTTP proxying, HTTPS CONNECT, and SOCKS5, rotate on a new connection, and can repeat an exit. A new connection does not guarantee a fresh IP.

Reject every option when the job requires a permanent allowlisted address, an exclusively assigned IP, guaranteed continuity, a specific mobile carrier, or access that the source has refused. Network class is a measurement input, not a permission or success tier.

Do not size a pilot from an advertised pool count. Open the matching live pricing page above and choose its smallest currently orderable package; the page supplies the current allowance, payable total, traffic validity, and checkout path. There is no paid-network free trial. Begin with one request and one worker, then measure bytes per accepted record before buying more traffic.

Prove only the proxy hop first. The command below uses Databay's owned IP diagnostic, finite timeouts, explicit Basic proxy authentication, and a configuration streamed over standard input so the expanded password is not a cURL argument. It is a connection check, not an inventory or destination-success benchmark.

{
  printf 'proxy = "%s"\n' 'http://gw.databay.co:8888'
  printf 'proxy-user = "%s:%s"\n' "$DATABAY_PROXY_USER" "$DATABAY_PROXY_PASS"
  printf '%s\n' 'proxy-basic' 'noproxy = ""' 'connect-timeout = 5' 'max-time = 15' 'fail-with-body' 'silent' 'show-error'
  printf 'url = "%s"\n' 'https://databay.com/what-is-my-ip/json'
} | curl --disable --config -

This POSIX-shell form was checked against an isolated loopback proxy harness; it does not contact a third-party target. Keep secrets in an approved secret store, disable shell tracing, and use curl.exe explicitly when PowerShell aliases curl. The tested cURL proxy guide covers Windows, 407, CONNECT, SOCKS5h, and redacted diagnostics; the Python Requests and Scrapy guides provide application examples. Replace the diagnostic only with an endpoint you own or are expressly authorized to collect.

Databay publishes connection-based rotation and bounded sticky-session ceilings, but it does not publish a universal concurrency allowance for customer collectors. Start at one in-flight request. Increase workers only while the source's documented source-wide allowance, your response validator, storage pipeline, bandwidth budget, and stop controls all remain inside their declared limits. Apply one per-host budget across every exit, account, process, and region; more IP addresses do not create more permission or quota.

Connection reuse matters. A pooled HTTP client can send several requests through one connection, while a browser can open many connections for one page. Without a session label, a new connection may select another shared exit but can repeat the prior exit. Use rotation only for independent approved samples. Request a sticky session when a short authorized flow needs cookies or route continuity, keep one label for that flow, and design for early route loss. Do not change labels or open more connections after a block, challenge, quota response, or account restriction.

Predeclare the maximum in-flight requests, requests per minute, bytes per minute, total pages, retry count, total deadline, and shutdown owner. Raise only one limit at a time and retain failed and repeated routes in the result rather than hiding them.

Prefer an official API, feed, bulk export, license, or publisher partnership. For a remaining public-page need, create a source register with the owner, written authority, permitted paths and fields, purpose, user agent and contact, robots URL, applicable terms, rate guidance, retention, republication rights, review date, and the person who can stop the run. Public visibility alone is not authorization.

The Robots Exclusion Protocol is a crawler-control standard, not a grant of access. Identify the crawler accurately and obey the rules that apply to it. If robots retrieval fails because the service is unavailable, use the standard's conservative complete-disallow behavior until it can be checked; do not interpret the outage as permission. Authentication, login, paywall, and technical-control boundaries still apply even when a path is not disallowed.

Limit the first collection to the smallest URL set and fields that can answer the stated decision. Cache stable reference data, use ETag or Last-Modified validators where the source supports them, and omit images, fonts, media, and scripts that are outside the approved scope. Re-run the authority review when a redirect changes host, the source terms change, the schema expands, or the downstream use changes.

Classify the first failed layer before retrying. A 407 belongs to the proxy hop: check the Databay credential and documented routing flags without exposing the password. A destination 401 or 403, login wall, CAPTCHA, or explicit refusal is a stop-and-review outcome. It is not a trigger for another exit, account, fingerprint, or protocol.

RFC 6585 defines 429 and its optional Retry-After signal, while RFC 9110 defines HTTP semantics and validators. Preserve status, headers, attempt number, and final decision. Never discard refusals to calculate a cleaner success rate.

Store the source URL, UTC request and response time, method, response status, content type, content length, ETag or Last-Modified value, requested and independently observed route, proxy product, client and parser version, schema version, content hash, field-level missingness, retry count, and result class. Record whether account, cookie, language, device, and session state were present, but hash or redact identifiers and never log proxy passwords, authorization headers, cookies, tokens, raw personal data, or full challenge pages.

Validate response type and required fields before parsing. Quarantine HTML error documents returned with a 200, challenge pages, consent walls, partial bodies, currency or locale mismatches, duplicate records, and implausible field changes. Manually review a declared sample from each source and parser revision. A large pool cannot correct selection bias, a stale cache, or a broken parser.

Measure accepted-record ratio, missingness, duplicate rate, age at use, transferred bytes, retry count, and cost per accepted record = (proxy spend + compute + review) / accepted records. Set pilot gates before the run and include all failures in the denominator. These are customer measurements for the approved target, not Databay success claims.

Use geographic targeting only when network origin is part of an authorized research question. Hold the URL, product or subject, language, cookies, account state, device profile, delivery assumption, client version, and collection window constant. Change one supported route selector, verify the observed exit independently, and repeat enough declared samples to distinguish a route-specific observation from normal variation.

Country, state, city, ZIP, coordinate, or ASN selection changes a network signal. It does not set GPS, browser language, timezone, tax residence, delivery address, account history, experiment bucket, or personalization. Report the bounded conclusion: the named response observed under the recorded conditions at that time. Do not describe one proxy result as what everyone in a location sees or infer city coverage from a product label; live route availability must be checked.

Databay's Acceptable Use Policy requires respect for target terms and robots instructions and prohibits harmful load, unauthorized access, abusive scraping, fake-account creation, and attempts to defeat security or rate controls. Do not collect login-gated or non-public data without the owner's express authorization, use stolen or shared credentials, evade a block, solve or route around a CAPTCHA, falsify client identity, or continue after the source withdraws permission.

Minimize personal data, secrets, copyrighted material, and sensitive fields. Define access control, encryption, retention, deletion, correction, and downstream-use rules before collection. Contract, copyright, privacy, database, consumer-protection, and computer-access obligations vary by source and jurisdiction; robots compliance and proxy access do not settle them. Obtain qualified legal and security review for the actual program.

Pause when authorization is missing, terms are ambiguous, robots cannot be evaluated, a response introduces a new access boundary, or the collection's purpose changes. A defensible web scraping proxy workflow can name its owner, source, allowed fields, source-wide budget, stop conditions, evidence record, and deletion path before the first paid request.