Python Requests Proxy Setup: Sessions, Retries, and Errors

For an application-owned request, pass the proxy mapping on the request, use separate connect and read timeouts, and check the HTTP result explicitly. The mapping keys describe the destination URL scheme; the values describe the proxy connection. An HTTPS destination through an http:// proxy is normal: Requests asks the proxy for a CONNECT tunnel, then performs destination TLS through that tunnel.

import os
from urllib.parse import quote

import requests

user = quote(os.environ["DATABAY_PROXY_USER"], safe="")
password = quote(os.environ["DATABAY_PROXY_PASS"], safe="")
proxy_url = f"http://{user}:{password}@gw.databay.co:8888"
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get(
    "https://api.ipify.org?format=json",
    proxies=proxies,
    timeout=(5, 20),
)
response.raise_for_status()
print(response.json())

quote(..., safe="") percent-encodes reserved characters before the username and password enter a URL. Environment variables are convenient inputs, not a universal secret store: load them from the deployment platform's secret manager, never print proxy_url or proxies, and keep exception and debug logging from serializing them. Use a harmless endpoint you control when possible. An IP response describes one observed request; it does not prove privacy, future routing, trustworthiness, or destination permission.

The most common Requests configuration error is assuming the dictionary key names the proxy protocol. In {"https": "http://proxy:8080"}, https means “use this entry when the destination URL begins with HTTPS.” The value begins with http:// because the client speaks HTTP proxy protocol to the gateway.

Using an https:// proxy value is a different feature: it requests TLS on the client-to-proxy hop. Do not change the proxy scheme merely because the destination is HTTPS. First confirm the gateway's documented protocol, then configure the destination key and proxy value independently.

We ran a version-pinned localhost harness on 2026-07-27 with Python 3.13.14, Requests 2.34.2, urllib3 2.7.0, PySocks 1.7.1 and certifi 2026.7.22. It bound synthetic HTTP and SOCKS5 listeners to 127.0.0.1, used example.test or localhost as the named destination, and did not contact that destination, Databay's gateway, or a public proxy. Lab credentials were redacted before output.

This is protocol-shape and client-behavior evidence, not a speed benchmark or availability claim. Rerun the harness after changing Python, Requests, urllib3, PySocks, certifi, or the retry and proxy code.

A requests.Session persists cookies and default headers and uses urllib3 connection pools. That is useful for repeated calls to the same authorized service, but it also means a rotating gateway may stay on one underlying proxy connection longer than a naive “one request, one exit” model suggests. The localhost capture sent two GETs through the same socket when both ran inside one Session.

import requests

with requests.Session() as session:
    for endpoint in approved_endpoints:
        response = session.get(
            endpoint,
            proxies=proxies,       # explicit on each request
            timeout=(5, 20),
        )
        response.raise_for_status()

Requests' official advanced guide warns that session.proxies values can be overwritten by environment proxies. Our capture reproduced that behavior: with one proxy stored on the Session and a different HTTP_PROXY in the process environment, the environment listener received the request. Passing proxies= on the request selected the explicit listener instead.

For a tightly controlled worker, session.trust_env = False disables environment-derived proxy behavior, but it is broader than a proxy switch: it also affects environment authentication and CA-bundle discovery. If you disable it, set every required proxy, authentication and trust input explicitly and test the deployed container rather than only a laptop shell.

HTTP 407 means a proxy is challenging proxy authentication. RFC 9110 requires a 407 response to include Proxy-Authenticate; the corresponding request field is Proxy-Authorization. This is different from a destination's 401, 403, or 429. Requests normally derives Basic proxy authorization from the user info in the proxy URL, so percent-encoding and redaction are part of the setup rather than cosmetic details.

Requests returns an HTTP response object for a plain HTTP 407. It becomes an HTTPError only when code calls raise_for_status(). The local harness recorded exactly one request, status 407, and then HTTPError. An HTTPS CONNECT rejection can surface as a ProxyError because the tunnel was never established, so preserve both the exception type and any proxy status in the message.

1. Confirm which host and port accepted the TCP connection.
2. Check whether the failure occurred on a plain proxy response or during CONNECT.
3. Verify that the username and password were encoded as URL user info and were not accidentally assigned to destination auth=.
4. Check documented pool, country and session flags one at a time.
5. Run one permitted destination without concurrency or retries.

Do not retry a 407 with new identities or random credentials. Correct the configuration or stop. Never place a complete credential-bearing proxy URL in a ticket, traceback, screenshot, metrics label, or structured log.

SOCKS support is an optional Requests extra. Pin it with the rest of the application dependencies, then choose the scheme from the DNS requirement:

# requirements.txt
requests[socks]==2.34.2

# Client resolves the destination name, then sends an IP address.
local_dns = "socks5://user:password@proxy.example:1080"

# Client sends the hostname and asks the SOCKS proxy to handle it.
proxy_dns = "socks5h://user:password@proxy.example:1080"

response = requests.get(
    "https://service.example/health",
    proxies={"http": proxy_dns, "https": proxy_dns},
    timeout=(5, 20),
)

RFC 1928 allows a SOCKS5 CONNECT request to contain an IPv4 address, IPv6 address, or domain name. In the local capture, Requests with socks5h:// used address type 3 and sent localhost; socks5:// resolved locally and used address type 1 with 127.0.0.1. That observation applies to the tested client versions. It does not prove that every lookup made by a larger application follows the same route: other libraries, service discovery, prefetching, telemetry, and direct fallbacks need their own review.

Requests does not time out by default. A tuple such as timeout=(5, 20) supplies a connect timeout and a read timeout. The read value is not a wall-clock cap on the entire download; it bounds how long the client waits for bytes on the socket. A peer that keeps sending data can keep a response alive beyond that number.

from requests.exceptions import ConnectTimeout, ReadTimeout

try:
    response = session.get(
        approved_url,
        proxies=proxies,
        timeout=(5, 20),
    )
    response.raise_for_status()
except ConnectTimeout:
    record_failure("proxy-or-destination-connect-timeout")
except ReadTimeout:
    record_failure("response-read-timeout")

Choose numbers from the worker's latency distribution and user-visible time budget. If the entire job has a hard deadline, enforce it in the job runner or task system as well as in Requests; do not mislabel a socket-inactivity timeout as a total deadline. Streaming downloads require their own byte limit, content-type validation, and deadline because every new chunk resets the read-wait calculation.

Requests does not provide a broad automatic retry policy for application responses. Configure urllib3's Retry through an HTTPAdapter, allow only methods whose replay is safe for the specific job, choose statuses deliberately, honor Retry-After, and keep one total attempt budget across every proxy exit.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=2,
    connect=2,
    read=0,
    status=2,
    allowed_methods={"GET", "HEAD", "OPTIONS"},
    status_forcelist={429, 502, 503, 504},
    backoff_factor=0.5,
    respect_retry_after_header=True,
    raise_on_status=False,
)

with requests.Session() as session:
    session.mount("https://", HTTPAdapter(max_retries=retry))
    response = session.get(
        approved_url,
        proxies=proxies,
        timeout=(5, 20),
    )
    response.raise_for_status()

The example is a starting contract, not a universal list. A 429 may require a pause longer than the job can accept; a source's terms or response may prohibit further calls. Do not include 401, 403, or 407 in a retry status list. Do not add POST merely because a service supports idempotency keys unless the complete server-side contract has been verified.

In the capture, a GET policy with total=2 made three attempts and reached a synthetic 204 after two 503 responses. A POST through the same adapter received one 503 and was not replayed. This is why a method allowlist is a safety boundary, not a performance tweak.

Requests verifies destination certificates by default. Keep verify=True. A plain HTTP proxy carrying an HTTPS destination does not require a fake destination certificate: it opens a tunnel, then Requests validates the destination inside that tunnel. If an approved corporate environment intentionally intercepts TLS, use its managed CA bundle and document the trust boundary; do not use verify=False as a proxy fix.

with requests.Session() as session:
    session.verify = "/run/secrets/approved-ca-bundle.pem"
    response = session.get(
        approved_url,
        proxies=proxies,
        timeout=(5, 20),
    )

Requests documents REQUESTS_CA_BUNDLE and a CURL_CA_BUNDLE fallback when environment trust is enabled. If code uses trust_env=False, set the approved bundle directly as shown and keep its update process separate from application releases. Classify SSLError only after identifying whether the failed TLS connection was to an HTTPS proxy or to the destination through a tunnel.

Catch the narrow Requests exceptions your code can act on, log a redacted classification, and let an unexpected RequestException reach the job's normal failure path. Do not turn every exception into a retry.

A proxy can succeed while the destination rejects a request, and a destination can be healthy while the proxy authentication fails. Keep those layers separate in metrics so a rise in 407 does not look like a rise in destination 403.

First establish one request with the base pool username. Then add one documented control at a time so each test introduces only one variable:

# Base pool
DATABAY_PROXY_USER="USER-zone-residential"

# Request a US exit
DATABAY_PROXY_USER="USER-zone-residential-countryCode-us"

# Request short-lived continuity for one authorized workflow
DATABAY_PROXY_USER="USER-zone-residential-countryCode-us-sessionId-qa-us-01"

A country selector changes one network-location signal; it does not recreate a local account, language, device, cookies, GPS, or platform state. A session identifier requests route continuity; it is not a dedicated identity or a guarantee that a destination will accept the traffic. Choose datacenter proxies for permitted jobs that prioritize infrastructure stability and throughput, or residential proxies when the approved test genuinely requires a consumer-network route or country sample.

Use the browser-only format converter when the problem is encoding one endpoint safely, the proxy checker methodology when the problem is point-in-time liveness evidence, and the HTTP versus SOCKS5 guide when the unresolved question is protocol or DNS ownership. None of those tools establishes permission to access a destination.

1. Pin Python, Requests, urllib3, PySocks when used, certifi and the application lockfile; record the tested build date.
2. Use a controlled destination and one request with no redirects, retries, concurrency, or session flags.
3. Pass the proxy mapping explicitly, or document every environment variable and NO_PROXY rule the worker inherits.
4. Percent-encode user info and confirm logs, traces, exception serializers and metrics never emit the credential-bearing URL.
5. Set connect and read timeouts plus an outer job deadline and maximum response size.
6. Call raise_for_status() or evaluate expected statuses before parsing a response body.
7. Classify proxy connection, proxy authentication, tunnel or SOCKS, destination TLS, and destination HTTP separately.
8. Allow retries only for approved replay-safe operations under a source-level attempt budget.
9. Stop on unclear permission, authentication boundaries, persistent 403 or 429, CAPTCHA, quota, or account enforcement.
10. Rerun the localhost harness after a dependency or routing change and investigate changed wire behavior before updating the tested date.

This guide solves the client setup without requiring a purchase. A Databay account is the commercial next step only when the workload is authorized and a managed residential or datacenter route fits the documented requirement.