Selenium Proxy Integration

Selenium's current browser-options documentation defines manual proxy configuration through WebDriver capabilities. Authenticated-proxy behavior still depends on the browser, driver, proxy scheme, and bindings, so there is no one cross-browser credential pattern to promise. The examples below are version-sensitive alternatives: pin the browser, driver, Selenium, and any extension or interceptor dependency; use a non-sensitive endpoint you control; and keep credentials outside source control.

Databay's gateway is gw.databay.co:8888; the pool, country and session are chosen with flags in the username, such as USER-zone-residential or USER-zone-datacenter. With Selenium you have three workable options for getting those credentials to the gateway: whitelist your server's IP so no credentials are needed, use Selenium Wire to inject authentication from a local proxy layer, or load a small browser extension that answers the auth challenge. IP whitelisting is the cleanest if your egress IP is static; the other two follow below.

Pointing Chrome at the gateway is one argument:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://gw.databay.co:8888')

driver = webdriver.Chrome(options=options)
driver.get('https://httpbin.org/ip')
print(driver.find_element('tag name', 'body').text)
driver.quit()

This works as-is only when your machine's IP is whitelisted in the Databay dashboard, because Chrome strips any credentials embedded in the --proxy-server URL and Selenium cannot type into the resulting auth popup. If you run from servers with stable IPs, whitelist them and stop here; it is the simplest production setup. To target the datacenter pool instead of residential, nothing changes on the Chrome side; the pool is a property of your credentials or whitelist configuration, not of the browser flags.

Firefox takes its proxy from preferences rather than a command-line flag:

from selenium import webdriver

options = webdriver.FirefoxOptions()
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.http', 'gw.databay.co')
options.set_preference('network.proxy.http_port', 8888)
options.set_preference('network.proxy.ssl', 'gw.databay.co')
options.set_preference('network.proxy.ssl_port', 8888)

driver = webdriver.Firefox(options=options)
driver.get('https://httpbin.org/ip')

The same caveat applies: there is no Firefox preference that supplies proxy credentials, and the master-password tricks that circulate in old forum posts are not reliable across versions. With Firefox your realistic options are IP whitelisting or Selenium Wire; the extension approach in this guide is Chrome-specific.

Selenium Wire extends the Python bindings with a local man-in-the-middle proxy that handles upstream authentication for you, so credential-based access works without whitelisting:

from seleniumwire import webdriver

seleniumwire_options = {
    'proxy': {
        'http': 'http://USER-zone-residential:PASS@gw.databay.co:8888',
        'https': 'http://USER-zone-residential:PASS@gw.databay.co:8888'
    }
}

driver = webdriver.Chrome(seleniumwire_options=seleniumwire_options)
driver.get('https://httpbin.org/ip')

Two things to know before adopting it. First, the project is no longer actively maintained, although it remains widely deployed and pinned versions continue to work; weigh that against your maintenance appetite. Second, because it decrypts HTTPS locally to do its job, browsers see Selenium Wire's own certificate; that is by design and local-only, but it means certificate errors you hit are usually about trusting the local CA, not about Databay's gateway. A useful bonus: driver.proxy can be reassigned at runtime, which makes Selenium Wire the only option here that can switch credentials without restarting the browser.

The third route is a tiny unpacked extension that sets the proxy and answers the auth challenge from inside Chrome. A Manifest V3 version needs two files. manifest.json:

{
  "manifest_version": 3,
  "name": "Databay Proxy Auth",
  "version": "1.0",
  "permissions": ["proxy", "webRequest", "webRequestAuthProvider"],
  "host_permissions": ["<all_urls>"],
  "background": { "service_worker": "background.js" }
}

and background.js:

chrome.proxy.settings.set({
  value: {
    mode: 'fixed_servers',
    rules: {
      singleProxy: { scheme: 'http', host: 'gw.databay.co', port: 8888 }
    }
  },
  scope: 'regular'
});

chrome.webRequest.onAuthRequired.addListener(
  (details, callback) => {
    callback({
      authCredentials: {
        username: 'USER-zone-residential',
        password: 'PASS'
      }
    });
  },
  { urls: ['<all_urls>'] },
  ['asyncBlocking']
);

Load it with options.add_argument('--load-extension=/path/to/extension'). Note that extensions do not load in Chrome's old headless mode; use the new headless implementation (--headless=new) or a virtual display. Generate the two files per worker if each worker needs different credentials, since the values are baked into the extension at load time.

Selenium has no per-tab proxy concept: the proxy belongs to the browser process. Session lifecycle therefore happens at driver granularity. A sessionId such as USER-zone-residential-sessionId-abc123 can provide short-term continuity, subject to live availability, while -countryCode-us selects a network-origin country. The worker loop below creates one documented session for each authorized batch:

import random, string
from seleniumwire import webdriver

def fresh_session_options():
    sid = ''.join(random.choices(string.ascii_lowercase + string.digits, k=8))
    upstream = f'http://USER-zone-residential-sessionId-{sid}:PASS@gw.databay.co:8888'
    return {'proxy': {'http': upstream, 'https': upstream}}

for batch in url_batches:
    driver = webdriver.Chrome(seleniumwire_options=fresh_session_options())
    try:
        for url in batch:
            driver.get(url)
            # ... authorized work ...
    finally:
        driver.quit()

Use one session per driver only when cookie and network continuity are required, and end it when the approved batch ends. A block or CAPTCHA is a stop or backoff signal, not a reason to create another session. The trade-offs between sticky and rotating behavior are covered in static vs rotating proxies.

Most Selenium proxy failures fall into four buckets, and each has a distinctive signature.

With plain Selenium plus --proxy-server, a 407 (or a hanging auth popup in headed mode) means credentials never reached the gateway, which is expected: Chrome discards credentials in the proxy URL and WebDriver cannot answer the prompt. Switch to one of the three auth mechanisms above. With Selenium Wire, a 407 means the upstream credentials in seleniumwire_options are wrong, the zone flag is misspelled, or a special character in the password needs URL-encoding inside the proxy URL. With the extension, check that the extension actually loaded (it will not in old headless mode) and that the credentials baked into background.js are current. In every case, validate credentials outside the browser first:

curl -x http://USER-zone-residential:PASS@gw.databay.co:8888 https://httpbin.org/ip

If curl succeeds, the account is fine and the failure is in how credentials are being delivered to the browser.

Chrome shows ERR_TUNNEL_CONNECTION_FAILED (Firefox: "The proxy server is refusing connections") when the CONNECT tunnel for an HTTPS site cannot be established. Verify the endpoint is exactly gw.databay.co:8888, then check the username flags: an invalid zone or malformed country code can make the gateway refuse the tunnel rather than return a clean 407. If curl through the same proxy works but the browser does not, look at what differs: a stale driver binary, a corporate firewall blocking the browser but not curl, or an extension that failed to apply its proxy settings before the first navigation. A one-second wait after driver startup before the first get() gives an auth extension's service worker time to register.

Databay's gateway tunnels HTTPS without terminating it, so genuine certificate errors rarely originate at the proxy. The big exception in a Selenium context is Selenium Wire, which by design decrypts traffic locally and presents its own certificate; its driver classes configure Chrome to accept that automatically, but hardened images or custom flags can break it, producing warnings on every page. If you are not using Selenium Wire and still see certificate errors, suspect corporate TLS interception on your own network or a genuinely broken target site. options.set_capability('acceptInsecureCerts', True) exists as an escape hatch and is reasonable for staging environments with self-signed certificates, but leaving it on for production scraping hides real interception warnings.

Residential exits add latency on every connection a page opens, and Selenium's defaults assume a fast direct line. Three settings cover it:

driver.set_page_load_timeout(60)

options = webdriver.ChromeOptions()
options.page_load_strategy = 'eager'  # return at DOMContentLoaded

The eager page-load strategy makes driver.get() return once the DOM is parsed instead of waiting for every image and tracking pixel, which both speeds up scripts and cuts metered proxy bandwidth. Pair it with explicit waits (WebDriverWait) for the specific elements you need rather than raising the global timeout further. If one driver's session times out repeatedly while others are healthy, treat it as a slow exit IP: discard the session id and start a fresh driver instead of tuning around it.

• Use IP allowlisting on systems you control when it simplifies authentication.
• Pin a driver to a sticky sessionId only when an authorized workflow needs cookie and network continuity.
• Treat 403, 429, and CAPTCHA responses as stop or backoff signals; do not swap session IDs to bypass them.
• The eager load strategy and omitting out-of-scope images can reduce source load, runtime, and bandwidth.
• Use accurate browser settings and do not use a proxy to conceal unauthorized automation.
• Choose datacenter or residential origins from the documented test requirement, not as an escalation path around blocking.