Selenium Proxy Integration

What Is Selenium

Selenium is the longest-established browser automation framework: WebDriver bindings exist for Python, Java, C#, Ruby and JavaScript, and the same script can drive Chrome, Firefox, Edge and Safari. That breadth is why so much scraping and testing infrastructure is built on it. For proxy work, though, Selenium has one structural quirk you need to understand up front: WebDriver can tell a browser which proxy to use, but it has no standard way to supply proxy credentials, because the username/password prompt is browser UI that WebDriver does not script. Everything in this guide flows from that constraint, and all three workarounds below are used in production every day.

Connecting Selenium to Databay Proxies

Databay's gateway is gw.databay.co:8888; the pool, country and session are chosen with flags in the username, such as USER-zone-residential or USER-zone-datacenter. With Selenium you have three honest options for getting those credentials to the gateway: whitelist your server's IP so no credentials are needed, use Selenium Wire to inject authentication from a local proxy layer, or load a small browser extension that answers the auth challenge. IP whitelisting is the cleanest if your egress IP is static; the other two follow below.

Chrome Setup

Pointing Chrome at the gateway is one argument:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://gw.databay.co:8888')

driver = webdriver.Chrome(options=options)
driver.get('https://httpbin.org/ip')
print(driver.find_element('tag name', 'body').text)
driver.quit()

This works as-is only when your machine's IP is whitelisted in the Databay dashboard, because Chrome strips any credentials embedded in the --proxy-server URL and Selenium cannot type into the resulting auth popup. If you run from servers with stable IPs, whitelist them and stop here; it is the simplest production setup. To target the datacenter pool instead of residential, nothing changes on the Chrome side; the pool is a property of your credentials or whitelist configuration, not of the browser flags.

Firefox Setup

Firefox takes its proxy from preferences rather than a command-line flag:

from selenium import webdriver

options = webdriver.FirefoxOptions()
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.http', 'gw.databay.co')
options.set_preference('network.proxy.http_port', 8888)
options.set_preference('network.proxy.ssl', 'gw.databay.co')
options.set_preference('network.proxy.ssl_port', 8888)

driver = webdriver.Firefox(options=options)
driver.get('https://httpbin.org/ip')

The same caveat applies: there is no Firefox preference that supplies proxy credentials, and the master-password tricks that circulate in old forum posts are not reliable across versions. With Firefox your realistic options are IP whitelisting or Selenium Wire; the extension approach in this guide is Chrome-specific.

Authenticated Proxies with Selenium Wire

Selenium Wire extends the Python bindings with a local man-in-the-middle proxy that handles upstream authentication for you, so credential-based access works without whitelisting:

from seleniumwire import webdriver

seleniumwire_options = {
    'proxy': {
        'http': 'http://USER-zone-residential:PASS@gw.databay.co:8888',
        'https': 'http://USER-zone-residential:PASS@gw.databay.co:8888'
    }
}

driver = webdriver.Chrome(seleniumwire_options=seleniumwire_options)
driver.get('https://httpbin.org/ip')

Two things to know before adopting it. First, the project is no longer actively maintained, although it remains widely deployed and pinned versions continue to work; weigh that against your maintenance appetite. Second, because it decrypts HTTPS locally to do its job, browsers see Selenium Wire's own certificate; that is by design and local-only, but it means certificate errors you hit are usually about trusting the local CA, not about Databay's gateway. A useful bonus: driver.proxy can be reassigned at runtime, which makes Selenium Wire the only option here that can switch credentials without restarting the browser.

Authentication via a Chrome Extension

The third route is a tiny unpacked extension that sets the proxy and answers the auth challenge from inside Chrome. A Manifest V3 version needs two files. manifest.json:

{
  "manifest_version": 3,
  "name": "Databay Proxy Auth",
  "version": "1.0",
  "permissions": ["proxy", "webRequest", "webRequestAuthProvider"],
  "host_permissions": ["<all_urls>"],
  "background": { "service_worker": "background.js" }
}

and background.js:

chrome.proxy.settings.set({
  value: {
    mode: 'fixed_servers',
    rules: {
      singleProxy: { scheme: 'http', host: 'gw.databay.co', port: 8888 }
    }
  },
  scope: 'regular'
});

chrome.webRequest.onAuthRequired.addListener(
  (details, callback) => {
    callback({
      authCredentials: {
        username: 'USER-zone-residential',
        password: 'PASS'
      }
    });
  },
  { urls: ['<all_urls>'] },
  ['asyncBlocking']
);

Load it with options.add_argument('--load-extension=/path/to/extension'). Note that extensions do not load in Chrome's old headless mode; use the new headless implementation (--headless=new) or a virtual display. Generate the two files per worker if each worker needs different credentials, since the values are baked into the extension at load time.

Proxy Rotation Patterns

Selenium has no per-tab proxy concept: the proxy belongs to the browser process. Rotation therefore happens at driver granularity, which fits naturally with how a rotating gateway works. Databay's gateway assigns exit IPs per connection unless you pin one, so the controlling lever is the sessionId flag: USER-zone-residential-sessionId-abc123 keeps one exit IP for the whole driver lifetime, and a fresh id on the next driver gets a fresh IP. Add -countryCode-us (or another country code) to pin geography. The standard worker loop looks like this with Selenium Wire:

import random, string
from seleniumwire import webdriver

def fresh_session_options():
    sid = ''.join(random.choices(string.ascii_lowercase + string.digits, k=8))
    upstream = f'http://USER-zone-residential-sessionId-{sid}:PASS@gw.databay.co:8888'
    return {'proxy': {'http': upstream, 'https': upstream}}

for batch in url_batches:
    driver = webdriver.Chrome(seleniumwire_options=fresh_session_options())
    try:
        for url in batch:
            driver.get(url)
            # ... extract ...
    finally:
        driver.quit()

One sticky session per driver, one driver per batch of work, a new session id when a batch ends or a block appears. Avoid rotating faster than that: a browser page load opens many parallel connections, and an unpinned rotating session can deliver a single page from several IPs at once, which is a detection signal in itself. The trade-offs between sticky and rotating behaviour are covered in static vs rotating proxies.

Common Errors and Fixes

Most Selenium proxy failures fall into four buckets, and each has a distinctive signature.

HTTP 407 Proxy Authentication Required

With plain Selenium plus --proxy-server, a 407 (or a hanging auth popup in headed mode) means credentials never reached the gateway, which is expected: Chrome discards credentials in the proxy URL and WebDriver cannot answer the prompt. Switch to one of the three auth mechanisms above. With Selenium Wire, a 407 means the upstream credentials in seleniumwire_options are wrong, the zone flag is misspelled, or a special character in the password needs URL-encoding inside the proxy URL. With the extension, check that the extension actually loaded (it will not in old headless mode) and that the credentials baked into background.js are current. In every case, validate credentials outside the browser first:

curl -x http://USER-zone-residential:PASS@gw.databay.co:8888 https://httpbin.org/ip

If curl succeeds, the account is fine and the failure is in how credentials are being delivered to the browser.

ERR_TUNNEL_CONNECTION_FAILED

Chrome shows ERR_TUNNEL_CONNECTION_FAILED (Firefox: "The proxy server is refusing connections") when the CONNECT tunnel for an HTTPS site cannot be established. Verify the endpoint is exactly gw.databay.co:8888, then check the username flags: an invalid zone or malformed country code can make the gateway refuse the tunnel rather than return a clean 407. If curl through the same proxy works but the browser does not, look at what differs: a stale driver binary, a corporate firewall blocking the browser but not curl, or an extension that failed to apply its proxy settings before the first navigation. A one-second wait after driver startup before the first get() gives an auth extension's service worker time to register.

TLS and Certificate Errors

Databay's gateway tunnels HTTPS without terminating it, so genuine certificate errors rarely originate at the proxy. The big exception in a Selenium context is Selenium Wire, which by design decrypts traffic locally and presents its own certificate; its driver classes configure Chrome to accept that automatically, but hardened images or custom flags can break it, producing warnings on every page. If you are not using Selenium Wire and still see certificate errors, suspect corporate TLS interception on your own network or a genuinely broken target site. options.set_capability('acceptInsecureCerts', True) exists as an escape hatch and is reasonable for staging environments with self-signed certificates, but leaving it on for production scraping hides real interception warnings.

Timeouts and Slow Pages

Residential exits add latency on every connection a page opens, and Selenium's defaults assume a fast direct line. Three settings cover it:

driver.set_page_load_timeout(60)

options = webdriver.ChromeOptions()
options.page_load_strategy = 'eager'  # return at DOMContentLoaded

The eager page-load strategy makes driver.get() return once the DOM is parsed instead of waiting for every image and tracking pixel, which both speeds up scripts and cuts metered proxy bandwidth. Pair it with explicit waits (WebDriverWait) for the specific elements you need rather than raising the global timeout further. If one driver's session times out repeatedly while others are healthy, treat it as a slow exit IP: discard the session id and start a fresh driver instead of tuning around it.

Best Practices for Selenium with Proxies

• Whitelist where you can. If your workers have static egress IPs, IP whitelisting removes the entire credential problem and works identically for Chrome and Firefox.
• One driver, one identity. Pin each driver to its own sticky sessionId and keep cookies, session and exit IP aligned for the driver's lifetime.
• Rotate on evidence. Swap session ids when you hit 403s, 429s or CAPTCHAs, not on a timer; an unblocked IP is an asset.
• Trim page weight. The eager load strategy and disabling images via preferences reduce both runtime and bandwidth spend on metered residential traffic.
• Remember IP is only half the story. Selenium leaves automation fingerprints that rotation cannot hide; how sites detect headless browsers explains what targets actually check.
• Choose the right pool per target. Datacenter for speed and volume on permissive targets, residential where IP reputation is scored; the decision factors are laid out in residential vs datacenter proxies.