Puppeteer Proxy Integration

Puppeteer is a Node.js library maintained by the Chrome team that drives Chrome or Chromium over the DevTools Protocol. It is useful for authorized browser testing, PDF generation, screenshots, and pages that require JavaScript. A proxy changes the browser's network route and can add an approved country-origin sample; it does not conceal automation, grant permission, or reproduce every local user's state.

Databay exposes a single gateway endpoint, gw.databay.co:8888, and you select the proxy pool, country and session behavior through flags appended to your username. That means the Puppeteer side is always the same two steps: pass the gateway to Chromium with --proxy-server, then supply credentials with page.authenticate(). The split matters because Chromium does not accept credentials embedded in the --proxy-server URL. If you write --proxy-server=http://user:pass@host:port, Chromium silently strips the credentials and you get a 407 on the first navigation. Authentication must happen through the DevTools protocol, which is exactly what page.authenticate() does.

Residential proxies exit through real household connections, which makes them the right pool for targets that aggressively filter datacenter IP ranges. The zone is selected with -zone-residential in the username:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--proxy-server=http://gw.databay.co:8888']
  });

  const page = await browser.newPage();
  await page.authenticate({
    username: 'USER-zone-residential',
    password: 'PASS'
  });

  await page.goto('https://httpbin.org/ip', { waitUntil: 'domcontentloaded' });
  console.log(await page.evaluate(() => document.body.innerText));

  await browser.close();
})();

Two details are load-bearing here. First, page.authenticate() must be called before the first page.goto(); it registers a handler for the proxy's 407 challenge, so calling it after navigation has already failed does nothing. Second, the --proxy-server argument applies to the whole browser process, not just one page; every page you open in this browser uses the gateway.

Datacenter proxies and residential proxies come from different network classes, with different measured cost, latency, and availability. Choose the class required by the authorized workflow rather than using another class to route around a source control. Only the username changes:

const page = await browser.newPage();
await page.authenticate({
  username: 'USER-zone-datacenter',
  password: 'PASS'
});

Test the selected pool against a system you control or have permission to access. Treat a block or CAPTCHA as a stop or backoff signal and move to an official API, licensed feed, or documented permission path.

Geo-targeting and session control are username flags. Appending -countryCode-us requests a US-geolocated exit; appending -sessionId-abc123 requests short-term continuity, subject to live availability:

// Authorized US network sample with short-term continuity
await page.authenticate({
  username: 'USER-zone-residential-countryCode-us-sessionId-abc123',
  password: 'PASS'
});

Use a session ID only when the approved test needs continuity. It is not a dedicated device or access guarantee, and it must not be used to cross a login, cart, CAPTCHA, block, or other restricted boundary.

A rotating gateway assigns exit IPs per connection, but a browser is not a polite single-connection client: one page load can open dozens of parallel connections for HTML, scripts, images and XHR. If those connections each rotate, a single page render arrives at the target from several IPs at once, which looks stranger than no proxy at all. For an authorized browser workflow that needs continuity, request one sticky session for that workflow and do not rotate during a page load. A session is not a person or device identity.

The simplest reliable pattern is one browser per session. Launch, authenticate with a fresh sessionId, do the work, close:

function freshSession() {
  const id = Math.random().toString(36).slice(2, 10);
  return {
    username: `USER-zone-residential-sessionId-${id}`,
    password: 'PASS'
  };
}

for (const url of urls) {
  const browser = await puppeteer.launch({
    args: ['--proxy-server=http://gw.databay.co:8888']
  });
  const page = await browser.newPage();
  await page.authenticate(freshSession());
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  // ... extract ...
  await browser.close();
}

Launching a browser per task is heavy, so recent Puppeteer versions offer a lighter alternative: per-context proxies. browser.createBrowserContext() accepts a proxyServer option, giving each context its own proxy and its own cookies, cache and storage:

const browser = await puppeteer.launch();

const context = await browser.createBrowserContext({
  proxyServer: 'http://gw.databay.co:8888'
});
const page = await context.newPage();
await page.authenticate(freshSession());
await page.goto('https://httpbin.org/ip');
// ...
await context.close();

The current Puppeteer BrowserContextOptions documentation defines proxyServer. A small pool of contexts can run independent authorized test sessions inside one browser process, subject to one source-level request budget. One honest caveat: Chromium caches proxy credentials at the network layer, so swapping page.authenticate() values between pages of the same context does not always retrigger the 407 challenge on reused tunnels. If you see one session's IP bleed into another, isolate the sessions in separate contexts or separate browsers rather than fighting the credential cache. For background on when rotating beats sticky and vice versa, see static vs rotating proxies.

Proxy problems in Puppeteer surface as a handful of recognizable error strings. The four below cover the overwhelming majority of real-world failures.

A 407 means the gateway never received valid credentials. In Puppeteer the usual causes are, in order of likelihood: page.authenticate() was never called on this page (it is per-page, and a page created with browser.newPage() after the first one does not inherit it); it was called after page.goto() instead of before; or credentials were embedded in the --proxy-server URL, which Chromium strips. The fix is mechanical: call page.authenticate({ username, password }) on every page you create, before its first navigation. If credentials are definitely flowing and you still get 407, verify them outside the browser:

curl -x http://USER-zone-residential:PASS@gw.databay.co:8888 https://httpbin.org/ip

If curl succeeds and Puppeteer fails, the problem is in how the browser is wired up, not the credentials.

Chromium raises net::ERR_TUNNEL_CONNECTION_FAILED when the CONNECT tunnel to an HTTPS site cannot be established through the proxy. Check three things. First, the endpoint: it must be exactly gw.databay.co:8888; a typo in host or port fails at the tunnel stage rather than at DNS. Second, the username flags: a misspelled zone or invalid flag combination can cause the gateway to refuse the tunnel. Third, compare an approved health-check host with the intended destination. If only the destination refuses the request, stop and review its access rules rather than changing identity to bypass the response.

HTTPS traffic through the gateway travels in a CONNECT tunnel, end-to-end encrypted; the proxy does not terminate or re-sign TLS. So a certificate error inside Puppeteer is almost never caused by the proxy itself. The usual suspects are corporate middleboxes or antivirus software intercepting TLS on your own network, a genuinely misconfigured target site, or an outdated Chromium bundle. Puppeteer's escape hatch is puppeteer.launch({ acceptInsecureCerts: true }), and it is fine for hitting a staging server with a self-signed certificate, but do not run it as a blanket default for scraping: it silences exactly the warning that would tell you someone is interfering with your traffic.

Proxy routing adds variable latency, so choose a timeout from measurements of the authorized workflow rather than assuming direct-connection timing. Three adjustments can help:

page.setDefaultNavigationTimeout(60000);
await page.goto(url, { waitUntil: 'domcontentloaded' });

// Skip out-of-scope images, media and fonts
await page.setRequestInterception(true);
page.on('request', (req) => {
  ['image', 'media', 'font'].includes(req.resourceType())
    ? req.abort()
    : req.continue();
});

Waiting for domcontentloaded can avoid stalling on long-lived connections. Request interception reduces load only when the omitted resources are outside the approved scope. If a destination repeatedly times out, reduce concurrency and diagnose the source response; do not rotate identities to evade a control.

• Give each authorized workflow its own sticky sessionId and browser context when short-term continuity is required; do not rotate mid-page load.
• Treat a 403, 429, or CAPTCHA as a stop or backoff signal. Do not switch session IDs to route around the source's control.
• Request interception can omit images, media, and fonts when those resources are outside the approved data scope, reducing load and metered bandwidth.
• Use accurate client settings; a proxy changes network origin and should not be used to conceal automation.
• A page.goto('https://httpbin.org/ip') health check can confirm wiring and log the exit used for an authorized test.
• Apply one domain-level request budget across the full pool, regardless of how many exits are available.

For broader planning, use official APIs or licensed feeds first and review the source's terms, robots controls, and rate guidance.