Price monitoring proxies for comparable offers

Define the pricing decision and its authorized source before buying traffic. An official merchant API, seller export, product feed, licensed dataset, or partner report is the first choice when it supplies the identifiers and offer fields the decision needs. A proxy becomes relevant only when a permitted public-page test must vary network origin or region while the other shopper conditions remain controlled.

Google's current Merchant API product guide identifies a merchant's products by content language, feed label, and offer ID, and its example keeps availability, price, currency, condition, and identifiers as separate fields. Google also warns that accepting a product input does not mean the product is approved to appear. That is useful evidence for the shape of a normalized record, not a competitor-price feed and not permission to collect another merchant's pages.

A number is not comparable until the sellable item and offer context match. Google's current product data specification treats price and ISO 4217 currency, sale price, availability, condition, product identifiers, and market-facing data as distinct attributes. It also distinguishes a public sale price from membership pricing and requires submitted price and availability to agree with the merchant's landing and checkout surfaces. Use those distinctions as a normalization discipline even when an approved source uses different field names.

Preserve null as unknown. Store unusable, incomplete, stale, refused, and rate-limited observations alongside usable records so downstream coverage and cost calculations include what the workflow actually encountered.

Choose by required network origin and location control, then validate the smallest suitable paid order against one authorized source. Databay products use shared pools, bill transferred traffic, and support HTTP proxying, HTTPS CONNECT, and SOCKS5. A new connection may select an available exit, but a fresh or different IP is not guaranteed. Databay does not offer a free trial.

Prices, quantities, totals, and validity can change. The linked pricing pages load the current catalogue and are the purchase authority; if a current rate card is unavailable, wait or request a current quote rather than relying on this dated review. Start with Datacenter only when its origin fits, Premium Residential when its finer controls are genuinely required, and Flex when its larger minimum and coarser controls still match the test.

Write the hypothesis as a controlled comparison: hold product, seller, variant, fulfillment, delivery assumption, language, account, membership, consent, device, parser, and time window fixed; change only the requested network region. Verify the observed exit before interpreting a difference. A country-targeted connection is evidence from one routed request, not proof of a shopper's residence, tax nexus, delivery eligibility, GPS location, or every price in that market.

Use a rotating connection for independent observations only when the sampling plan calls for it. Use one sticky session for a short, permitted multi-request flow that requires route continuity. Keep the same session across an approved 429 follow-up; changing exits in response to a quota or refusal would corrupt the experiment and can turn rate handling into evasion. Connection reuse can retain an exit, and a requested sticky duration is a maximum request rather than a continuity guarantee.

1. Confirm current country availability for the chosen product rather than inferring it from a location name.
2. Create the full product username with the proxy configuration generator.
3. Verify one benign route with the generated client setup before contacting the approved source.
4. Keep source scheduling and rate budgets global across workers, sessions, and exits.
5. Record requested and observed region separately; quarantine an unfulfilled route.

The example below requires Node.js 22 or newer and uses only built-in modules and the runtime's fetch implementation. Save it as price-monitor.mjs, then run node price-monitor.mjs --self-test. The self-test binds to 127.0.0.1, contacts no public source and no Databay gateway, returns a 429 followed by a valid fixture, then exercises a 403 and malformed JSON. It prints all three records so errors cannot disappear from the result set.

import { createServer } from 'node:http';
import { setTimeout as delay } from 'node:timers/promises';
import { pathToFileURL } from 'node:url';

const MAX_RETRY_AFTER_MS = 60_000;
const PARSER_VERSION = 'normalized-offer-json-v1';

export function retryAfterMs(value, nowMs = Date.now()) {
  if (!value) return null;
  const input = value.trim();
  if (/^\d+$/.test(input)) {
    const seconds = Number(input);
    return Number.isSafeInteger(seconds) && seconds <= 60 ? seconds * 1_000 : null;
  }
  const retryAt = Date.parse(input);
  if (!Number.isFinite(retryAt)) return null;
  const waitMs = Math.max(0, retryAt - nowMs);
  return waitMs <= MAX_RETRY_AFTER_MS ? waitMs : null;
}

function failure(state, attempts, kind, detail = {}) {
  return { state, offer: null, error: { kind, ...detail }, attempts };
}

async function discardBody(response) {
  if (response.body) await response.body.cancel();
}

function normalizedOffer(payload, context, retrievedAtUtc) {
  return {
    identity: {
      sourceKey: context.sourceKey,
      offerId: payload.offerId ?? null,
      productId: payload.productId ?? null,
      variant: payload.variant ?? null
    },
    seller: {
      id: payload.sellerId ?? null,
      fulfillment: payload.fulfillment ?? null
    },
    availability: payload.availability ?? null,
    price: {
      regularAmountMicros: payload.price?.amountMicros ?? null,
      saleAmountMicros: payload.salePrice?.amountMicros ?? null,
      currency: payload.price?.currencyCode ?? null,
      shippingAmountMicros: payload.shipping?.amountMicros ?? null,
      taxTreatment: payload.taxTreatment ?? 'unknown',
      membership: payload.membership ?? 'public'
    },
    market: { ...context.market },
    evidence: { retrievedAtUtc, responseStatus: 200, parserVersion: PARSER_VERSION }
  };
}

function missingFields(offer) {
  const required = [
    ['identity', 'sourceKey'], ['identity', 'offerId'], ['identity', 'productId'],
    ['seller', 'id'], ['seller', 'fulfillment'], ['price', 'regularAmountMicros'],
    ['price', 'currency'], ['market', 'requestedCountry']
  ];
  return required
    .filter(([group, field]) => offer[group][field] === null || offer[group][field] === '')
    .map(([group, field]) => group + '.' + field);
}

export async function observeOffer({
  sourceUrl,
  context,
  fetchImpl = fetch,
  sleep = delay,
  now = () => Date.now(),
  allowOne429Retry = false
}) {
  const attempts = [];
  for (let number = 1; number <= 2; number += 1) {
    const atUtc = new Date(now()).toISOString();
    let response;
    try {
      response = await fetchImpl(sourceUrl, {
        headers: { accept: 'application/json' },
        redirect: 'manual',
        signal: AbortSignal.timeout(10_000)
      });
    } catch (error) {
      attempts.push({ number, atUtc, status: null, retryAfter: null });
      const kind = error instanceof Error ? error.name : 'UnknownTransportError';
      return failure('transport_error', attempts, kind);
    }

    const retryAfter = response.headers.get('retry-after');
    attempts.push({ number, atUtc, status: response.status, retryAfter });

    if (response.status === 429) {
      const waitMs = retryAfterMs(retryAfter, now());
      await discardBody(response);
      if (allowOne429Retry && number === 1 && waitMs !== null) {
        await sleep(waitMs);
        continue;
      }
      return failure('rate_limited', attempts, 'http_429', { retryAfter });
    }

    if (response.status === 401 || response.status === 403) {
      await discardBody(response);
      return failure('refused', attempts, 'destination_refusal', { status: response.status });
    }

    if (response.status === 407) {
      await discardBody(response);
      return failure('proxy_auth_error', attempts, 'proxy_authentication', { status: 407 });
    }

    if (!response.ok) {
      await discardBody(response);
      return failure('http_error', attempts, 'http_status', { status: response.status });
    }

    let payload;
    try {
      payload = await response.json();
    } catch {
      return failure('parse_error', attempts, 'invalid_json', { status: response.status });
    }

    const offer = normalizedOffer(payload, context, atUtc);
    const missing = missingFields(offer);
    if (missing.length) {
      return {
        state: 'incomplete',
        offer,
        error: { kind: 'missing_fields', fields: missing },
        attempts
      };
    }
    return { state: 'usable', offer, error: null, attempts };
  }
  return failure('rate_limited', attempts, 'retry_budget_exhausted');
}

async function selfTest() {
  let rateHits = 0;
  const server = createServer((request, response) => {
    if (request.url === '/rate-limit' && rateHits++ === 0) {
      response.writeHead(429, { 'Retry-After': '0' });
      response.end('rate limited');
      return;
    }
    if (request.url === '/refused') {
      response.writeHead(403);
      response.end('refused');
      return;
    }
    if (request.url === '/malformed') {
      response.writeHead(200, { 'Content-Type': 'application/json' });
      response.end('{');
      return;
    }
    response.writeHead(200, { 'Content-Type': 'application/json' });
    response.end(JSON.stringify({
      offerId: 'sku12345', productId: 'gtin-0001', variant: 'blue-m',
      sellerId: 'authorized-merchant', fulfillment: 'merchant', availability: 'IN_STOCK',
      price: { amountMicros: '15990000', currencyCode: 'USD' },
      salePrice: null, shipping: null, taxTreatment: 'recorded-separately', membership: 'public'
    }));
  });
  await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
  const address = server.address();
  const base = 'http://127.0.0.1:' + address.port;
  const context = {
    sourceKey: 'local-fixture',
    market: {
      requestedCountry: 'US', contentLanguage: 'en', feedLabel: 'US',
      deliveryAssumption: 'none', accountState: 'signed-out'
    }
  };
  try {
    return [
      await observeOffer({ sourceUrl: base + '/rate-limit', context, allowOne429Retry: true }),
      await observeOffer({ sourceUrl: base + '/refused', context }),
      await observeOffer({ sourceUrl: base + '/malformed', context })
    ];
  } finally {
    await new Promise((resolve, reject) =>
      server.close((error) => error ? reject(error) : resolve())
    );
  }
}

async function main() {
  if (process.argv.includes('--self-test')) return selfTest();
  const sourceUrl = process.env.PRICE_SOURCE_URL;
  if (!sourceUrl) throw new Error('Set PRICE_SOURCE_URL to an approved JSON endpoint.');
  return observeOffer({
    sourceUrl,
    context: {
      sourceKey: process.env.PRICE_SOURCE_KEY ?? 'approved-source',
      market: {
        requestedCountry: process.env.PRICE_MARKET ?? null,
        contentLanguage: process.env.PRICE_LANGUAGE ?? null,
        feedLabel: process.env.PRICE_FEED_LABEL ?? null,
        deliveryAssumption: process.env.PRICE_DELIVERY_ASSUMPTION ?? null,
        accountState: process.env.PRICE_ACCOUNT_STATE ?? 'documented-separately'
      }
    },
    allowOne429Retry: process.env.ALLOW_ONE_429_RETRY === 'true'
  });
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
  console.log(JSON.stringify(await main(), null, 2));
}

The fixture's zero-second Retry-After keeps the self-test fast; production values can be either delay-seconds or an HTTP date. The example accepts only a valid delay no longer than 60 seconds and performs at most one follow-up when ALLOW_ONE_429_RETRY=true. Without that explicit opt-in, or when the header is absent, invalid, or longer, it retains a rate-limited record and stops. Choose the ceiling and retry permission with the source owner; 60 seconds is an example guardrail, not a claim about any marketplace.

RFC 6585 section 4 defines 429 as too many requests in a given time and says the response may include Retry-After. It deliberately does not define whether counting is per resource, server, credential, cookie, or another scope. Therefore one source-wide scheduler (not each proxy, worker, or session) must own the permitted request budget.

RFC 9110 section 10.2.3 permits Retry-After as either an HTTP date or non-negative delay-seconds. Parse both forms, cap automated waiting, retain the original response, and use at most the preapproved retry budget. Do not rotate an exit, alter headers, swap accounts, or increase concurrency after 429. A 401, 403, CAPTCHA, login wall, or explicit objection is a stop-and-review outcome rather than a retry candidate.

The loopback run validates the request and classification logic; it does not test a marketplace, extraction parser, Databay credential, route availability, or destination acceptance. Before a real pilot, map the approved source into the normalized JSON contract, add a source-wide scheduler and cache, then configure one client through the Databay proxy configuration generator. Keep credentials in a secret manager and never print the proxy URL or authorization header.

Use the Python Requests setup or cURL setup and diagnostic model to prove gateway authentication against a benign endpoint you control. Then inject that reviewed proxy-aware client into observeOffer as fetchImpl; do not assume the direct self-test is routed. Verify requested versus observed exit once, run the authorized offer endpoint, and preserve the same response classification. A successful diagnostic proves only that one connection worked; it does not guarantee that the source will return, expose, or permit the desired offer.

For a public HTML page, an authorized parser must be versioned and tested separately against retained, permitted fixtures. The example intentionally consumes a normalized JSON shape so it cannot be mistaken for a universal extractor. Layout changes, personalization, stock, promotions, consent state, and experiments can still make two otherwise valid observations incomparable.

Run the smallest representative pilot that can reject the design: one authorized source, important product and offer shapes, only the markets needed for the decision, a frozen parser version, and one network configuration at a time. Count planned observations, all attempts, bytes, status classes, incomplete offers, location fulfillment, review time, and retained errors. Report latency percentiles with failures separately; do not calculate performance only across successful responses.

Expand only if the chosen network supplies decision value that the approved direct source does not, the workflow meets its predeclared quality and cost gates, and stop behavior is verified. No product class guarantees access, extraction, completeness, a unique exit, or a particular success rate.

This page owns the commercial decision for price monitoring proxies: whether a network-origin variable is needed, which Databay product and smallest paid path fit it, and how to test that choice without dropping errors. The ecommerce proxy guide covers broader catalog QA, localization, storefront testing, seller research, and repricing safeguards. The web-scraping guide covers source approval, request budgets, provenance, and collection design across data types.

All three workflows share a boundary: public display is not blanket permission. RFC 9309 states that robots rules are not access authorization. Review contracts, API terms, robots instructions where applicable, rate guidance, privacy, intellectual-property, competition, retention, and reuse requirements, and keep every request within Databay's Acceptable Use Policy. Stop when authorization, source behavior, or controls change.