Price Monitoring Proxies: Selection and Safe Workflow

Define the pricing decision and its authorized source before buying traffic. An official merchant API, seller export, product feed, licensed dataset, or partner report is the first choice when it supplies the identifiers and offer fields the decision needs. A proxy becomes relevant only when a permitted public-page test must vary network origin or region while the other shopper conditions remain controlled.

Google's current Merchant API product guide identifies a merchant's products by content language, feed label, and offer ID, and its example keeps availability, price, currency, condition, and identifiers as separate fields. Google also warns that accepting a product input does not mean the product is approved to appear. That is useful evidence for the shape of a normalized record, not a competitor-price feed and not permission to collect another merchant's pages.

A number is not comparable until the sellable item and offer context match. Google's current product data specification treats price and ISO 4217 currency, sale price, availability, condition, product identifiers, and market-facing data as distinct attributes. It also distinguishes a public sale price from membership pricing and requires submitted price and availability to agree with the merchant's landing and checkout surfaces. Use those distinctions as a normalization discipline even when an approved source uses different field names.

Preserve null as unknown. Store unusable, incomplete, stale, refused, and rate-limited observations alongside usable records so downstream coverage and cost calculations include what the workflow actually encountered.

Choose by required network origin and location control, then validate the smallest suitable paid order against one authorized source. Databay products use shared pools, bill transferred traffic, and support HTTP proxying, HTTPS CONNECT, and SOCKS5. A new connection may select an available exit, but a fresh or different IP is not guaranteed. Databay does not offer a free trial.

Prices, quantities, totals, and validity can change. The linked pricing pages load the current catalogue and are the purchase authority; if a current rate card is unavailable, wait or request a current quote rather than relying on this dated review. Start with Datacenter only when its origin fits, Premium Residential when its finer controls are genuinely required, and Flex when its larger minimum and coarser controls still match the test.

The example below requires Node.js 22 or newer and uses only built-in modules and the runtime's fetch implementation. Save it as price-monitor.mjs, then run node price-monitor.mjs --self-test. The self-test binds to 127.0.0.1, contacts no public source and no Databay gateway, returns a 429 followed by a valid fixture, then exercises a 403 and malformed JSON. It prints all three records so errors cannot disappear from the result set.

import { createServer } from 'node:http';
import { setTimeout as delay } from 'node:timers/promises';
import { pathToFileURL } from 'node:url';

const MAX_RETRY_AFTER_MS = 60_000;
const PARSER_VERSION = 'normalized-offer-json-v1';

export function retryAfterMs(value, nowMs = Date.now()) {
  if (!value) return null;
  const input = value.trim();
  if (/^\d+$/.test(input)) {
    const seconds = Number(input);
    return Number.isSafeInteger(seconds) && seconds <= 60 ? seconds * 1_000 : null;
  }
  const retryAt = Date.parse(input);
  if (!Number.isFinite(retryAt)) return null;
  const waitMs = Math.max(0, retryAt - nowMs);
  return waitMs <= MAX_RETRY_AFTER_MS ? waitMs : null;
}

function failure(state, attempts, kind, detail = {}) {
  return { state, offer: null, error: { kind, ...detail }, attempts };
}

async function discardBody(response) {
  if (response.body) await response.body.cancel();
}

function normalizedOffer(payload, context, retrievedAtUtc) {
  return {
    identity: {
      sourceKey: context.sourceKey,
      offerId: payload.offerId ?? null,
      productId: payload.productId ?? null,
      variant: payload.variant ?? null
    },
    seller: {
      id: payload.sellerId ?? null,
      fulfillment: payload.fulfillment ?? null
    },
    availability: payload.availability ?? null,
    price: {
      regularAmountMicros: payload.price?.amountMicros ?? null,
      saleAmountMicros: payload.salePrice?.amountMicros ?? null,
      currency: payload.price?.currencyCode ?? null,
      shippingAmountMicros: payload.shipping?.amountMicros ?? null,
      taxTreatment: payload.taxTreatment ?? 'unknown',
      membership: payload.membership ?? 'public'
    },
    market: { ...context.market },
    evidence: { retrievedAtUtc, responseStatus: 200, parserVersion: PARSER_VERSION }
  };
}

function missingFields(offer) {
  const required = [
    ['identity', 'sourceKey'], ['identity', 'offerId'], ['identity', 'productId'],
    ['seller', 'id'], ['seller', 'fulfillment'], ['price', 'regularAmountMicros'],
    ['price', 'currency'], ['market', 'requestedCountry']
  ];
  return required
    .filter(([group, field]) => offer[group][field] === null || offer[group][field] === '')
    .map(([group, field]) => group + '.' + field);
}

export async function observeOffer({
  sourceUrl,
  context,
  fetchImpl = fetch,
  sleep = delay,
  now = () => Date.now(),
  allowOne429Retry = false
}) {
  const attempts = [];
  for (let number = 1; number <= 2; number += 1) {
    const atUtc = new Date(now()).toISOString();
    let response;
    try {
      response = await fetchImpl(sourceUrl, {
        headers: { accept: 'application/json' },
        redirect: 'manual',
        signal: AbortSignal.timeout(10_000)
      });
    } catch (error) {
      attempts.push({ number, atUtc, status: null, retryAfter: null });
      const kind = error instanceof Error ? error.name : 'UnknownTransportError';
      return failure('transport_error', attempts, kind);
    }

    const retryAfter = response.headers.get('retry-after');
    attempts.push({ number, atUtc, status: response.status, retryAfter });

    if (response.status === 429) {
      const waitMs = retryAfterMs(retryAfter, now());
      await discardBody(response);
      if (allowOne429Retry && number === 1 && waitMs !== null) {
        await sleep(waitMs);
        continue;
      }
      return failure('rate_limited', attempts, 'http_429', { retryAfter });
    }

    if (response.status === 401 || response.status === 403) {
      await discardBody(response);
      return failure('refused', attempts, 'destination_refusal', { status: response.status });
    }

    if (response.status === 407) {
      await discardBody(response);
      return failure('proxy_auth_error', attempts, 'proxy_authentication', { status: 407 });
    }

    if (!response.ok) {
      await discardBody(response);
      return failure('http_error', attempts, 'http_status', { status: response.status });
    }

    let payload;
    try {
      payload = await response.json();
    } catch {
      return failure('parse_error', attempts, 'invalid_json', { status: response.status });
    }

    const offer = normalizedOffer(payload, context, atUtc);
    const missing = missingFields(offer);
    if (missing.length) {
      return {
        state: 'incomplete',
        offer,
        error: { kind: 'missing_fields', fields: missing },
        attempts
      };
    }
    return { state: 'usable', offer, error: null, attempts };
  }
  return failure('rate_limited', attempts, 'retry_budget_exhausted');
}

async function selfTest() {
  let rateHits = 0;
  const server = createServer((request, response) => {
    if (request.url === '/rate-limit' && rateHits++ === 0) {
      response.writeHead(429, { 'Retry-After': '0' });
      response.end('rate limited');
      return;
    }
    if (request.url === '/refused') {
      response.writeHead(403);
      response.end('refused');
      return;
    }
    if (request.url === '/malformed') {
      response.writeHead(200, { 'Content-Type': 'application/json' });
      response.end('{');
      return;
    }
    response.writeHead(200, { 'Content-Type': 'application/json' });
    response.end(JSON.stringify({
      offerId: 'sku12345', productId: 'gtin-0001', variant: 'blue-m',
      sellerId: 'authorized-merchant', fulfillment: 'merchant', availability: 'IN_STOCK',
      price: { amountMicros: '15990000', currencyCode: 'USD' },
      salePrice: null, shipping: null, taxTreatment: 'recorded-separately', membership: 'public'
    }));
  });
  await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
  const address = server.address();
  const base = 'http://127.0.0.1:' + address.port;
  const context = {
    sourceKey: 'local-fixture',
    market: {
      requestedCountry: 'US', contentLanguage: 'en', feedLabel: 'US',
      deliveryAssumption: 'none', accountState: 'signed-out'
    }
  };
  try {
    return [
      await observeOffer({ sourceUrl: base + '/rate-limit', context, allowOne429Retry: true }),
      await observeOffer({ sourceUrl: base + '/refused', context }),
      await observeOffer({ sourceUrl: base + '/malformed', context })
    ];
  } finally {
    await new Promise((resolve, reject) =>
      server.close((error) => error ? reject(error) : resolve())
    );
  }
}

async function main() {
  if (process.argv.includes('--self-test')) return selfTest();
  const sourceUrl = process.env.PRICE_SOURCE_URL;
  if (!sourceUrl) throw new Error('Set PRICE_SOURCE_URL to an approved JSON endpoint.');
  return observeOffer({
    sourceUrl,
    context: {
      sourceKey: process.env.PRICE_SOURCE_KEY ?? 'approved-source',
      market: {
        requestedCountry: process.env.PRICE_MARKET ?? null,
        contentLanguage: process.env.PRICE_LANGUAGE ?? null,
        feedLabel: process.env.PRICE_FEED_LABEL ?? null,
        deliveryAssumption: process.env.PRICE_DELIVERY_ASSUMPTION ?? null,
        accountState: process.env.PRICE_ACCOUNT_STATE ?? 'documented-separately'
      }
    },
    allowOne429Retry: process.env.ALLOW_ONE_429_RETRY === 'true'
  });
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
  console.log(JSON.stringify(await main(), null, 2));
}

The fixture's zero-second Retry-After keeps the self-test fast; production values can be either delay-seconds or an HTTP date. The example accepts only a valid delay no longer than 60 seconds and performs at most one follow-up when ALLOW_ONE_429_RETRY=true. Without that explicit opt-in, or when the header is absent, invalid, or longer, it retains a rate-limited record and stops. Choose the ceiling and retry permission with the source owner; 60 seconds is an example guardrail, not a claim about any marketplace.

RFC 6585 section 4 defines 429 as too many requests in a given time and says the response may include Retry-After. It deliberately does not define whether counting is per resource, server, credential, cookie, or another scope. Therefore one source-wide scheduler (not each proxy, worker, or session) must own the permitted request budget.

RFC 9110 section 10.2.3 permits Retry-After as either an HTTP date or non-negative delay-seconds. Parse both forms, cap automated waiting, retain the original response, and use at most the preapproved retry budget. Do not rotate an exit, alter headers, swap accounts, or increase concurrency after 429. A 401, 403, CAPTCHA, login wall, or explicit objection is a stop-and-review outcome rather than a retry candidate.

Run the smallest representative pilot that can reject the design: one authorized source, important product and offer shapes, only the markets needed for the decision, a frozen parser version, and one network configuration at a time. Count planned observations, all attempts, bytes, status classes, incomplete offers, location fulfillment, review time, and retained errors. Report latency percentiles with failures separately; do not calculate performance only across successful responses.

Expand only if the chosen network supplies decision value that the approved direct source does not, the workflow meets its predeclared quality and cost gates, and stop behavior is verified. No product class guarantees access, extraction, completeness, a unique exit, or a particular success rate.

This page owns the commercial decision for price monitoring proxies: whether a network-origin variable is needed, which Databay product and smallest paid path fit it, and how to test that choice without dropping errors. The ecommerce proxy guide covers broader catalog QA, localization, storefront testing, seller research, and repricing safeguards. The web-scraping guide covers source approval, request budgets, provenance, and collection design across data types.

All three workflows share a boundary: public display is not blanket permission. RFC 9309 states that robots rules are not access authorization. Review contracts, API terms, robots instructions where applicable, rate guidance, privacy, intellectual-property, competition, retention, and reuse requirements, and keep every request within Databay's Acceptable Use Policy. Stop when authorization, source behavior, or controls change.