prices · ShopBench · arthgupta.dev

prices — system design

An agent-first price reference that aims to cover every physical good sold online. For each leaf category: the minimum spec that does the job, the cheapest product meeting it by unit price, and what each pricier option's premium buys. Static JSON is the product; the HTML table is a thin render; a small API turns lookup misses into crawl jobs. Rendered at /about; source in price-reference/DESIGN.md.

Why

Textbook economics models consumers as rational self-optimizers. People aren't, mostly because comparing hundreds of listings across retailers, sizes and pack counts is expensive. Agents can be, if they have the information. This index is that information, and ShopBench measures whether having it moves an agent's purchases toward the optimum.

People shop on heuristics because exact comparison costs more attention than a purchase is worth. Behavioral economics catalogs those heuristics, and retailers price against them: anchoring on a "was" price, decoy tiers that make the middle option look reasonable, charm pricing ($4.99), multi-buy deals that aren't cheaper per unit, and smaller packages at the same price. An agent doesn't need the shortcuts. It can afford to compare every listing exactly, so the biases stop working once it has the data. This index gives it that data in a form that corrects for each one. Every listing reduces to a price per unit against a stated minimum spec, which removes the anchor, the decoy and the bundle, and the premium column says what any extra money actually buys.

Pipeline

stores (registry: hand-written + auto-probed)      → which hosts, how to find products, how to parse
discover  (listing pages · product sitemaps · Amazon search per category · Shopify /products.json)   → urls
fetch     (per-host pacing ≥2.5 s + jitter, robots.txt, daily cap, cool-off on blocks, one thread per retailer)
parse     (Walmart __NEXT_DATA__ · Amazon HTML · Target __TGT_DATA__ · generic schema.org Product JSON-LD · Shopify JSON)
          → raw_pages (gzipped HTML, kept forever: a parser fix is a reparse, not a re-crawl) → observations
normalize (LLM, idempotent passes):
          categorize → leaf category (+ root department); "other" only for non-products
          merge      → fold thin / duplicate leaves
          taxonomy   → map each leaf to Shopify Standard Product Taxonomy, GS1 GPC brick, Google Product Taxonomy
          schemas    → per-leaf comparison schema: unit, fields (named after the standard attributes), minimum spec, premium dimensions
          extract    → per listing: brand, product line, variant, size in the category unit, specs, meets-minimum, tier, premium note
          link       → canonical items: same GTIN (with a title sanity check) or same brand+line+variant+size
          labels     → per-product Shopify category + misfiled flag; per-item standard attribute values
build     → data/site: index.html (table), table.json, index.json, categories/*.json|html, items/*.json|html, gtin/*.json, llms.txt, sitemap.xml
serve     → FastAPI: static mount + GET /api/search (pure) · POST /api/want · GET /api/want/:id · /api/docs
wants     → a search miss (or POST) becomes a want; the worker finds candidates (Amazon search, sitemap slug index),
            fetches, normalizes just those, republishes — within ~10 minutes

Every pricer run tees its output to data/logs/pricer.log. Three LaunchAgents keep it running: serve (KeepAlive), crawl (continuous loop: discover → Shopify ingest → parallel fetch → normalize → build, 10 min pause), worker (every 10 min, wants only).

Sources

Scraping only; no retailer APIs or affiliate feeds. robots.txt is honoured per URL; each host gets at most one request every 2.5–6 seconds with jitter; three consecutive blocks pause a host for 20 minutes; a daily cap per host is the safety net. Prices are sticker prices as shown on the product page: no shipping, tax or membership. Each offer carries the retailer URL, observed_at, seller, first-party flag, stock, and the page's shipping hint when it states one.

The big three have hand-written parsers (Walmart ships UPC + price + unit price in __NEXT_DATA__; Amazon has no GTIN; Target has UPC but no server-side price). Everything else is generic: schema.org Product JSON-LD, or a Shopify catalog via /products.json. New stores are qualified automatically (pricer stores probe): Shopify? → sitemap → one product page → does JSON-LD carry a price? The product-URL pattern is derived from the sitemap.

Data model (PostgreSQL 16, embedded, data/pg)

Storage moved from SQLite to PostgreSQL once a dozen crawler and normalizer processes were writing concurrently: SQLite allows one writer at a time and its lock retry is not fair, so writers starved for hours. Postgres gives real concurrent writers, tsvector full-text search and window functions for the derived item_best table.

A category schema is {unit, unit_note, fields[{key,label,type,unit,values,why_it_matters}], minimum_spec{description, rules}, premium_dimensions}. unit_price_norm = price / size_value in that unit. A category's baseline is the cheapest item meeting the minimum spec; every other item's premium is its unit-price % over the baseline plus the spec fields that differ along the premium dimensions.

Published JSON

• index.json — categories with the cheapest min-spec item, counts, taxonomy ids, aggregate stats
• categories/<slug>.json — schema, minimum spec, taxonomy (Shopify id + attributes, Google, GPC), items ranked by unit price with offers and premium diffs, unranked items with the reason (size_missing / no_price)
• items/<id>.json — one item: offers per retailer (gtin, price, unit_price, in_stock, seller, first_party, shipping, observed_at, age), specs, standard attributes, history
• gtin/<gtin>.json — GTIN → item · table.json — the flat table · llms.txt — what to call first · sitemap.xml

API

GET /api/search is a pure read: q (a name or a spec: "oily skin moisturizer spf"), gtin, category, shopify_category, google_id, gpc_brick, max_price, max_unit_price, in_stock, limit. Spec queries pick the category from the words that name one and test the rest against item spec keys/values; words nothing in the category can speak to come back as ignored. A miss returns not_indexed with the exact POST /api/want to call and an ETA; GET never queues work. OpenAPI at /api/openapi.json.

Taxonomy

Three standards, cross-referenced and served per category. Shopify Standard Product Taxonomy (~14.6k categories with attributes and values) is the primary identity and the source of standard attribute names; Google Product Taxonomy (~5.6k) for discoverability (Shopify ships the mapping); GS1 GPC (~5.3k bricks) as the supply-chain cross-reference built around GTINs. Coverage (share of taxonomy leaves with at least one product) is reported in index.json.

Cost

The whole LLM pipeline runs on a cheap model: roughly $0.35 per 1,000 products through categorize → schema → extract, about $0.05 per 1,000 for per-product taxonomy labels. Crawling is free apart from politeness; the constraint on growth is per-host pacing, so breadth comes from more hosts, not faster hosts.

Known limits

Delivered cost needs a cart session, so prices are sticker prices. Target prices are not server-rendered. Minimum specs are written by an LLM from listings and can be debatable. A marketplace listing can reuse a UPC; GTIN links require a shared title word. Amazon rate-limits aggressively; the profile that passes changes.