Appearance
Source:
docs/adr/0006-queued-merchant-price-ingestion.md. Edit it there; this page is generated at build time.
Queued merchant price ingestion behind a driver seam, where a fetch that fails is not a price that is wrong
Phase 1 shipped priced affiliate links and a read-time freshness guard, but nothing that ever writes a price except an editor in the admin. We considered three ways to close that: fetch on page render with a cache, a queued job per link driven by a scheduled command, or a nightly bulk import. Render-time fetching is out on the CLS grounds §5 of the product plan already argues, and it puts a merchant's rate limit on the critical path of a reader's page load. A nightly bulk import fails a different way: Amazon meters the Creators API per second (1 TPS for the first 30 days, earned upward to 10) and explicitly asks callers to spread requests across all 24 hours, so a single nightly sweep throttles no matter how far inside the daily allowance it sits. So: an hourly command that takes a bounded slice of what is due, and queued jobs that pace themselves.
The unit of work is a batch of up to ten links for one merchant, not a link, because one getItems call costs one transaction whether it carries one ASIN or ten — a job-per-link design would spend a tenth of the quota on a tenth of the catalog. Batching makes result-matching load-bearing: a 200 response can carry a partial errors array and does not come back in request order, so results are keyed on ASIN throughout, never on position. Writing one product's price onto another is the worst thing this pipeline could do, and it is a test rather than a comment.
The decision that shapes everything else is that "fetched, no offer" and "could not fetch" are different outcomes. A merchant that answers and has nothing to sell means the stored price is gone: it is cleared outright, not merely hidden, because the licence requires a cached price to be refreshed or removed and displayPrice() hiding it is not removal. A fetch that fails means we learned nothing at all — so a price still inside its merchant's TTL is kept and retried (a throttle or a 503 must not be able to empty the catalog), while a price already past its TTL is cleared, since that is exactly the price the licence says may no longer be held. A driver that throws instead of returning a result is treated as a failure too: a bug in a driver must not be able to erase good data. The fourth outcome, "skipped", exists so that a merchant left on the default manual price source — every merchant, until an operator changes one — is provably untouched, rather than merely happening not to be selected.
Clearing a price, though, is not a claim that the item is unbuyable, and the first cut of this conflated the two by writing in_stock = false for every "no offer". Most reasons Amazon gives us nothing to print — a minimum advertised price, a featured listing that quotes no figure — leave the product perfectly on sale; only OffersV2 omitting the listing altogether is Amazon actually reporting that nothing is buyable. Availability is therefore an allow-list in both directions: IN_STOCK/IN_STOCK_SCARCE mean true, OUT_OF_STOCK/UNAVAILABLE mean false, and everything else — PRE_ORDER, which is purchasable and priced, plus whatever Amazon adds next — stays null. This matters beyond the database because in_stock is published as a schema.org availability, where the old mapping told Google a pre-orderable product with a live price was OutOfStock. A null now omits the property rather than guessing at it: unstated is a legitimate third answer, and the only one that cannot be wrong.
The queue also needs to know when we last asked, which is not the same as when we last stored a price, and price_updated_at cannot serve as both. A cleared link's price_updated_at is null forever, so ordering the due-queue on it parked every link the pipeline could not price at the head of that queue permanently — and once there were links_per_run of them, nothing with a live price was ever refreshed again. So there is a price_checked_at, stamped on every attempt whatever the outcome, and the schedule selects and orders on that: least-recently-attempted first, which is what makes "a link crowded out of this run goes ahead of the ones that displaced it" true rather than merely hoped for. It is deliberately not the price history the next paragraph rules out — it holds no price, no previous value and no outcome, and nothing can be reconstructed from a bare timestamp. It is also not an operator setting, so it has no field in the admin.
There is deliberately no price history, anywhere. Amazon now prohibits price tracking outright, so a refresh is a plain overwrite: no history table, no previous-value column, and no price of any kind in the log lines — a log of every price we ever saw is a price history with extra steps. The OfferSnapshot value object exists only for as long as it takes to write it over the old values. If a future ticket asks for a price chart, a "lowest ever" badge or a drop alert, the answer is that this site cannot have one.
Two things the verified Creators API contract changed about the Phase 1 design. First, OffersV2 discontinued the offer summaries and its condition parameter now accepts only Any and New, so Amazon can no longer supply a lowest used price — the premise behind §4's "lowest_used_price is a first-class column" no longer holds for Amazon, though the column and the display logic stay as they are for merchants that can supply one. The obligation to show a used price is conditional on Amazon providing it, so omitting it is not a compliance gap; but "this source cannot report a used price" had to become distinguishable in the snapshot from "there is no used offer", or every refresh would silently erase a used price an editor had entered by hand. Second, a merchant's max_price_age_minutes turns out to be a licence ceiling rather than a target: Amazon's own guidance is an hourly TTL on offers, and refreshing at least hourly is also what removes the obligation to print an "as of {time}" stamp. So the effective refresh interval is the sooner of the driver's recommended interval and half the merchant's TTL, and no existing merchant's setting was rewritten to say so.
The trade-off we accept is latency and blank rows: a price is fetched by a queue that may be minutes behind, and a merchant whose feed is down eventually shows a CTA where a number used to be. The alternative is showing a figure we cannot stand behind, and every publisher examined in §5 of the plan makes the same choice.