Refresh, Storage, and Snapshots

Independent refresh jobs

Market snapshots, regular holder collection, wallet P/L and watch monitoring run as independent jobs so an upstream wallet timeout cannot stop price refreshes or alerts. After a successful market scrape, a bounded catch-up pass collects missing holders from the new snapshot before bootstrap snapshots are rebuilt.

Cadence

Job	Current cadence	Purpose
Market scrape	Every 60 minutes	Refresh active/open market snapshot and tags
Holder positions	Every 60 minutes	Refresh the $10k volume / $10k liquidity universe, including still-open markets past kickoff
Missing-holder catch-up	After each successful scrape	Up to 200 unsampled eligible markets, oldest attempts first
Wallet P/L	Every 15 minutes	Bounded batches; reuse observations for 24 hours
Watch monitor	Every 10 minutes	Refresh saved-market monitoring

Opening a market with no stored holders can trigger one bounded recovery attempt for a user with analysis access. Recovery does not invent P/L values or change endpoint permissions.

Snapshot replacement rules

The scraper drops and rebuilds:

active_market_outcomes
market_tags

This means each scrape creates a clean active snapshot instead of incrementally updating old rows.

Storage tables

Table	Role
`active_market_outcomes`	Main outcome-level market snapshot
`market_tags`	Tag rows attached to market ids
`holders`	Sampled holder rows keyed by condition id
`holder_collection_attempts`	Last holder attempt time, status and count per condition
`wallets_stats`	Wallet-level PnL and alias data
`market_smart_money_stats`	Background `smart_money_win_rate` storage and timestamp
`request_logs`	API request timing logs in the metrics database

How holders are refreshed

For each distinct condition_id in the active snapshot, PolyLab fetches holders, deletes prior holder rows for that condition, and inserts the new sampled rows with a fresh snapshot_at.

How wallet stats are refreshed

PolyLab extracts unique wallet addresses from sampled holder rows, fetches wallet PnL, and upserts into wallets_stats.

What survives between scrapes

The smart-money-related tables are not dropped on every market scrape. They are updated by the smart-money job and can therefore be newer or older than the market snapshot depending on job timing.

Why this architecture was chosen

The current design keeps the read path straightforward:

the app reads from a compact snapshot table
tag logic stays simple
holder and wallet enrichment can be refreshed on a slower schedule

The cost is that PolyLab is a snapshot system, not a live state mirror.