Investigate sudden increase in Zyte usage #192

New Issue

2026-07-04T01:03:35Z

gambit-admin commented

2026-07-04 01:03:35 +00:00

Reported by @kris on 2026-07-04 via chat. A sudden increase in Zyte usage has been observed, requiring investigation to identify the root cause and prevent potential cost or performance issues.

Reported via Mattermost bot by Gambit Agent

Reported by @kris on 2026-07-04 via chat. A sudden increase in Zyte usage has been observed, requiring investigation to identify the root cause and prevent potential cost or performance issues. --- Reported via Mattermost bot by Gambit Agent

gambit-admin added the api performance claimed:kris labels 2026-07-04 01:04:25 +00:00

gambit-admin commented

2026-07-04 01:09:16 +00:00

Root cause found (design box logs, last 6 min since container restart):

eBay sold-listing scraping is currently 100% broken: every Zyte http-mode extract call returns unparseable HTML, forcing fallback to the pricier browser-render mode on literally every request (200/200 in logs). Even the browser-rendered HTML then parses 0 listings (0 for 0 across 100 browser-mode calls sampled). The last-resort direct scrape also 403s every time. Net: every price lookup burns 2x Zyte calls and returns nothing.
This gets amplified by scheduled_refresh_loop() in backend/app/services/ebay_cache_policy.py — an hourly background job (enabled by default, EBAY_CACHE_REFRESH_ENABLED=true here) that refreshes up to 100 due comps queries/hour at concurrency 8 across the whole active-inventory catalog, fully automated regardless of who is testing. Matches the log pattern exactly: ~95-100 distinct card queries burned through in a ~6min burst.
Design and production use separate Zyte API keys (verified via Railway), so this isn't a shared-quota drain — but production runs the identical code/scheduler, so it likely has the same 100% failure rate burning its own Zyte budget for zero return.
Not yet determined: whether eBay changed sold-listings page markup (breaking the regex parser) or is now blocking Zyte's IPs/UA with a captcha/block page that matches neither the s-card nor legacy s-item layout.

Root cause found (design box logs, last 6 min since container restart): - eBay sold-listing scraping is currently 100% broken: every Zyte http-mode extract call returns unparseable HTML, forcing fallback to the pricier browser-render mode on literally every request (200/200 in logs). Even the browser-rendered HTML then parses 0 listings (0 for 0 across 100 browser-mode calls sampled). The last-resort direct scrape also 403s every time. Net: every price lookup burns 2x Zyte calls and returns nothing. - This gets amplified by scheduled_refresh_loop() in backend/app/services/ebay_cache_policy.py — an hourly background job (enabled by default, EBAY_CACHE_REFRESH_ENABLED=true here) that refreshes up to 100 due comps queries/hour at concurrency 8 across the whole active-inventory catalog, fully automated regardless of who is testing. Matches the log pattern exactly: ~95-100 distinct card queries burned through in a ~6min burst. - Design and production use separate Zyte API keys (verified via Railway), so this isn't a shared-quota drain — but production runs the identical code/scheduler, so it likely has the same 100% failure rate burning its own Zyte budget for zero return. - Not yet determined: whether eBay changed sold-listings page markup (breaking the regex parser) or is now blocking Zyte's IPs/UA with a captcha/block page that matches neither the s-card nor legacy s-item layout.

gambit-admin commented

2026-07-04 03:12:36 +00:00

Zyte disabled across the board (both design and Railway production) per explicit request — this is a mitigation, not a fix for the underlying eBay-parser breakage described above.

Design (.env, recreated recontcg_design_api container):

EBAY_LIVE_SCRAPE_ENABLED: true -> false
EBAY_CACHE_REFRESH_ENABLED: true -> false (stops the hourly automated sweep)
EBAY_SCRAPER_PROVIDER: zyte -> direct

Railway production (runrecon-back, redeployed):

EBAY_LIVE_SCRAPE_ENABLED: true -> false
EBAY_SCRAPER_PROVIDER: zyte -> direct
TCGPLAYER_ZYTE_API_KEY: cleared

Why EBAY_SCRAPER_PROVIDER and TCGPLAYER_ZYTE_API_KEY needed to change too, not just the live-scrape flag: tcgplayer_client.py's _zyte_api_key() has its own fallback — if no dedicated TCGPLAYER_ZYTE_API_KEY is set, it silently reuses EBAY_SCRAPER_API_KEY whenever EBAY_SCRAPER_PROVIDER=='zyte'. That path is completely independent of EBAY_LIVE_SCRAPE_ENABLED and is on the hot path of ordinary TCGplayer price lookups (card views, catalog, inventory) in production, since production has no TCGPLAYER_PROXY_URL configured to take priority. Flipping only the live-scrape flag would have left this path live. Verified via Railway logs that runrecon-back is Online and serving 200s post-redeploy.

Root cause of the original spike (100% eBay parse failure forcing every request into the pricier browser-render fallback) is still open — didn't spend another Zyte call chasing it further since usage is now fully stopped.

Zyte disabled across the board (both design and Railway production) per explicit request — this is a mitigation, not a fix for the underlying eBay-parser breakage described above. Design (.env, recreated recontcg_design_api container): - EBAY_LIVE_SCRAPE_ENABLED: true -> false - EBAY_CACHE_REFRESH_ENABLED: true -> false (stops the hourly automated sweep) - EBAY_SCRAPER_PROVIDER: zyte -> direct Railway production (runrecon-back, redeployed): - EBAY_LIVE_SCRAPE_ENABLED: true -> false - EBAY_SCRAPER_PROVIDER: zyte -> direct - TCGPLAYER_ZYTE_API_KEY: cleared Why EBAY_SCRAPER_PROVIDER and TCGPLAYER_ZYTE_API_KEY needed to change too, not just the live-scrape flag: tcgplayer_client.py's _zyte_api_key() has its own fallback — if no dedicated TCGPLAYER_ZYTE_API_KEY is set, it silently reuses EBAY_SCRAPER_API_KEY whenever EBAY_SCRAPER_PROVIDER=='zyte'. That path is completely independent of EBAY_LIVE_SCRAPE_ENABLED and is on the hot path of ordinary TCGplayer price lookups (card views, catalog, inventory) in production, since production has no TCGPLAYER_PROXY_URL configured to take priority. Flipping only the live-scrape flag would have left this path live. Verified via Railway logs that runrecon-back is Online and serving 200s post-redeploy. Root cause of the original spike (100% eBay parse failure forcing every request into the pricier browser-render fallback) is still open — didn't spend another Zyte call chasing it further since usage is now fully stopped.

gambit-admin commented

2026-07-05 22:28:09 +00:00

Verified Zyte mitigation across all environments. The serving checkout (recontcg-design) already had Zyte disabled, but Kris, Josh, and Rudy personal dev clones still had EBAY_SCRAPER_PROVIDER=zyte and EBAY_LIVE_SCRAPE_ENABLED/CACHE_REFRESH_ENABLED=true — a ship from any personal clone would have burned Zyte credits. Updated all three .env files to match the serving checkout: provider=direct, live_scrape=false, cache_refresh=false. Railway production was already mitigated via redeploy.

gambit-admin closed this issue

2026-07-05 22:28:14 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gambit/gambit#192