A side project that ingests Hacker News items, top stories and nested comment trees into Postgres through a job queue, with a small HTTP API for search and stats. Runs locally via Docker Compose. Not deployed yet.
Scraping deeply nested comment trees through recursive API requests quickly exhausts rate limits, starves the single-threaded Node.js event loop, and causes database connection saturation.
Engineered an asynchronous queue-driven architecture utilizing Hono for lightweight HTTP endpoints, BullMQ with Redis for job scheduling and bounded worker concurrency, and PostgreSQL connection pooling with batched upserts.
Ingestion runs across a 50-concurrency worker with five exponential-backoff retry attempts, batched upserts, and a Postgres full-text search index over ingested items.