Job Scrapers Break at the Seams. I Built One Around the Seams.
Scrapers usually fail long before anti bot systems kill them. They fail when parsing logic, storage, retries, and configuration get tangled into one script nobody wants to touch. That was the problem I cared about. Not scraping more jobs. Building a scraper that did not collapse under its own shortcuts.
This project became a modular job ingestion system built around that idea. Different scraper strategies sit behind one interface. Site config lives outside runtime settings. Deduplication happens in PostgreSQL instead of brittle in memory checks. Small decisions, but they change how the system behaves.
Pushing Reliability Into the Database
The most important decision was treating duplicate handling as a database problem, not application logic. Each job URL has a uniqueness constraint and inserts use conflict handling. If a scraper runs again and sees the same listing, the database ignores it.
I liked this because it removed a whole category of state bugs. No manual duplicate tracking. No weird edge cases when scheduled runs overlap. The database already solves that.
Building Around Replaceable Parts
I split scraping into strategies. A simple scraper for straightforward sites. A stealth scraper for sites with heavier defenses. Both follow the same fetch, parse, run structure, which made the system feel extensible early.
A separate choice I liked was pushing site definitions into YAML instead of hardcoding behavior. Adding a source became more about configuration than editing scraper logic.
- Playwright handled rendering and navigation
- Typer made the CLI useful enough to run, inspect, and export data
- APScheduler let the whole thing behave more like a pipeline than a script
None of those choices are exotic. Together they made the project feel closer to infrastructure than automation glue.
What Was Incomplete
The biggest limitation is also the most honest one. The parsers are still placeholders returning example job data. The architecture is real. The extraction layer is not finished.
That matters. It means this project is more foundation than fully realized scraper network. If I revisited it, I would prioritize real parsers, structured validation before writes, and stronger logging before adding more features.
I would also add rate limiting much earlier. I underweighted operational behavior compared to architecture.
What It Taught Me
The non obvious lesson was that many engineering decisions look unnecessary when a project is small. Configuration separation feels excessive at fifty lines. Upserts feel overthought when one script works. Then scale appears and those decisions stop looking optional.
I also learned restraint. Some parts were intentionally basic because I wanted the interfaces right before chasing complexity. That was the correct tradeoff.
Good systems often start as boundaries, not features.