Work · a professional network and jobs platform
Profiles, skills and real-time messaging on one search index
The problem
A professional network needs three things that pull against each other: profiles rich enough to be worth searching, search fast enough to feel instant, and messaging immediate enough that people come back. Built naively, each one degrades the others — richer profiles slow the search, and polling for messages hammers the same database the search runs on.
The approach
Profiles model people and companies separately but share the structures worth searching on — expertise, skills, certificates, languages, location. Search runs against Elasticsearch through a document layer rather than against the relational tables, so adding a field to a profile does not make the search slower. Messaging runs over its own Redis-backed real-time channel instead of polling, and the notification and follow graphs are written asynchronously through Celery.
What was technically hard
Keeping the search index honest. Once search stops reading the database, every profile edit, new certificate and changed company becomes an index update that can fail on its own, and a member whose profile is invisible to search has effectively been deleted from the product. The indexing had to be idempotent and re-runnable from the source of truth, because the recovery path — rebuild it — is the one that gets used.
One decision we made, and why
We kept the relational schema as the single source of truth and treated the index as derived, disposable state. It costs a rebuild path we have to maintain, but it means no data lives only in Elasticsearch, and a corrupted index is an inconvenience rather than a loss.
More work
- Reconciliation that reports breaks the day they happen →
Time to detect a reconciliation break fell from weeks to one day.
- Machine data that arrives twice, late, or not at all →
Shift totals now reconcile to the plant's own counts.
- Population statistics for any polygon a user can draw →
$ tell us what you're building and where it's stuck
Book a 30-min discovery call
We reply within one business day.