Collecting data is easy. Working out which piece matters to whom is the real work.
OmgevingsAlert started in 2012 with a simple idea: choose a location and receive relevant permits and spatial developments in the neighbourhood. Today the platform processes many more kinds of information. Every source has its own format, rhythm, quality and geographic logic. Most of the engineering is in everything that has to happen before you can show any data.
The case in brief
In development since 2012
From government XML to live public transport feeds
Geographic and content-based per location
Apps, dashboard, website, push and email
Where is the complexity?
The user only wants to know one thing: what is happening around me that matters to me? Behind that simple question is a data platform in which information from different organisations, protocols and formats has to come together.
See the approachEvery source works differently
Official publications come from mutation-event XML and metadata, roadworks from a gzipped DATEX feed, public transport disruptions via a continuous ZeroMQ pub/sub channel, weather warnings via KNMI open data and local news via RSS, Atom or HTML. So there is no such thing as one generic importer.
Source data does not say who it is relevant to
A permit, traffic measure or warning may be available nationwide, but a user only wants information for their own home, neighbourhood, route or managed property. For that, the data needs reliable geographic information and then has to be matched against locations and areas.
The same data has to work in several places
The normalised information is used by the mobile apps, Mijn OmgevingsAlert, OmgevingsAlert Pro, the public website and OverUwBuurt. The same backend also drives push notifications, email digests and other product flows.
How did we approach it?
- 01
Retrieve every source reliably
For each source we built the right integration: scheduled imports, API connections, XML processing, RSS and HTML imports and processes that continuously listen to live data streams. Import work runs asynchronously so large or slow sources do not block the rest of the product.
- 02
Translate everything into one shared model
Source records keep their own specific information, but are then indexed into a shared Feature model. This means apps and other consumers do not need to understand how a KNMI warning, announcement or public transport message was originally delivered.
- 03
Geography at the core
With PostgreSQL and PostGIS we link points, areas and other geometry to the locations users follow. Assets can be, for example, addresses, points, polygons or imported KML areas.
- 04
Only then decide what is relevant
After the geographic selection we apply filters on, among other things, category, distance, words that must appear and words that must be excluded. Only relevant combinations are linked to a user's inbox.
What did we build?
The core is a data platform that reduces very different sources to one usable stream of local information. On top of that sit matching, filtering, notifications and several product interfaces.
A broad import layer
The platform processes, among other things, official publications, zoning plans, police reports, KNMI weather warnings, NDW roadworks, NDOV disruptions, local news, earthquakes and municipalities' own messages.
One normalised Feature pipeline
Every source is translated into shared fields such as category, publication date, geometry, source information and the parts needed to show the message in the different products. New source types therefore do not have to be implemented separately in every client again.
A geographic matching engine
New data is linked to relevant locations and objects. For consumers that can be a point with a radius; professional users can also monitor larger areas or their own geometry. Matching for larger numbers of assets is processed in batches to keep the background queues manageable.
What does it deliver?
One product on top of completely different data sources
To the user, a permit, a road closure or a weather warning feels like the same kind of product experience, while the underlying sources technically have little in common.
From national data to personal relevance
The backend translates large streams of general information into a much smaller selection per location. Users do not have to search government sites, news feeds and traffic sources themselves.
A foundation we can keep building on
New categories can use the same import, normalisation, matching and notification infrastructure. On that basis we later also added AI filters that can judge, for certain workflows, whether an alert is relevant in terms of content.
The complexity is under the hood
OmgevingsAlert shows well why data engineering is more than reading an API and storing records. The value only emerges when unreliable, changing and dissimilar input can move safely through a complete production chain.
View casesBatch, real-time and periodic side by side
One source is checked daily, police data every ten minutes and public transport messages arrive through a permanently listening pub/sub process. The backend has to process all these forms reliably side by side.
PostGIS decides what is relevant where
Features and assets contain real geometry. Matching uses spatial calculations, distance rules and area relations instead of just textual place names or postcodes.
The pipeline does not end at storage
Import is followed by normalisation, matching, filtering, inbox status, access rules and notification. Push can be sent immediately or bundled, and Pro users can receive relevant alerts via an email digest.
Related AI challenges
Make your own documents, data and knowledge usable for AI
The valuable information is already there. Now AI has to be able to work with it reliably, securely and practically.
View use caseAdd AI to existing software, an app or SaaS
The product already exists. Now you want to add AI functionality without losing sight of the existing architecture, user experience and reliability.
View use caseBuild & Integrate
We build the technical layer between your data sources and the product that has to work with them: imports, API connections, normalisation, processing, matching and reliable integrations.
View offerFixed price after scoping
- Result
- A production-ready data flow integrated into your software
- Duration
- Depends on scope
- Next step
- First we map sources, volumes, processing and the desired output