The Roads and Traffic Agency (Agentschap Wegen en Verkeer)  and the Flemish Traffic Centre (Vlaams Verkeerscentrum)  provide open data about the current traffic situation . More of 4000 measurement points dispersed all over Flanders – but mainly located on highways, detect each minute the number of vehicles and their speed.
We use the same front-end and data pipeline as presented for the pollution data . Because the amount of data is pretty big (more than 4000 measurements each minute), only a small selection of measurement points (~ 100) is captured every 10 minutes by an edge node and send to flume for further processing . Two streams are captured, one with the list of measurement points and the second with the real-time traffic data (see figure).
The traffic data are send to Apache Kafka  and two topics are created. The topic with measurement points are stored in Apache Hive  to have a structured query interface for further processing. The topic with the real-time traffic data is consumed with a python script and enriched with the measurement points stored in Hive to create a new Kafka topic with the enriched traffic data (see figure). This new topic is read by Druid  for visualization and data discovery using several tools such as Apache Superset .
The two other Kafka topics are send to Kudu tables which are used for the visualizations (see live map here).
 Meten-in-Vlaanderen (MIV) http://opendata.vlaanderen.be/dataset/minuutwaarden-verkeersmetingen-vlaanderen
 Agentschap Wegen en Verkeer http://www.wegenenverkeer.be/
 Vlaams Verkeerscentrum http://www.verkeerscentrum.be/
 Map of pollution in Flanders http://www.bigdatareflections.net/blog/?p=1
 Sending data to Flume using Python http://www.bigdatareflections.net/blog/?p=35
 Apache Kafka https://kafka.apache.org/
 Apache Hive https://hive.apache.org/
 Druid http://druid.io/
 Using Superset with Kudu http://www.bigdatareflections.net/blog/?p=52
One Reply to “Real-time traffic data”