The client was looking for insights with the kind of data granularity that required drilled down reporting. They needed a big data pipeline for ingesting real-time data collected from their viewers globally. We built a robust platform that generates quantitative analytics reports based on customer usage patterns to provide the client with insights they need to make better, more informed decisions.
The player and ad events are processed using spark structured streaming in Databricks. Structured Streaming is a scalable and fault-tolerant stream processing engine. One can express the streaming computation the same way one would express a batch computation on static data. The Spark SQL engine takes care of running it incrementally and continuously and updating the result as streaming data arrives continuously.