Skip to main content

Complex Event Processing

Try Databricks for free
Get Started

What is Complex Event Processing [CEP]?

Complex event processing [CEP] also known as event, stream or event stream processing is the use of technology for querying data before storing it within a database or, in some cases, without it ever being stored. Complex event processing is an organizational tool that helps to aggregate a lot of different information and that identifies and analyzes cause-and-effect relationships among events in real time. CEP matches continuously incoming events against a pattern and provides insight into what is happening. and allows you to proactively take effective actions.

Complex events are commonly related to important business events [such as opportunities or threats] with the implication that they will be responded to in real-time or at least in something approaching to real-time.

Major Application Areas for Complex Event Processing [CEP]:

  • Business Activity Monitoring aims at identifying problems and opportunities in early stages by monitoring business processes and other critical resources.
  • Sensor Networks that are used for monitoring of industrial facilities. These are usually derived from raw numerical measurements [e.g., temperature, smoke].
  • Market data such as stock or commodity prices; they need to be derived from several events and their relationships through CEP.

The Most Common Tools Used for Complex Event Processing Are:

  • Apache Spark Streaming used by Databricks
  • Apache Flink used by data Artisans
  • Apache Samza used by LinkedIn
  • Apache Storm used by Twitter
  • Hadoop/MapReduce.
  • Amazon Kinesis Analytics
  • Microsoft Azure Stream Analytics, Stream Insight
  • Fujitsu Software Interstage Big Data Complex Event Processing Server
  • IBM Streams, Operational Decision Manager [ODM]
  • Oracle Stream Analytics and Stream Explore

Complex Event Processing is mainly used to address the following requirements: Latency has to be low. It is usually expected to be less than a few milliseconds, but sometimes we can have as little as less than one millisecond, between the time that an event arrives and the moment it is processed. The volume of input events per second is quite high. We can typically expect to have hundreds or even a few thousand events per second. Event patterns to be detected are complex: such as patterns based on temporal or spatial relationships.

Additional Resources

Back to Glossary