TIBCO Software Inc. (Nasdaq: TIBX ) continued its market leadership in the fast-growing Complex Event Processing (CEP) space, according to a new report from IDC. TIBCO thus marked another year as the undisputed CEP leader, with a market share of 40.2 percent — twice the share of its closest competitor-while experiencing 52 percent year-over-year growth, according to the IDC study.

“CEP is the fastest-growing segment of the global event-driven middleware market,” according to Maureen Fleming, director of IDC’s BPM and middleware research program. “We expect this growth to continue as enterprises build event-driven applications that, in essence, act as real-time navigation systems for business.”

Reference (PRNewsWire):

TIBCO Leads Fast-Growing CEP Space, Says Leading IT Analyst Firm

Even thought TIBCO’s BusinessEvents does not yet support Bayesian Classifiers, Artificial Neural Networks and other advanced decision support algorithms, it is just a matter of time before TIBCO will add these advanced features “out of the box”.  On the other hand, the extensible nature of TIBCO’s BE makes it possible to add probabalistic computing functionality, however this requires quite a lot of programming and integration work.

When I see a great release like this for TIBCO, it makes me a little nostalgic for “the good old days” travelling the world in the front of the aircraft for TIBCO.   TIBCO has a rich and diverse customer base.  This customer base includes financial services companies; however, TIBCO is much less dependent on financial services than other event processing companies.   So, with TIBCO you not only get great technology, but rock-solid stability in an unstable and uncertain business world.

As a side note, an S&P analyst recently downgraded TIBCO’s stock (TIBX), primarily due to chao in the financial services sector.    Because of TIBCO’s global reach and stability, plus forward vision, advanced technologies and many years of commericial success, the S&P downgrade will create a buying opportunity for TIBCO stock.

Freud wrote that dreams are the "Royal Road" to the unconscious. Messaging Systems and mainframes can be the Royal Road for a malicious user to attack a banking system, unless the security thinking moves from a "claim by claim" thinking to thinking about the full transaction.

I agree that these gentlemen are top notch researchers, witnessed by the fact that the authors do not mention nor claim to be “complex event processing” anywhere in their paper!  This paper is not about CEP, nor does it claim to be about CEP, it is about stream processing and unifying SQL standards.

ABSTRACT: This paper describes a unification of two different SQL extensions for streams and its associated semantics. We use the data models from Oracle and StreamBase as our examples. Oracle uses a time-based execution model while StreamBase uses a tuple-based execution model. Time-based execution provides a way to model simultaneity while tuple-based execution provides a way to react to primitive events as soon as they are seen by the system.

Asmentioned on numerous occasions, stream processing is a very important area in CEP/EP.   It is important not to confuse the higher situational knowledge from object-object correlation and state management with the single-object event refinement that occurs in stream processsing.    Event stream processing is fundamentally different than complex event processing. 

Event stream processing performs operations on streaming event objects.   In almost all advanced CEP/EP applications is is necessary to perform robust track and trace operations on streaming event objects, like tracking the position of an airplane.    Tracking the position of an aircraft can be modelled very nicely with event stream processing.  Tracking individual event objects is a precuror to multiclass object situation refinement.

When we manage the state of all the aircraft in the skies over New York, you need more than a stream processing construct.  You need to manage the state of all the aircraft.  Paul Vincent of TIBCO Software being to address this important point in The Value of State…  

Again, we will be better equiped to solve complex distributed event processing problems if we do not confuse the notion of event stream processing and complex event processing.   These technologies are indeed complimentary, both very important, but they are not the same.

I applaud Oracle and StreamBase’s work toward a unified standard for SQL extensions for streams.

I agree with Mark that SOR is important and very interesting; but in his reply he seems to be confusing CEP with “complex EAI” or a “complex messaging” application.  For example, Mark says,

“It’s not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it’s a stunningly complicated environment in which to create a complex, real-time model in which to apply “simple” routing decisions. On this basis alone, SOR needs CEP.”

Connecting to many market feeds with multiple asset classes might be complicated, but “complicated connections” are an EAI  (adaptation layer) function, not a core CEP function.   In fact, TIBCO Software has been doing this type of low latency back-office order routing for many years, and TIBCO historically calls this “messaging.”  Adding some rules to high speed, low latency messaging does not make it a “CEP” application.

Mark goes on to set up a counter argument to ILOG’s Changhai Ke, comments with,

“SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.”

This is the well travelled argument the “new stream processing vendors in capital markets” have been saying, still unconvincingly, for the last few years.  Basically their perspective is that if you have a lot of ”feeds” and a core requirement for “speed” - “feeds and speed” - you are doing “complex event processing.” 

Mark Palmer forcefully stated his opinon that the folks who do not agree with him do not “understand” modern day SOR.    However,  a strong counter argument can be made that the “newcomers” to capital markets like StreamBase do not understand that “feeds and speeds” with order routing is little more than moderan day EAI.   This is a basic message routing capability and it has been around for a long time.  After all, Wall Street operated quite well before the term CEP was coined!  TIBCO technology was providing Wall Street back office, low latency, smart order routing a decade ago, and they called this technology “messaging”.  

So, I remain unconvinced, at least by Mark’s passionate counter post, that SOR is CEP.   SOR, as Mark and other have described it, is a low latency messaging technology.  Message routing rules have exisited in this technology space for decades.

I agree with Mark completely that low latency EAI (like SOR has been described) can be quite complex, from a “feeds and speeds” perspective.   However,  I remain skeptical that “feeds and speeds” is much more than  modern day messaging and message routing.

In closing, in the network and security management world we have been dealing with “myriad feeds and speeds” for as long as I can remember, but admitted not like capital markets.    Taking myriad feeds, running rules against the feeds and then routing the messages/events for further processing, regardless of the complexity of the feeds and the data, is actually more of a messaging/ESB technology than a CEP technology. 

I remain completely open minded to any convincing counter arguments.

My goal at that time was to show everyone that there was a very mature (functional) reference architecture with decades of maturity that applies to (complex) event processing, adapted from the JDL model for information fusion.  Kum Bai Ya.

There is plenty of room for everyone in this model.  Kum Bai Ya.

The model is inclusine not exclusive. Kum Bai Ya.

The JDL model is based on years of operational maturity.  Kum Bai Ya.

The model is functionally and technically accurate.  Kum Bai Ya.

Everyone at the first event processing symposium (March 2006) seemed to agree with this model, at least publicly, because there was no “push back” at the symposium.  Kum Bai Ya.

Professor David Luckham did not discuss architecture in his book, The Power of Events.  Kum Bai Ya.

David’s research at Stanford, some CEP related, was funded by DARPA, who also support the JDL information fusion model.  Kum Bai Ya.

TIBCO Software adopted the JDL model (Note: I worked for TIBCO the time.) Kum Bai Ya.

We built a functional reference architecture around this mature model.  Kum Bai Ya.

We did not claim we invented it.  Kum Bai Ya.

We did not patent the model, only shared it. Kum Bai Ya.

The model is free and open for everyone to use.  Kum Bai Ya.

The folks in the military and government totally agree with this model for CEP/EP.  Kum Bai Ya.

Complex operational problems are addressed every day with this model. Kum Bai Ya.

Air traffic control uses this model.  Kum Bai Ya.

Missile defense uses this model to protect us from harm everyday.  Kum Bai Ya.

Intrusion detection and network management now use this model (Note: I published an ACM paper on adapting this model for cybersecurity 10 years ago). Kum Bai Ya.

Oh, blog-o-sphere.  Kum Bai Ya.

Jeff uses the example, “Smart Order Routing” as an example of taking an event and routing the resulting market order match based on some simple rules.    Routing a order kicked off by a simple order match against a deep liquidity pool (or other market factor) does not define complex event processing nor detecting a complex event - the core idea behind CEP.   Order routing based on simple rules is BPM, plain and simple.

Let’s take another example, fraud.  In this example, there is some complex neural network monitoring for credit card fraud and a potential fraud is detected - this is CEP, detecting a complex event based on some sophisticated analytics.   

After a possible fraud has been detected, a process looks into a database and the routes the incident to someone in the company who is a (1) specialist in credit card fraud, (2) working at the same time of the discovered threat, and (3) immediately available to act on this type of task.   Routing the incident is not CEP, it is BPM.

Jeff makes the argument that it is OK to call an event-driven BPM task CEP because “it fits the EPTS definition” in the CEP glossary.   He also avoids the discussion of detection accuracy, and instead insists that latency is a ”very important” factor in a CEP application.

If you read the various post by vendors in the blog-o-sphere, it is obvious that they are continually defining CEP as BAM, BPM, BRE, BRMS, SOA and just about every other related processing activity that is complimentary to the event correlation and analysis required to detect an opportunity or threat to your business.

I’m not picking on Aleri.  TIBCO has been doing the same thing recently in their CEP blog, continually attempting to redefine CEP as BRMS.    Detecting business opportunities and threats with high confidence requires sophisticated analytics, and their tools have not yet evolved to “real CEP” capabilities.  Instead, vendors are attempting to redefine BPM, BRMS, BRE, and even SOA to some degree, as CEP. 

CEP is Not BPM, BAM, BRE, BRMS or SOA.

adaptive process management (n.) an element of resource and business process management, adaptive search and event processing. Sometimes referred to as “Level 4” event processing or process refinement.

application concept (n.) a definition of a set of properties that represent the data fields of an application entity. An application concept can describe relationships among themselves. For example, an order concept might have a parent/child relationship with an item concept. A department concept might be related to a purchase requisition concept based on the shared property, department_id. Application concepts can include an application state model.

application state modeler (n.) a UML-compliant application that allows you to model the life cycle of a concept instance — that is, for each instance of a given concept, you can define which states it will pass through and how it will transition from state to state. States have entry actions, exit actions, and conditions, providing precision control over the behavior of an event processing agent. Transitions between states also may have rules. Multiple types of states and transitions maximize the versatility and power of the application state modeler.

derived event (n.) an event that is created as a result of processing one or more other events.

complex event (n.) an event that is a situation-entity abstraction of two or more simple, derived or other complex events.

complex event processing (n.) CEP is a technology for extracting information from message-based systems. CEP is primarily an event processing concept that deals with the task of processing multiple events from an event cloud with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes.

event (n.) a instance of an event definition. It is an immutable object that represents a business activity that happened at a single point in time. Just as one cannot change the fact that a given activity occurred, one cannot change an event — events are immutable.

event aggregation (n.) the aggregation of simple, derived or complex events into higher levels of event abstractions.

event definition (n.) a set of properties related to a given activity that represents an important or interesting change of state in a human, system or computational activity. An event definition includes event properties such as event priority, event time to live (TTL), and a description of the payload, which is comprehensive information related to the activity that occurred. Events expire when the TTL has elapsed, unless the event processing agent has instructions to consume them prior to that time.

event channel (n.) a communications channel in which events are transmitted from event source to event receivers, typically received as electronic messages. Each channel can have multiple destination and. events can be configured to transmit to a default destination. JMS is an example of an event channel.

event cloud (n.) a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events. Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud.

event-driven (n.) the behavior of a human, system or computational entity whose execution or actuation is in response to events, typically received as electronic messages.

event-driven architecture (n.) an architectural style for distributed computing applications in which some of the components are event-driven and communicate by means of events.

event processing (n.) computing that performs operations on events, including modifying, creating and destroying events.

event-object (n.) an software object that represents an event, generally for the purpose of computer processing, that exhibits both encapsulation, inheritance and polymorphism.

event prediction (n.) computational activity where the impact of events, complex events, and situations caused by events identified, including both opportunity or threat. Sometimes referred to as “Level 2” event processing, impact assessment or predictive analytics.

event pre-processing (n.) computational activity where events are cleansed or normalized to produce semantically understandable data. Sometimes referred to as “Level 0” event processing.

event processing (n.) computational activities on events dealing with the association, correlation, and combination of event data and information from single and multiple event sources to achieve refined identity and situation estimates for observed event objects, and to achieve complete and timely assessments of opportunities, threats, and their significance. Event processing is characterized by continuous refinements of event estimates and assessments and by evaluation of the need for additional sources, or modification of the process itself, to achieve improved results.

event processing agent (n.) an EPA is a computational entity that performs event processing.

event processing network (n.) a set of event processing agents and a set of event channels connecting them.

event properties (n.) data representation of an event, typically by name-value pairs of type string, integer, real, boolean or a complex data type.

event refinement (n.) filter, identify and track events & make initial processing decisions based on association, correlation and state estimation. Sometimes referred to as “Level 1” event, or event-object, track and trace.

event stream (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

event stream processing (n.) a time-ordered sequence of events. An event stream may be bounded by a certain time interval or other contextual dimension (content, space, source, certainty), or be open ended and unbounded.

rule (n.) defines what triggers unusual, suspicious, problematic, or advantageous activity within an event processing agent and what the EPA does when it discovers these types of activities. Rules execute actions based on certain conditions on events, instances, or a combination of both. A rule includes a group of condition-rule statements and action-rule statements. The condition statements instruct the EPA what to look for in events, and action statements instruct the EPA how to respond when conditions are met. If all the conditions in a rule are satisfied by events or instances or both, the EPA fires the actions. The action might be to execute tasks, create an event instance, modify property values in an event instance, create and send an event, or something else.

rules engine (n.) a type of event processing agent that uses a declarative programming model to process events. Formally described as “an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet“. Informally, it can be any system that uses rules, in any form, that can be applied to data to produce outcomes.

rule language (n.) is an artificial language that is used to control the behavior of an event processing agent. Rules languages, like human languages, have syntactic and semantic rules to define meaning.

situation refinement (n.) identify situations, or complex events, based on event clustering, event-event relationships and relationship analysis and context. Sometimes referred to as “Level 2” event processing.

simple event (n.) an event that is not an abstraction or composition of other events.

virtual event (n.) an event that is imagined, modeled or simulated.

However, I should briefly clarify Paul’s note that “blackboard systems historically used a single memory model (i.e. multiple threads or processes using a single machine’s memory model)“.

In fact, there were many blackboard systems, some more than a decade old, that used a distributed memory data-model. What I think Paul meant to say, and my apologies to Paul for being so literal, is that “blackboard systems originally used a single memory model (i.e. multiple threads or processes using a single machine’s memory model)”

John McManus, former CTO of NASA, wrote an excellent PhD dissertation in 1992, Design and Analysis Techniques for Concurrent Blackboard Systems. John’s thesis, now more than 16 years old, examined many details of concurrent blackboards where memory is distributed. For example, refer to Figure 2.3. Distributed Blackboard System with Distributed Blackboard Data Structure, page 36 of John’s dissertation.

Quoting directly from page 37 of John’s disseration;

Rice, Aiello and Nii [20] present several options for gaining speedups in a distributed blackboard system.

• 1) Eliminate the centralized scheduling mechanism
• 2) Optimize system design for a distributed memory, message-passing hardware
• 3) Distribute the data across the blackboard to reduce hotspots

Quoting further from the same page;

Poligon [21] is based on a distributed memory hardware model when each processor is viewed as a blackboard node. They define a blackboard node as follows: “a blackboard node is a process on a processor, surrounded by a collection of processors able to service its requests to execute rules.” [22] The implicit assumption in this definition is that all knowledge sources are rule–based systems. This assumption may severely limit the performance of systems implemented using Poligon, and limits the types of problems it is suited to address.

In Blackboards for Complex Event Processing, Paul concludes,

“One suspects the blackboard systems domain and terminology is overdue some updates thanks to developments in the Complex Event Processing space.”

If you look at the historical literature, I would say that the following restatement is more accurate:

“The CEP domain and terminology is overdue some updates because folks working in CEP did not reference or incorporate the advanced event processing prior art in a number of very important areas, blackboard systems being only one.”

On the other hand, commercial off-the-shelf rule-processing technology such as TIBCO’s BusinessEvents (BE), advances the ability to economically implement myriad complex problems that blackboard systems are designed to address.