TIBCO Software Inc. (Nasdaq: TIBX ) continued its market leadership in the fast-growing Complex Event Processing (CEP) space, according to a new report from IDC. TIBCO thus marked another year as the undisputed CEP leader, with a market share of 40.2 percent — twice the share of its closest competitor-while experiencing 52 percent year-over-year growth, according to the IDC study.

“CEP is the fastest-growing segment of the global event-driven middleware market,” according to Maureen Fleming, director of IDC’s BPM and middleware research program. “We expect this growth to continue as enterprises build event-driven applications that, in essence, act as real-time navigation systems for business.”

Reference (PRNewsWire):

TIBCO Leads Fast-Growing CEP Space, Says Leading IT Analyst Firm

1. The Sun-Sentinel newspaper in Fort Lauderdale accidentally republishes a six-year-old news story about the bankruptcy of UAL. It wasn't on the home page, but instead buried somewhere inside the web site.
2. Google's news crawler (an automated thing, remember) finds the story and incorporates it as part of its news feed.
3. Investors see the story, and immediately react. When UAL's stock plunged 76% to a low of $3, Nasdaq shut down trading. Eventually trading resumed, and the stock closed at just under $11, losing about 11%.
4. United blamed Tribune Company (the owner of the Sun-Sentinel) for "irresponsibly" changing the date on the story and demanded a retraction.
5. Tribune Company blamed Google, claiming they've had issues with Google's crawler "for months."

Who will blame be shifted to next?

Look -- if people haven't realized by now that the Internet pretty much lacks a delete function, then (IMNSHO) it becomes the requirement of each and every one of us to pay close attention to what we're reading, to use our own big brains and fine-tuned bullshit detectors to suss out whether something makes sense.

Since this is my blog, I'm going to parcel out blame the way I see it:

• United: 0%. If the concept of "negative blame" made any sense, then I'd actually write −∞ (that's a negative infinity, in case your character set is different than mine).
• Google: 5%. How can an automated crawler know that a newly-dated story isn't really new? Well, those folks over there at Google are smart. Certainly it shouldn't be that difficult to compare a "new" article against existing ones. Content hashes won't work as a comparison tool, because the date would be included in the hash computation, thus making the hashes different anyway. Full-text comparisons? Sure, it would take a lot of horsepower. Perhaps not every "new" story needs comparison, but at least the crawler could submit to the comparator any stories that ought to be verified (say those with the word "bankruptcy" in them).
• Tribune Company: 30%. Hey guys, you changed the date on the article. Don't go blaming someone else for your screw-up.
• Investors: 65%. If you're using an automated news aggregator (remember, an aggregator is not a source of news) to make major financial decisions -- decisions that affect the livelihoods of thousands (maybe millions) of people -- well, you're a moron. You should know that incorrect information can be just as instantly available as correct information. Verify potentially damaging claims before engaging in reckless behavior.

What's this got to do with security? I don't know, maybe nothing directly related. But it certainly raises the question -- what if someone intentionally wanted to cause nearly permanent damage to a person or a corporation? Malicious content, disguised as "news," certainly seems to have become a potentially successful attack vector this week.

Worried about a social engineering attack on a massive scale? I suspect that what happened Monday (8 September) was the largest social engineering attack in history -- although I wouldn't classify it as intentionally malicious. Just you wait until the idea spreads.

Mr Tancredi said Verisign's fraud detection kit would help "decrease the time between the attack being launched and the brokerage being able to respond".

Before now, he said, brokerages relied on counter measures such as restrictive stock trading or analysis packages that only spotted a problem when money had gone.

Verisign's software is a module that brokers can add to their in-house trading system that alerts anti-fraud teams to look more closely at trades that exhibit certain behaviour patterns.

"What this self-learning behavioural engine does is look at the different attributes of the event, not necessarily about the computer or where you are logging on from but about the actual transaction, the trade, the amount of the trade," said Mr Tancredi.

"For example have you liquidated all of your assets in stock that you own in order to buy one penny stock?" he said. "Another example is when a customer who normally trades tech stock on Nasdaq all of a sudden trades a penny stock that has to do with health care and is placing a trade four times more than normal."

This is a good use of data mining because, as I said previously:

Data mining works best when there's a well-defined profile you're searching for, a reasonable number of attacks per year, and a low cost of false alarms.

Another news article here.

Ironically, our favorite software vendors have decided, in a nutshell, to redefine Dr. David Luckham’s definition of “event cloud” to match the lack-of-capabilities in their products.  

This is really funny, if you think about it. 

The definition of “event cloud” was coordinated over a long (over two year) period with the leading vendors in the event processing community and is based on the same concepts in David’s book, The Power of Events. 

But, since the stream-processing oriented vendors do not yet have the analytical capability to discover unknown causal relationship in contextually complex data sets, they have chosen to reduce and redefine the term “event cloud” to match their product’s lack-of-capability.  Why not simply admit they can only process a subdomain of the CEP space as defined by both Dr. Luckham and the CEP community-at-large? 

What’s the big deal?   Stream processing is a perfectly respectable professional!

David, along with the ”event processing community” defined the term “event cloud” as follows:

Event cloud: a partially ordered set of events (poset), either bounded or unbounded, where the partial orderings are imposed by the causal, timing and other relationships between the events.

Notes: Typically an event cloud is created by the events produced by one or more distributed systems. An event cloud may contain many event types, event streams and event channels. The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud. A stream is a cloud, but the converse is not necessarily true.

Note: CEP usually refers to event processing that assumes an event cloud as input, and thereby can make no assumptions about the arrival order of events.

Oddly enough, quite a few event processing vendors seem to have succeeded at confusing their customers, as evident in this post, Abstracting the CEP Engine, where a customer has seemingly been convinced by the disinformational marketing pitches  - “there are no clouds of events, only ordered streams.”

I think the problem is that folks are not comfortable with uncertainty and hidden causal relationships, so they give the standard “let’s run a calculation over a stream” example and state “that is all their is…” confusing the customers who know there is more to solving complex event processing problems.

So, let’s make this simple (we hope). referencing the invited keynote at DEBS 2007, Mythbusters: Event Stream Processing Versus Complex Event Processing.

In a nutshell…. (these examples are in the PDF above, BTW)

The set of market data from Citigroup (C) is an example of multiple “event streams.”

The set of all events that influence the NASDAQ is an “event cloud”.

Why?

Because a stream  of market data is a linear ordered set of data related by the timestamp of each transaction linked (relative speaking) in context because it it Citigroup market data.    So, event processing software can process a stream of market data, perform a VWAP if they chose, and estimate a good time to enter and exit the market.  This is “good”.

However, the same software, at this point in time, cannot process many market data feeds in NASDAQ and provide a reasonable estimate of why the market moved a certain direction based on a statistical analysis of a large set of event data where the cause-and-effect features (in this case, relationships) are difficult to extract.  (BTW, this is generally called “feature extraction” in the scientific community)

Why?

Because the current-state-of-the-art of stream-processing oriented event processing software cannot perform the required backwards chaining to infer causality from large sets of data where causality is unknown, undiscovered and uncertain.

Forward chaining, continuous query, time series analytics across sliding time windows of streaming data can only perform a subset of the overall CEP domain as defined by Dr. Luckham et al.

It is really that simple.   Why cloud and confuse the community?

We like forward chaining using continuous queries and time series analysis across sliding time windows of streaming data. 

There is nothing dishonorable about forward chaining using continuous queries and time series analysis across sliding time windows of streaming data.   

There is nothing wrong with forward chaining using continuous queries and time series analysis across sliding time windows of streaming data. 

There is nothing embarrassing about forward chaining using continuous queries and time series analysis across sliding time windows of streaming data. 

Forward chaining using continuous queries and time series analysis across sliding time windows of streaming data is a subset of the CEP space, just like the definition above, repeated below:

The difference between a cloud and a stream is that there is no event relationship that totally orders the events in a cloud. A stream is a cloud, but the converse is not necessarily true.

It is really simple.   Why cloud a concept so simple and so accurate?

Dan Kaplan over at SC Magazine had an article up today  (they use Intense Debate for comments too)about ArcSight's first day of trading.  It seems that in spite of the overall condition of the market, they went ahead with their planned IPO.  They picked a bad day to do so, as the NASDAQ was off 1.74%. Opening at 9 dollars a share (the low end of their expected range), they closed at 8.78, bouncing off an intra day low of 8.07.

OK not an auspicious start, but I think they deserve credit for putting the ship out in this storm. I remember when I was at Interliant and we were planning our IPO.  Trying to time the market is a fools game.  Sometimes you just have to go for it.  Only time will tell if the market rewards ArcSights gumption to go public at this time or punish them as they have done recently with Sourcefire. For reasons that include purely selfish ones would love to see the public markets be a viable alternative for security companies to pursue liquidity events and access to capital.  Without them no one will be able to gain the girth necessary to compete with the current security monoliths.

Dan Kaplan over at SC Magazine had an article up today  (they use Intense Debate for comments too)about ArcSight's first day of trading.  It seems that in spite of the overall condition of the market, they went ahead with their planned IPO.  They picked a bad day to do so, as the NASDAQ was off 1.74%. Opening at 9 dollars a share (the low end of their expected range), they closed at 8.78, bouncing off an intra day low of 8.07.

OK not an auspicious start, but I think they deserve credit for putting the ship out in this storm. I remember when I was at Interliant and we were planning our IPO.  Trying to time the market is a fools game.  Sometimes you just have to go for it.  Only time will tell if the market rewards ArcSights gumption to go public at this time or punish them as they have done recently with Sourcefire. For reasons that include purely selfish ones would love to see the public markets be a viable alternative for security companies to pursue liquidity events and access to capital.  Without them no one will be able to gain the girth necessary to compete with the current security monoliths.

That would probably be all fine and dandy except some of the characters actually mean things in programming languages. for instance % * # $ ~ + ! @ are included in the list of possible legal characters. How many lines of code do you think need to be reviewed and fixed before this actually will work seamlessly? My guess is many millions. How many new exploits do you think this will open? Hard to say, but it should be interesting to watch.