Most Real Time is Soft
Real time tends to get used in the Big Data and analytics space to imply timely answers, and not about making guarantees about response times. But it's worth thinking more about what timely means.
Real Time is Relative
Humans are only so fast (and we aren't getting faster). Jakob Nielsen wrote in 1993 that "0.1 second is about the limit for having the user feel that the system is reacting instantaneously; … 1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay.".
Interactions between machines demand lower latency. But definitions and expectations still vary wildly. In algorithmic trading systems in the financial services industry, engineers optimise hardware and hack on networking stacks to shave fractions of milliseconds of the time to respond to fluctuations in the pricing of stocks with updated orders. The notion of "real time" in that world is clearly different to the one in Big Data analytics.
The real time developments in the Hadoop framework are concerned with helping users find so-called "needles in haystacks", insights from large unstructured datasets, in something like the time it would take them to do so using existing tools on a small dataset.
Doug Cutting, Cloudera's chief architect, described the "real time" offered by Impala as waiting less: “It’s when you sit and wait for it to finish, as opposed to going for a cup of coffee or even letting it run overnight…. That’s ‘real time.’”.
Acunu is "API Real Time": for humans and applications
Acunu provides a solution where there is an understanding of the range of questions that need answering in advanced. This is a different and complementary set of problems from those tackled by Hadoop and data warehousing environments. Customers deploy us alongside (or in front of) Hadoop and Teradata in their environment. Hadoop helps where you don't know what the needle in your haystack looks like, or what it even means to compare the multiple sorts of data in one haystack. Where you have roughly an idea what the needle looks like, Acunu can help.
With Acunu, queries take milliseconds, and newly arrived events feature in query results typically within fractions of a second. We do this by asking users to set up cubes, essentially materialized views, that define the pattern of queries that we should deliver in real-time: which aggregates and roll-ups, what filtering, what "group by"s, etc. Using this knowledge, we pre-compute intermediate results for all of these answers. So, when a query is issued, most of the work has already been done to deliver the answer.
Whereas Doug Cutting's "no-coffee-break real time" works for investigative analytics, Acunu’s "API real time" works for operational intelligence and monitoring. Not only are results returned interactively, within the latency of a web page refresh or typical API call, but the analytics is incremental and continuous -- the numbers themselves are fresh. But more than this, the fresh data in queries is combined with the historic data so trends, exceptions, comparisons are all immediately detectable.
Often our customers use Acunu's built-in dashboards to monitor metrics they care about, and also power features of their application directly using the same data feed. Our recent webinar with Hailo gives an example of this. I'll explore this in more detail in a future post.