The importance of data analytics and being able to capture insight in real-time

DATA ANALYTICS, Data, The importance of data analytics and being able to capture insight in real-time
Listen to this article
Listen to
this article
Text to speech by Listencat
Text to speech
by Listencat

by Dan Seal, Product Lead, Streaming Analytics, Kx

Data analytics refers to the process of examining datasets to draw conclusions and extract valuable insights. Empowered with the most effective analytics systems, organisations can make decisions that drive profitability, grow customer numbers, set new trends and futureproof operations for years to come.

Access to such data has become so important that it is estimated that by 2022, more than half of major new business systems will incorporate continuous intelligence – this is a system that uses real-time data to improve decision making.

The transformation of decision making

One of the primary drivers for implementing a data analytics platform is that organisations have historically risen (and fallen) based on the collective efficiency of the decisions they make each day. These decisions can range from what products to develop, which target markets to engage with, and can be the difference between success or failure.

Previously, decision making was left in the hands of senior management, based on industry expertise and calculated forward planning. While this still is the case for many businesses, we now live in an environment where data and machines are also able to make efficient decisions. These can take place in real time and be based on past information. 

For example, you might have an analytic that measures a spike in temperature in an IoT application, but then this will also want to use a machine learning model to predict its future behaviour. Therefore, this platform will have to support both the analytic process and machine learning model to make the correct decision in real-time. Additionally, a supply chain could decide its carrier assignment to meet customer delivery requirements at the lowest cost, but that decision has to be done immediately in that moment. If not done in real-time, customer demand could not be met or enterprises would end up overpaying for the required service in that moment.

, , Next Level Trading with ForexTB, the premier platform for CFD Trading!

The data challenge 

Although a plethora of new data capture and analysis platforms now exist, finding the right one can still prove difficult. In today’s digitised world, over 2,500,000 terabytes of data is produced every day, and even the best of systems often store data in silos, meaning it’s difficult to access, analyse or compare. This generally leads to perishable insight, where once valuable data becomes worthless due to not being analysed quickly enough.

As such, we’re beginning to see great clamour towards streaming analytics platforms; these are systems that are able to process vast swathes of data in real-time, and contrast this with historical datasets to create new insights, make decisions in the moment and drive new value. 

Why streaming analytics?

Most organisations produce real-time data of some description. The question is whether they can drive value by reacting to it in the moment. With the ongoing digitisation of all industries and growth in new technologies such as IoT, more organisations will be able to find value in real-time data. To extract this value the timeframe in which decisions need to be made can be seconds, if not milliseconds. With the shelf life of valuable data limited, streaming analytics removes the requirement of batch processing data where data is processed in large volumes at once, enabling organisations to pursue a course, or change direction near instantly.

All data is potentially streaming as it all originates in dozens, hundreds, and sometimes thousands of applications in large enterprises, and so can be analysed to inform decisions. Data is the fuel for streaming analytics, but also for historical analytics, such as to train machine learning models, and real-time data enrichment. Having time series analytics, the process of analysing time series data to extract meaningful insights enables businesses to find insights and build models that will be used for streaming analytics use cases. 


Ultimately there are three questions that businesses need to focus on when it comes to making a decision. What outcomes will this decision have? What data sources do we need to make it happen, and what else can we do to make the customer/end-user happy? To address these, organisations need a streaming analytics platform that integrates within its existing application architecture and has the ability to crunch data and compare this insight with information that has already been stored. 

Modern businesses have come to expect immediate access to the information they are seeking. This information brings new insights which allows them to make decisions on the next action and alter the course of their business objectives. Although there are many data tools out there that promise such a solution, it’s hard to look away from streaming analytics as a centralised tool for better decision making.

DATA ANALYTICS, Data, The importance of data analytics and being able to capture insight in real-time

Dan Seal

Dan Seal is Product Lead, Streaming Analytics at Kx Systems

Data science skills: what’s driving the surge in demand?

Boris Paillard • 19th February 2021

The demand for people with specialist data skills — like data scientists for example — has more than tripled over the last five years (+231%). With nearly every single sector, including banking, transport and retail generating an explosion in data, employers are desperately on the hunt for skilled experts who can make sense of it.

Data and AI in Sport

John Gale • 17th February 2021

Data & AI increase in Sports The increased use and controversy of VAR in football has been the subject of much debate in recent times, however many athletes across all sports – and officials, have used artificial intelligence (AI) to improve their performance, in a number of different ways.

Join our TBT LinkedIn Group

John Gale • 09th February 2021

For TopBusinessTech networking and marketing opportunities, please join our LinkedIn Group and connect with us and your peers on LinkedIn

How is AI and data analytics improving performance at the...

Bruno Dagnino • 03rd February 2021

Bruno Dagnino is the CTO of Metrica Sports, a leading data and video analysis platform that has provided cutting edge solutions to elite football clubs since 2014. Aspiring analysts, players and coaches, can download Metrica for free here and try their hand at video and data analysis.

Using Data to Make Smarter Cybersecurity Risk Decisions

Stephen Roostan • 02nd February 2021

Most enterprise security teams are already drowning in a sea of sensor and scanner data that needs to be manually correlated, analysed, and interpreted. No easy task when you take into account the growing volumes of vulnerability data and the increasing complexity of today’s enterprise IT environments.