Internet of Things – IoT – Big Data to Data Lake
Contribution by Prashant Mendki – Director Alliances & Business Development at Bristlecone
In last 2 articles we discussed IoT as a third wave of internet and what are different scenarios of IoT communication from machine (https://supplychainminded.com/internet-of-things-scenarios/
The next question is – with all these devices creating lot of data as they communicate with each other, how is that data being consumed and how will this data really end up as Big Data? Now when I say big data – its consumable data which needs to be retained over a period to analyze trends and make inferences to make decisions for actions in present or future. Volume of data, speed of processing and analytical power of your dashboard are 3 critical aspects of big data. Let’s look at four high level IoT scenarios and how big data is relevant to each.
Machine to Dashboard
In this scenario, the device will sense a particular signal and pass it on to dashboard for analysis. If something is unusual – then action will be defined. Now the life of data here is momentary. You may not need the “life as usual” data for any analytical purpose. So this scenario will not generate much data or leverage Big Data for point-to-point usage.
Machine to Machine
Here devices are talking to each other to synchronize actions or complete specific processes. Once a signal is communicated successfully – and another machine picks it up, the utility of that data is not much for analysis. If something goes wrong, it can be communicated with the dashboard to enable some action. Since the life of the data is again momentary – the data generated may not be much relevant to big data being a point-to-point usage.
Machine to big data
This is the most important scenario where machine is talking to big data. Data generated by all devices during the business process will be gathered and analyzed to make predictions. Think about the scale of data here. In the second wave of Internet – we invented big data and cloud to manage the amount of data. What will IoT need? The answer is “Data Lake”. The future of big data is in Data Lake. Data lakes are uncleansed and pure form of data. Since data needs to be leveraged as soon as it’s there – there is no time to make it structured and cleaned before it is used. The consumption mechanism of these data lakes through various analytical means are important topics to look for.
Machine to Business process
In this scenario – all devices are directly integrated into business process and there will be multilevel communication. This data will directly contribute to the data lake to continuous feedback, analysis, optimization, improvements as well as decision making. All this will generate huge data which will get into data lakes.
So all in all – in IoT era we will move from Big Data to data lakes – which will create a different data world.
In next article we will look at 2 things – how speed of processing provided by technologies like HANA and data from “data lakes” will be a major factor contributing to the IoT wave.