Internet of Things – IoT – Big Data to Data Lake

Contribution by Prashant Mendki – Director Alliances & Business Development at Bristlecone

In last 2 articles we discussed IoT as a third wave of internet and what are different scenarios of IoT communication from machine (

The next question is – with all these devices creating lot of data as they communicate with each other, how is that data being consumed and how will this data really end up as Big Data? Now when I say big data – its consumable data which needs to be retained over a period to analyze trends and make inferences to make decisions for actions in present or future. Volume of data, speed of processing and analytical power of your dashboard are 3 critical aspects of big data. Let’s look at four high level IoT scenarios and how big data is relevant to each.

Machine to Dashboard

In this scenario, the device will sense a particular signal and pass it on to dashboard for analysis. If something is unusual – then action will be defined. Now the life of data here is momentary. You may not need the “life as usual” data for any analytical purpose. So this scenario will not generate much data or leverage Big Data for point-to-point usage.

Machine to Machine

Here devices are talking to each other to synchronize actions or complete specific processes. Once a signal is communicated successfully – and another machine picks it up, the utility of that data is not much for analysis. If something goes wrong, it can be communicated with the dashboard to enable some action. Since the life of the data is again momentary – the data generated may not be much relevant to big data being a point-to-point usage.

Machine to big data

This is the most important scenario where machine is talking to big data. Data generated by all devices during the business process will be gathered and analyzed to make predictions. Think about the scale of data here. In the second wave of Internet – we invented big data and cloud to manage the amount of data. What will IoT need? The answer is “Data Lake”. The future of big data is in Data Lake. Data lakes are uncleansed and pure form of data. Since data needs to be leveraged as soon as it’s there – there is no time to make it structured and cleaned before it is used. The consumption mechanism of these data lakes through various analytical means are important topics to look for.

Machine to Business process

In this scenario – all devices are directly integrated into business process and there will be multilevel communication. This data will directly contribute to the data lake to continuous feedback, analysis, optimization, improvements as well as decision making. All this will generate huge data which will get into data lakes.

So all in all – in IoT era we will move from Big Data to data lakes – which will create a different data world.

In next article we will look at 2 things – how speed of processing provided by technologies like HANA and data from “data lakes” will be a major factor contributing to the IoT wave.

Previous articleBuilding an Effective Supply Chain Team
Next article5 Things Every Supply Chain Professional Should Know About Infrastructure
Prashant is a business strategy expert with over 15 year professional experience in Analytics, Data Management, Sales Management, Business Development, Strategic Alliances and strategic business planning. He possesses strong experience in solution practice management, go-to-market strategy and business development in high growth and mature markets. At Bristlecone he is responsible for driving revenue by developing partner strategy, working closely with partner ecosystem, and defining & executing demand generation programs. He has successfully implemented strategic collaboration with business partners and key customers to derive business value throughout the engagement value chain.