METEOROLOGICAL DATA ANALYSIS USING HDINSIGHT WITH RTVS

Mugdha Kulkarni, Priyusha Nair, Shruti Kulkarni, Swati Shekapure

Abstract: Weather prediction is a vital application in meteorology and has been one of the most scientifically and technologically challenging problems across the world. As per survey weather reports generated are huge in amount and in unstructured format. There is need for analysis of real time weather data for giving predictions. Data Mining is the computer assisted process of digging through and analyzing enormous sets of data and then extracting the meaningful data. In today’s world Big Data processing is the need of an hour. Ability to represent and query data with little and no apparent structure arises in several fields. We have focused on three main algorithms that are important for prediction on any kind of data. Meteorological data analysis is a system which considers real time data while making predictions and giving out weather forecasts. It should be scalable, portable and should work on variety of client systems. It should be able to handle Big Data and give outputs according to visualization effect entered by the end user. The developer must have greatest privilege over all other users, including the weather forecasting personnel and authenticated users. Till date, various weather mobile applications have been developed using clustering and regression algorithms, but real time analysis is still a big challenge. There is also need of more accurate predictions based on any type of dataset. We propose a solution to this by using R programming language for analysis of weather data using Microsoft Azure HDInsight for good long term predictions. This solution can also be as a SMART CITY application

Keywords: Azure, HDInsight, Hive, RTVS, R, Time-Series, Naive forecast, Shiny, Markdown

DOI: https://doi.org/10.15623/ijret.2016.0504049