REAL-TIME ROOT CAUSE IDENTIFICATION ON STREAMING HETEROGENEOUS DATA USING SPARK

S. Charles Britto, S.P. Victor

Abstract: Root cause identification can provide huge breakthroughs in the process of business decision making. However, the huge size of input data requires a very highly sophisticated system to perform the analysis. This paper presents a parallelized root cause identification architecture that identifies a ranked list of root causes for the user’s query. The major components of the architecture includes conflict identification and elimination, parallelized data pre-processing, polarity identification and polarity based root cause identification. Experiments were conducted on dynamically retrieved data using APIs and the retrieved data is passed to the parallelized components implemented in Spark architecture. The in-memory processing architecture of Spark provides effective results in real-time, making the architecture scalable and robust.

Keywords: Root Cause Analysis, Sentiment Analysis, Big Data, Spark, Parallelization

DOI: https://doi.org/10.15623/ijret.2016.0512005