Home > Published Issues > 2020 > Volume 15, No. 11, November 2020 >

The Experimental Study of Performance Impairment of Big Data Processing in Dynamic and Opportunistic Environments

Wei Li and William W. Guo
School of Engineering & Technology, Central Queensland University, Australia

Abstract—In contrast to HPC clusters, when big data is processing in a distributed, particularly dynamic and opportunistic environment, the overall performance must be impaired and even bottlenecked by the dynamics of overlay and the opportunism of computing nodes. The dynamics and opportunism are caused by churn and unreliability of a generic distributed environment, and they cannot be ignored or avoided. Understanding impact factors, their impact strength and the relevance between these impacts is the foundation of potential optimization. This paper derives the research background, methodology and results by reasoning the necessity of distributed environments for big data processing, scrutinizing the dynamics and opportunism of distributed environments, classifying impact factors, proposing evaluation metrics and carrying out a series of intensive experiments. The result analysis of this paper provides important insights to the impact strength of the factors and the relevance of impact across the factors. The production of the results aims at paving a way to future optimization or avoidance of potential bottlenecks for big data processing in distributed environments.
 
Index Terms—Big data, performance impairment, dynamic environment, MapReduce, scalability

Cite: Wei Li and William W. Guo, "The Experimental Study of Performance Impairment of Big Data Processing in Dynamic and Opportunistic Environments," Journal of Communications vol. 15, no. 11, pp. 776-789, November 2020. Doi: 10.12720/jcm.15.11.776-789

Copyright © 2020 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.