A Review of Real World Big Data Processing Structure: Problems and Solutions

Khalid Imtiaz; M. Junaid Arshad

A Review of Real World Big Data Processing Structure: Problems and Solutions

Khalid Imtiaz *

Contact

M. Junaid Arshad

Contact

A Review of Real World Big Data Processing Structure: Problems and Solutions

Article Fingerprint

ReserarchID

CSTSDE70F7C

A Review of Real World Big Data Processing Structure: Problems and Solutions Banner

Key Research Insights

Synthesized scholarly intelligence & interactive research assistant

This converted HTML version may contain rendering inconsistencies. Please refer to the PDF for the authoritative version, or click here to provide feedback.

Abstract

Information sort and sum in human culture is developing in astonishing pace which is brought about by rising new administrations as distributed computing, web of things and area-based administrations, the time of enormous information has arrived. As information, has been principal asset, how to oversee and use enormous information better has pulled in much consideration. Particularly, with the advancement of web of things, how to handling huge sum continuous information has turned into an extraordinary test in research and applications. As of late, distributed computing innovation has pulled in much consideration with elite, yet how to utilize distributed computing innovation for substantial scale ongoing information preparing has not been contemplated. This paper concentrated the difficulties of huge information firstly and finishes up every one of these difficulties into six issues. Keeping in mind the end goal to enhance the execution of constant handling of substantial information, this paper manufactures a sort of real-time big data processing (RTDP) design considering the distributed computing innovation and after that proposed the four layers of the engineering, and various leveled figuring model.

References

53 Cites in Article

Reference Format

1. Zhigao Zheng,Ping Wang,Jing Liu (2015). Unknown Title. Appl. Math. Inf. Sci, 9(6), 3169-3190.
2. O Tene,J Polonetsky (2012). Big data for all: Privacy and user control in the age of analytics. J]. Nw. J. Tech. & Intell. Prop, 11
3. Lohrs (2012). Theage ofbigdata. NewYorkTimes, 11.
4. R Bryant,R Katz,E Lazowska Big-Data Computing: Creating revolutionary breakthroughs in commerce, science, and society. Big-Data Computing: Creating revolutionary breakthroughs in commerce, science, and society
5. R (2008). Unknown Title. Ver, 8
6. Jonathan Overpeck,Gerald Meehl,Sandrine Bony,David Easterling (2011). Climate Data Challenges in the 21st Century. Science, 331(6018), 700-702.
7. Alexandros Labrinidis,H Jagadish (2012). Challenges and opportunities with big data. Proc. VLDB Endow, 5, 2032-2033.
8. Wang Wang Shan,Qin Hui-Ju,Xiong-Pai,Xuan Zhou (2011). Architecting Big Data: Challenges, Studies and Forecasts [J]. Chinese Journal of Computer, 34(10), 1741-1752.
9. Lu Weixing,Shou Yinbiao,Lianjun Shi (1996). Unknown Title. WSCC DISTURBANCE ON AUGUST 10
10. P Palermo (2004). The August 14, 2003 blackout and Its importance to China. The August 14, 2003 blackout and Its importance to China, 32, 2-6.
11. Li Cuiping,Wang Minfeng (2013). Excerpts from the Translation of Challenges and Opportunities with Big Data. J].e-Science Technology & Application, 4(1), 12-18.
12. W Dobbie,R Fryer (2011). Getting beneath the veil of effective schools: Evidence from New YorkCity. Getting beneath the veil of effective schools: Evidence from New YorkCity
13. H Jagadish,Johannes Gehrke,Alexandros Labrinidis,Yannis Papakonstantinou,Jignesh Patel,Raghu Ramakrishnan,Cyrus Shahabi (2014). Bigdata and its technical challenges. Communications of the ACM, 57(7), 86-94.
14. M Flood,H Jagadish,A Kyle (2011). Using Data for Systemic Financial Risk Management. Proceedings of The 5th biennial Conference on Innovative Data Systems Research, 144-147.
15. Y Genovese,S Prentice (2011). Pattern-based strategy: getting value from big data. J]. Gartner Special Report G, 214032.
16. Albert-Lszl Barabsi (2012). The network takeover. Nature Physics, 8(1), 14-16.
17. Alexandros Labrinidis,H Jagadish (2012). Challenges and opportunities with big data. Proceedings of the VLDB Endowment, 5(12), 2032-2033.
18. S Lohr (2012). How big data became so big. How big data became so big, 11.
19. A Gattiker,F H Gebara,H Hofstee,J Hayes,A Hylick (2013). Big Data text-oriented benchmark creation for Hadoop. IBM Journal of Research and Development, 57(3/4), 10:1-10:6.
20. Min Chen,Shiwen Mao,Yunhao Liu (2014). Big Data: A Survey. Mobile Networks and Applications, 19(2), 171-209.
21. Li Guojie,Cheng Xueqi (2012). Research Status and Scientific Thinking of Big Data. Bulletin of the Chinese Academy of Sciences, 27(6), 647-657.
22. S Yadagiri,Prashanth Thalluri (2011). Information Technology on Surge: Information Literacy on Demand. DESIDOC Journal of Library & Information Technology, 32(1), 64-69.
23. Jeffrey Cohen,Brian Dolan,Mark Dunlap,Joseph Hellerstein,Caleb Welton (2009). MAD skills. Proceedings of the VLDB Endowment, 2(2), 1481-1492.
24. Randal Bryant,Joan Digney (2007). Data-Intensive Supercomputing: The case for DISC. R, 10, 1-14.
25. John Boyle (2008). Biology must develop its own big-data systems. Nature, 499(7456), 7-7.
26. Wang Yuan-Zhuo,Jin Xiao-Long,Chen Xue-Qi (2013). Network Big Data: Present and Future [J]. Chinese Journal of Computer, 36(6), 1125-1138.
27. Zou Guowei,Cheng Jianbo (2013). The Application of Internet of Things Technology in Spatio-Temporal Big Data Aggregation System of Smart City. J]. POWER SYSTEM TECHNOLOGY, 4, 25-28.
28. Wang Qin Xiong-Pai,D Hui-Ju,Wang Xiao-Yong,Shan (2012). Big Data Analysis-Competition and Symbiosis of RDBMS and MapReduce. Journal of Software, 23(1), 32-45.
29. Tan Xiongpai,Wang Huiju,Li Furong (2013). New Landscape of Data Management Technologies [J]. Journal of Software, 24(2), 175-197.
30. Chen Hai-Ming,Li,Xie Kai-Bin (2013). A Comparative Study on Architectures and Implementation Methodologies of Internet of Things [J]. Chinese Journal of Computers, 36(1), 168-188.
31. E A Lee,S Seshia (2011). Introduction to embedded systems: A cyber-physical systems approach. Introduction to embedded systems: A cyber-physical systems approach
32. Ashish Thusoo,Joydeep Sarma,Namit Jain,Zheng Shao,Prasad Chakka,Ning Zhang,Suresh Antony,Hao Liu,Raghotham Murthy (2010). Hive - a petabyte scale data warehouse using Hadoop. 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010), 996-1005.
33. Azza Abouzied,Kamil Bajda-Pawlikowski,Jiewen Huang,Daniel Abadi,Avi Silberschatz (2010). HadoopDB in action. Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, 1111-1114.
34. Chen Songting (2010). Cheetah: A high performance, custom data warehouse on top of MapReduce. PVLDB, 3(2), 1459-1468.
35. Rakesh Agrawal,Ramakrishnan Srikant (2000). Privacy-preserving data mining. ACM SIGMOD Record, 29(2), 439-450.
36. Cynthia Dwork (2006). Differential Privacy. Lecture Notes in Computer Science, 1-12.
37. D Norman (2002). The Design of Everyday Things. The Design of Everyday Things
38. Christopher Olston,Benjamin Reed,Utkarsh Srivastava,Ravi Kumar,Andrew Tomkins (2008). Pig latin. Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 1099-1110.
39. Rob Pike,Sean Dorward,Robert Griesemer,Sean Quinlan (2005). Interpreting the Data: Parallel Analysis with Sawzall. Scientific Programming, 13(4), 277-298.
40. Ronnie Chaiken,Bob Jenkins,Per-Åke Larson,Bill Ramsey,Darren Shakib,Simon Weaver,Jingren Zhou (2008). SCOPE. Proceedings of the VLDB Endowment, 1(2), 1265-1276.
41. Michael Isard,Yuan Yu (2009). Distributed data-parallel computing using a high-level programming language. Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, 987-994.
42. L Fegaras,C Li,U Gupta (2011). XML query optimization in MapReduce. Proc. of WebDB 2011
43. K Morton,M Balazinska,D Grosstman (2010). Para Timer: A progress indicator for MapReduce DAGs. Proc. of SIGMOD 2010, 507-518.
44. Kristi Morton,Abram Friesen,Magdalena Balazinska,Dan Grossman (2010). Estimating the progress of MapReduce pipelines. 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010), 681-684.
45. Shi Huang Dachuan,Ibrabim Xuanhua,Shadi (2010). MR-scope: A real-time tracing tool for MapReduce. Proc. of HPDC 2010, 849-855.
46. Meng Xiaofeng,Ci Xiang (2013). Big Data Management: Concepts, Techniques and Challenges [J. Journal of Computer Research and Development, 50(1), 146-169.
47. Y Chen (2012). We dont know enough to make a big data benchmark suite-an academia-industry view. Proc. of WBDB
48. Chen Yanpei,A Ganapathi,R Griffith (2011). The case for evaluating MapReduce performance using workload suites. Proc. of MASCOTS 2011, 390-399.
49. Line Van Den Berg,Jérôme Euzenat (2012). Class ? En classe : jouer avec des classifications pour combiner mathématiques et informatique. Radix, 1(1)
50. Jiaqi Tan,Soila Kavulya,Rajeev Gandhi,Priya Narasimhan (2012). Light-weight black-box failure detection for distributed systems. Proceedings of the 2012 workshop on Management of big data systems, 13-18.
51. Jun-Ming Zhao,Wen-Shuan Wang,Xian Liu,You-Fu Chen (2014). Big Data Benchmark - Big DS. Lecture Notes in Computer Science, 49-57.
52. S Patil,M Polte,K Ren (2011). YCSB++: Benchmarking and performance debugging advance features in scalable table stores [C. Proc. of SoCC 2011
53. Yanpei Chen,Sara Alspaugh,Randy Katz (2012). Interactive Query Processing in Big Data Systems: A Cross Industry Study of MapReduce Workloads. PVLDB, 5(12), 1802-1813.

Download References

Funding

No external funding was declared for this work.

Conflict of Interest

The authors declare no conflict of interest.

Ethical Approval

No ethics committee approval was required for this article type.

Data Availability

Not applicable for this article.

How to Cite This Article

Khalid Imtiaz, M. Junaid Arshad. 2018. "A Review of Real World Big Data Processing Structure: Problems and Solutions". Global Journal of Computer Science and Technology - C: Software & Data Engineering GJCST-C Volume 18 (GJCST Volume 18 Issue C3).

More Citation Formats

Select Citation Style:

Download Citation

GJCST Volume 18 Issue C3

Explore Journals Explore Volume Read This Issue

Journal Specifications

Crossref Journal DOI 10.17406/gjcst

Print ISSN 0975-4350

e-ISSN 0975-4172

Keywords

Not Found

Classification

GJCST-C Classification H.3.m

Submission ReceivedDecember 12, 2017
Peer Review Double Blind
Handling Editor
Accepted December 31, 2017
Published July 27, 2018

Version of record

v1.2

Issue date

July 27, 2018

Language

English

Order Article Reprint

Experience in AR

Explore published articles in an immersive Augmented Reality environment. Our platform converts research papers into interactive 3D books, allowing readers to view and interact with content using AR and VR compatible devices.

View in VR

Read in 3D

Your published article is automatically converted into a realistic 3D book. Flip through pages and read research papers in a more engaging and interactive format.

View in 3D

Article Matrices

Total Score: 102

Country: Pakistan

Subject: Global Journal of Computer Science and Technology - C: Software & Data Engineering

Authors: Khalid Imtiaz, M. Junaid Arshad (PhD/Dr. count: 0)

View Count (all-time): 486

Total Views (Real + Logic): 1927

Total Downloads (simulated): 175

Publish Date: 2018 01, Mon

Monthly Totals (Real + Logic):

Month 1: 34 views
Month 2: 52 views
Month 3: 25 views
Month 4: 56 views
Month 5: 36 views
Month 6: 39 views
Month 7: 33 views
Month 8: 36 views
Month 9: 29 views
Month 10: 51 views
Month 11: 39 views
Month 12: 23 views
Month 13: 33 views
Month 14: 26 views
Month 15: 33 views
Month 16: 37 views
Month 17: 20 views
Month 18: 33 views
Month 19: 24 views
Month 20: 24 views
Month 21: 33 views
Month 22: 37 views
Month 23: 23 views
Month 24: 29 views
Month 25: 29 views
Month 26: 30 views
Month 27: 28 views
Month 28: 40 views
Month 29: 30 views
Month 30: 25 views
Month 31: 38 views
Month 32: 23 views
Month 33: 10 views
Month 34: 20 views
Month 35: 35 views
Month 36: 27 views
Month 37: 25 views
Month 38: 36 views
Month 39: 38 views
Month 40: 24 views
Month 41: 22 views
Month 42: 45 views
Month 43: 49 views
Month 44: 23 views
Month 45: 39 views
Month 46: 26 views
Month 47: 29 views
Month 48: 39 views
Month 49: 25 views
Month 50: 23 views
Month 51: 29 views
Month 52: 25 views
Month 53: 41 views
Month 54: 30 views
Month 55: 16 views
Month 56: 29 views
Month 57: 32 views
Month 58: 40 views
Month 59: 41 views
Month 60: 10 views
Month 61: 41 views
Month 62: 34 views
Month 63: 21 views
Month 64: 31 views
Month 65: 40 views
Month 66: 22 views
Month 67: 28 views
Month 68: 24 views
Month 69: 15 views
Month 70: 20 views
Month 71: 45 views
Month 72: 24 views
Month 73: 31 views
Month 74: 34 views
Month 75: 46 views
Month 76: 12 views
Month 77: 41 views
Month 78: 30 views
Month 79: 39 views
Month 80: 35 views
Month 81: 44 views
Month 82: 19 views
Month 83: 43 views
Month 84: 24 views
Month 85: 26 views
Month 86: 44 views
Month 87: 31 views
Month 88: 32 views
Month 89: 39 views
Month 90: 32 views
Month 91: 46 views
Month 92: 34 views
Month 93: 42 views
Month 94: 48 views
Month 95: 35 views
Month 96: 44 views
Month 97: 33 views
Month 98: 21 views
Month 99: 22 views
Month 100: 32 views
Month 101: 27 views
Month 102: 51 views
Month 103: 35 views
Month 104: 41 views
Month 105: 53 views
Month 106: 55 views

Total Views: 1.9K

Total Downloads: 175

All Trends