Today I will tell you about the common tools for big data analysis and processing, I hope it can help you.
HPCC is a plan implemented by the United States to implement the information superhighway. The implementation of this plan will cost tens of billions of dollars. Its main goals are: to develop scalable computing systems and related software to support terabit-level network transmission performance, develop Gigabit network technology to expand research and educational institutions and network connectivity. The program consists of five components: High-Performance Computer Systems, National Research and Education Grid, Information Infrastructure and Applications, Advanced Software Technologies and Algorithms, and Fundamental Research and Human Resources.
Hadoop is a software framework capable of distributed processing of large amounts of data. But Hadoop does it in a reliable, efficient, and scalable way. Hadoop is reliable because it assumes that compute elements and storage will fail, so it maintains multiple copies of working data, ensuring that processing can be redistributed against failed nodes. Hadoop is efficient because it works in parallel, speeding up processing through parallel processing. Hadoop is also scalable, capable of processing petabytes of data. Also, Hadoop relies on community servers, so it’s cheap and anyone can use it. Hadoop is a distributed computing platform that can be easily architected and used by users. Users can easily develop and run applications that process massive amounts of data on Hadoop. Hadoop comes with a framework written in the Java language, so it is ideal to run on a Linux production platform. Applications on Hadoop can also be written in other languages, such as C++.
The main advantages are: high reliability, high efficiency, high scalability, and high fault tolerance.
Storm is free and open source software, a distributed, fault-tolerant real-time computing system. Storm can handle huge data streams very reliably and is used to process Hadoop batches of data. Storm is simple, supports many programming languages, and is a lot of fun to use. Storm is open sourced from Twitter, and other well-known application companies include Groupon, Taobao, Alipay, Alibaba, Le Element, Admaster and so on.
Storm has many application areas: real-time analytics, online machine learning, non-stop computing, distributed, RPCETL (i.e. data extraction, transformation, and loading), and more. Storm’s processing speed is amazing, Storm is scalable, fault-tolerant, and easy to set up and operate.
RapidMiner is the world’s leading data mining solution with advanced technology to a very large extent. It covers a wide range of data mining tasks, including various data arts, and can simplify the design and evaluation of data mining processes. It can be applied in many different application areas, including text mining, multimedia mining, feature design, data flow mining, integrated development methods and distributed data mining.
Main functions and advantages: free data mining technology and libraries; 100% Java code (which can run on the operating system); data mining process is simple, powerful and intuitive; internal XML ensures a standardized format to express and exchange data mining process; Automating large-scale processes with simple scripting languages; multi-level data views to ensure valid and transparent data; interactive prototyping of graphical user interfaces; command-line (batch mode) automated large-scale applications; Java API (application programming interface); Simple plug-in and promotion mechanism; powerful visualization engine, visual modeling of many cutting-edge high-dimensional data; more than 400 data mining operators supported.
The Pentaho BI platform is a process-centric, solution-oriented framework. Its purpose is to integrate a series of enterprise-level BI products, open source software, API and other components to facilitate the development of business intelligence applications. Its appearance makes a series of independent products oriented to business intelligence, such as Jfree, Quartz, etc., can be integrated together to form a complex and complete business intelligence solution.
ThinkMo CCNA Dump exam information exchange group：
CCNA/CCNP/CCIE telegram study group：https://t.me/ccie_ei_lab
ThinkMo CCNA 200-301 Tutorial VIP Exclusive：
The complete EVE_NG file, free learning PDF and PPT that can be used directly, as well as video explaining the technical points are all here!