and Apache Spark are two Big Data frameworks but do different functions. Hadoop
is a distributed data structure while Spark is a data processing tool that
operates on distributed data collection, which means it does not do distributed
storage. In other words, Hadoop and Spark are two different systems with
most common Big Data tool is Apache
Hadoop, which is a framework for distributed processing and storage. It is
an Apache open source. It can scale form single computer system to thousands of
computer systems that have local storage and have computational path. Through
replication of data Hadoop become fault tolerant. Hadoop is composed of many
modules and that work together to create Hadoop framework. The primary Hadoop
modules are Hadoop Distributed File System(HDFS), for distributed storage and
second one is MapReduce which is a Data processing layer. The second one is Apache Spark, which is a data processing tool. Spark is also
an Apache open source. It can process data in HDFS, HBase, Cassandra and Hive.
Resilient Distributed Dataset(RDD) is used by Spark to generate fault
tolerance. Main components of Spark are Spark Core, Spark SQL, Spark Streaming,
Mlib and Graph, which all are data processing tools and do not support any of
distributed computer capabilities.
considering the similarities Hadoop and Spark are similar in some ways. Both
are used for Big Data processing.
Another similarity is in case of Speed.
When compared to Hadoop MapReduce Spark is 100X faster in terms of reading from
memory and 10X faster in terms of reading from disk. Hadoop MapReduce is
preferred when goes for batch mode operations. Next similarity is in the case
of Fault tolerance. Both can run
independently from each other. It doesn’t have any of Distributed storage and
can access diverse data sources that of HBase, S3, Cassandra or that of HDFS.
is some advantage for Spark over Hadoop. First one is in case of speed. It is a
lightning fast computational tool. Because of reducing the number of disk write
operation it can speed up the computational power and make it real time
possible. Second advantage is Ease to program. Spark provides tons of high
level operators with RDD so that we can apply any of programming models and
methods on top of it. The next one is Real time analysis. It can process data
that are coming in real time that is in terms of rate of millions of events per
second. The next one is Latency. Spark provides low latency computing.
provides an application framework to write Big Data applications. However, it
still needs to run on a storage system or on a NoSQL system and Hadoop provides
an efficient distributed storage system HDFS.