zoom_in Click to enlarge

Apache Hadoop

▲ ▼

language

description Apache Hadoop Overview

Apache Hadoop is the foundational framework that launched the big data era. It provides a distributed file system (HDFS) and a processing model (MapReduce) that allow for the storage and processing of massive datasets across clusters of commodity hardware.

While modern cloud-native tools have largely replaced MapReduce and HDFS for most use cases, Hadoop remains a significant part of the big data landscape, particularly in on-premises environments. It serves as the base for many other big data tools, including Hive, HBase, and Spark, which have evolved to provide better performance and usability.

help Apache Hadoop FAQ

What do HDFS, MapReduce, and YARN each do in Apache Hadoop?

HDFS stores files across multiple machines, MapReduce processes those files in distributed batch jobs, and YARN manages cluster resources and scheduling. Together, these components form the classic Apache Hadoop platform.

How does Hadoop store files across cluster machines?

HDFS splits large files into blocks and stores those blocks on DataNodes while the NameNode tracks their locations. It also keeps replicated copies so a failed machine does not automatically make the whole file unavailable.

Can Apache Hadoop process a dataset larger than one server?

Yes, Hadoop is designed to spread storage and computation across clusters of commodity hardware. HDFS distributes the data, while MapReduce sends processing tasks to machines that hold the relevant blocks.

How does Hadoop MapReduce differ from Apache Spark?

MapReduce is a disk-oriented batch model that writes intermediate results between processing stages. Apache Spark can keep working data in memory for iterative jobs and can still run on a Hadoop YARN cluster.

swap_horiz

Looking for Apache Hadoop alternatives? Compare top competitors ranked & scored

arrow_forward