Partner Article

How to setup a single node Hadoop cluster?

In this post, Hadoop experts are going to describe how to develop and configure single node hadoop cluster setup installation to swiftly perform easy operations with Hadoop MapReduce and Hadoop Distributed File System (HDFS).

Things required for setting up the hadoop cluster:

Supported platforms

• Linux/GNU is supported platform that is used for development and production purposes. Hadoop has been shown on Linux/GNU clusters using 2000 nodes.
• You can also use Windows platform to set up Hadoop. In this post, we have used Linux.

Software requirement for Linux-

• You need to install java. Experts recommend HadoopJavaVersions for this setup.
• You must install ssh and run sshd in order to use Hadoop scripts, which help in managing hadoop daemons.

In case your cluster doesn’t have the required software, you have to install it for further operation.

How to start hadoop cluster?

You must unpack the downloaded hadoop distribution. You need to edit the file in the distribution – etc/hadoop/hadoop-env.sh to define few parameters. You can use the below code-

# set to the root of your Java Installation export Java_HOME=/usr/java/latest

# Assuming your installation directory is / usr/local/hadoop export HADOOP_PREFIX=/usr/local/hadoop

You can use this code that displays the documentation usage for the hadoop script:

$ bin/hadoop

So, you can start your hadoop cluster in one of the three following supported modes –

• Local mode
• Pseudo-distributed mode
• Fully distributed mode

Standalone or local operation

Hadoop is configured (by default) as a single java process to run in a non-distributed mode. User can use it for debugging. The below example will let you show how to copy the unpacked conf directory in order to use it as input and then determine and show every match of provided regular expression.

$ mkdir input $ cp etc/hadoop/*.xml input $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output ‘dfs[a-z]+’ $ cat output/*

Pseudo-distributed operation

Hadoop cluster can be run on single-node in pseudo-distributed mode.

For configuration, use the following code-

• etc/hadoop/core-site.xml:

fs.defaultFS

hdfs://localhost:9000

• etc/hadoop/hdfs-site.xml:

dfs.replication

1

Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:

$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen – t dsa -P ’ ’ -f ~/.ssh/id_dsa

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Execution process

1. Format the filesystem
2. Start DataNode Daemon and NameNode daemon
3. Search the web interface for the NameNode
4. Make the HDFS directories needed for execution of MapReduce jobs
5. Now copy the input files inside the distributed filesystem
6. Run it
7. Examine the output files
8. Once done, stop the daemons

This is how hadoop cluster setup can be executed by big data hadoop developers or expert Hadoop professional.

This was posted in Bdaily's Members' News section by Samual Alister .