Geek asks and answers

Moving a file from local to HDFS

I have a.txt file in my local, I want to move this file to hadoop file system (HDFS) as shown below.

hadoop fs -put a.txt / user / hive / warehouse / sample_db / sample_table /

What happens in the background when a.txt file moves from local to Hadoop location ??

+3
source share
2 answers

In the background, the source file is split into HDFS blocks, the size of which is customizable (usually 128 MB, default 64 MB). For fault tolerance, each block is automatically replicated by HDFS. By default, three copies of each block are written to three different DataNodes. The replication rate is user-configurable (three by default). DataNodes are servers that are physical machines or virtual machines / cloud instances. DataNodes form a Hadoop cluster in which you write your data and on which you run MapReduce / Hive / Pig / Impala / Mahout / etc. programs.

DataNodes are Hadoop cluster workers, NameNodes are wizards.

When the file is to be written to HDFS, the client writing the file receives from the NameNode a list of DataNodes that can host replicas of the first block of the file.

, DataNodes. DataNode DataNode ( ) DataNode. , , DataNodes , . DataNode , DataNode . , .

DataNodes , , DataNodes NameNode . , HDFS. , .

: Hadoop: .

+5

Hadoop fs -put hasoop, HDFS.

Hadoop fs -put, hasoop datanode , hadoop.

hadoop -D, hadoop .

+1

More articles:


All Articles