emmm……

0x00.前言

0x01.引用

1.0 教程环境和说明

软硬件环境

  • CentOS 7.2 64位
  • OpenJDK-1.7
  • Hadoop-2.7.1

1.1 关于本教程的说明

云实验室云主机自动使用root账户登录系统,因此本教程中所有的操作都是以root用户来执行的。若要在自己的云主机上进行本教程的实验,为了系统安全,建议新建一个账户登录后再进行后续操作。

2.0 安装SSH客户端

2.1 安装SSH

安装SSH
sudo yum install openssh-clients openssh-server
安装完成后,可以使用下面命令进行测试:
ssh localhost
输入root账户的密码,如果可以正常登录,则说明SSH安装没有问题。测试正常后使用exit命令退出ssh

3.0 安装JAVA环境

3.1 安装JDK

使用yum来安装1.7版本OpenJDK
sudo yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel
安装完成后,输入javajavac命令,如果能输出对应的命令帮助,则表明jdk已正确安装。

3.2 配置JAVA环境变量

执行命令:
编辑~/.bashrc,在结尾追加:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk
保存文件后执行下面命令使JAVA_HOME环境变量生效:
source ~/.bashrc
为了检测系统中JAVA环境是否已经正确配置并生效,可以分别执行下面命令:
java -version
$JAVA_HOME/bin/java -version
若两条命令输出的结果一致,且都为我们前面安装的openjdk-1.7.0的版本,则表明JDK环境已经正确安装并配置。

1
[root@VM_36_253_centos ~]# java -version
2
java version "1.7.0_141"
3
OpenJDK Runtime Environment (rhel-2.6.10.1.el7_3-x86_64 u141-b02)
4
OpenJDK 64-Bit Server VM (build 24.141-b02, mixed mode)
5
[root@VM_36_253_centos ~]# $JAVA_HOME/bin/java -version
6
java version "1.7.0_141"
7
OpenJDK Runtime Environment (rhel-2.6.10.1.el7_3-x86_64 u141-b02)
8
OpenJDK 64-Bit Server VM (build 24.141-b02, mixed mode)

4.0 安装Hadoop

4.1 下载Hadoop

本教程使用hadoop-2.7.1版本,使用wget工具在线下载:
wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

4.2 安装Hadoop

Hadoop安装到/usr/local目录下:
tar -zxf hadoop-2.7.1.tar.gz -C /usr/local
对安装的目录进行重命名,便于后续操作方便:
cd /usr/local
mv ./hadoop-2.7.1/ ./hadoop
检查Hadoop是否已经正确安装:
/usr/local/hadoop/bin/hadoop version
如果成功输出hadoop的版本信息,表明hadoop已经成功安装。

1
[root@VM_36_253_centos local]# /usr/local/hadoop/bin/hadoop version
2
Hadoop 2.7.1
3
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
4
Compiled by jenkins on 2015-06-29T06:04Z
5
Compiled with protoc 2.5.0
6
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
7
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

5.0 Hadoop伪分布式环境配置

Hadoop伪分布式模式使用多个守护线程模拟分布的伪分布运行模式。

5.1 设置Hadoop的环境变量

编辑~/.bashrc,在结尾追加如下内容:

1
export HADOOP_HOME=/usr/local/hadoop
2
export HADOOP_INSTALL=$HADOOP_HOME
3
export HADOOP_MAPRED_HOME=$HADOOP_HOME
4
export HADOOP_COMMON_HOME=$HADOOP_HOME
5
export HADOOP_HDFS_HOME=$HADOOP_HOME
6
export YARN_HOME=$HADOOP_HOME
7
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
8
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

使Hadoop环境变量配置生效:
source ~/.bashrc

5.2 修改 Hadoop 的配置文件

Hadoop的配置文件位于安装目录的/etc/hadoop目录下,在本教程中即位于/url/local/hadoop/etc/hadoop目录下,需要修改的配置文件为如下两个:
/usr/local/hadoop/etc/hadoop/core-site.xml
/usr/local/hadoop/etc/hadoop/hdfs-site.xml
编辑core-site.xml,修改<configuration></configuration>节点的内容为如下所示:

1
<configuration>
2
    <property>
3
        <name>hadoop.tmp.dir</name>
4
        <value>file:/usr/local/hadoop/tmp</value>
5
        <description>location to store temporary files</description>
6
    </property>
7
    <property>
8
        <name>fs.defaultFS</name>
9
        <value>hdfs://localhost:9000</value>
10
    </property>
11
</configuration>

同理,编辑hdfs-site.xml,修改<configuration></configuration>节点的内容为如下所示:

1
<configuration>
2
    <property>
3
        <name>dfs.replication</name>
4
        <value>1</value>
5
    </property>
6
    <property>
7
        <name>dfs.namenode.name.dir</name>
8
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
9
    </property>
10
    <property>
11
        <name>dfs.datanode.data.dir</name>
12
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
13
    </property>
14
</configuration>

5.3 格式化NameNode

格式化NameNode:
/usr/local/hadoop/bin/hdfs namenode -format
在输出信息中看到如下信息,则表示格式化成功:

1
Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
2
Exiting with status 0

5.4 启动NameNodeDataNode守护进程

启动NameNodeDataNode进程:
/usr/local/hadoop/sbin/start-dfs.sh
执行过程中会提示输入用户密码,输入root用户密码即可。另外,启动时ssh会显示警告提示是否继续连接,输入yes即可。

实测启动时需输入三次密码

检查NameNodeDataNode是否正常启动:
jps
如果NameNodeDataNode已经正常启动,会显示NameNodeDataNodeSecondaryNameNode的进程信息:

1
[hadoop@VM_80_152_centos ~]$ jps
2
3689 SecondaryNameNode
3
3520 DataNode
4
3800 Jps
5
3393 NameNode
1
[root@VM_36_253_centos local]# /usr/local/hadoop/sbin/start-dfs.sh
2
17/08/22 16:11:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for y
3
our platform... using builtin-java classes where applicable
4
Starting namenodes on [localhost]
5
root@localhost's password:localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-VM_
6
36_253_centos.outroot@localhost's password:
7
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-VM_36_253_centos.out
8
Starting secondary namenodes [0.0.0.0]
9
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
10
ECDSA key fingerprint is 22:49:b2:5c:7c:8f:73:56:89:29:8a:bd:56:49:74:66.
11
Are you sure you want to continue connecting (yes/no)? yes
12
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
13
root@0.0.0.0's password:
14
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-VM_36_253_centos.out
15
17/08/22 16:11:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for y
16
our platform... using builtin-java classes where applicable
17
[root@VM_36_253_centos local]# jps
18
3355 SecondaryNameNode
19
3472 Jps
20
3051 NameNode
21
3191 DataNode

6.0 运行Hadoop伪分布式实例

Hadoop自带了丰富的例子,包括wordcountgrepsort 等。下面我们将以grep例子为教程,输入一批文件,从中筛选出符合正则表达式dfs[a-z.]+的单词并统计出现的次数。

6.1 查看Hadoop自带的例子

Hadoop附带了丰富的例子, 执行下面命令可以查看:
cd /usr/local/hadoop
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar

1
An example program must be given as the first argument.
2
Valid program names are:
3
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the
4
 input files.
5
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram o
6
f the words in the input files.
7
  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of
8
Pi.
9
  dbcount: An example job that count the pageview counts from a database.
10
  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
11
  grep: A map/reduce program that counts the matches of a regex in the input.
12
  join: A job that effects a join over sorted, equally partitioned datasets
13
  multifilewc: A job that counts words from several files.
14
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
15
  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
16
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
17
  randomwriter: A map/reduce program that writes 10GB of random data per node.
18
  secondarysort: An example defining a secondary sort to the reduce.
19
  sort: A map/reduce program that sorts the data written by the random writer.
20
  sudoku: A sudoku solver.
21
  teragen: Generate data for the terasort
22
  terasort: Run the terasort
23
  teravalidate: Checking results of terasort
24
  wordcount: A map/reduce program that counts the words in the input files.
25
  wordmean: A map/reduce program that counts the average length of the words in the input files.
26
  wordmedian: A map/reduce program that counts the median length of the words in the input files.
27
  wordstandarddeviation: A map/reduce program that counts the standard deviation of thelength of the words in the input files.

6.2 在 HDFS 中创建用户目录

在 HDFS 中创建用户目录 hadoop:
/usr/local/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop

1
[root@VM_36_253_centos hadoop]# /usr/local/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop
2
17/08/22 16:15:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
[root@VM_36_253_centos hadoop]# /usr/local/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop
4
17/08/22 16:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

这就很尴尬了……

6.3 准备实验数据

本教程中,我们将以Hadoop所有的xml配置文件作为输入数据来完成实验。执行下面命令在HDFS中新建一个input文件夹并将 hadoop配置文件上传到该文件夹下:

1
cd /usr/local/hadoop
2
./bin/hdfs dfs -mkdir /user/hadoop/input
3
./bin/hdfs dfs -put ./etc/hadoop/*.xml /user/hadoop/input

使用下面命令可以查看刚刚上传到HDFS的文件:
/usr/local/hadoop/bin/hdfs dfs -ls /user/hadoop/input

emmm……翻车了

1
[root@VM_36_253_centos hadoop]# ./bin/hdfs dfs -mkdir /user/hadoop/input
2
17/08/22 16:18:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
[root@VM_36_253_centos hadoop]# ./bin/hdfs dfs -put ./etc/hadoop/*.xml /user/hadoop/input
4
17/08/22 16:18:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

6.4 运行实验

运行实验:
cd /usr/local/hadoop
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep /user/hadoop/input /user/hadoop/output 'dfs[a-z.]+'
上述命令以HDFS文件系统中的input为输入数据来运行Hadoop自带的grep程序,提取其中符合正则表达式dfs[a-z.]+的数据并进行次数统计,将结果输出到HDFS文件系统的output文件夹下。

提醒,下方高能(误)版面较长……

1
[root@VM_36_253_centos hadoop]# cd /usr/local/hadoop
2
[root@VM_36_253_centos hadoop]# ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep /user/hadoop/input /user/hadoop/output 'dfs[a-z.]+'
3
17/08/22 16:20:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
4
17/08/22 16:20:39 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
5
17/08/22 16:20:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
6
17/08/22 16:20:39 INFO input.FileInputFormat: Total input paths to process : 8
7
17/08/22 16:20:39 INFO mapreduce.JobSubmitter: number of splits:8
8
17/08/22 16:20:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local598548963_0001
9
17/08/22 16:20:41 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
10
17/08/22 16:20:41 INFO mapreduce.Job: Running job: job_local598548963_0001
11
17/08/22 16:20:41 INFO mapred.LocalJobRunner: OutputCommitter set in config null
12
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
13
17/08/22 16:20:41 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Waiting for map tasks
15
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000000_0
16
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17
17/08/22 16:20:41 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
18
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-policy.xml:0+9683
19
17/08/22 16:20:41 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
20
17/08/22 16:20:41 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
21
17/08/22 16:20:41 INFO mapred.MapTask: soft limit at 83886080
22
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
23
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
24
17/08/22 16:20:41 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
25
17/08/22 16:20:41 INFO mapred.LocalJobRunner:
26
17/08/22 16:20:41 INFO mapred.MapTask: Starting flush of map output
27
17/08/22 16:20:41 INFO mapred.MapTask: Spilling map output
28
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
29
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
30
17/08/22 16:20:41 INFO mapred.MapTask: Finished spill 0
31
17/08/22 16:20:41 INFO mapred.Task: Task:attempt_local598548963_0001_m_000000_0 is done. And is in the process of committing
32
17/08/22 16:20:41 INFO mapred.LocalJobRunner: map
33
17/08/22 16:20:41 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000000_0' done.
34
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000000_0
35
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000001_0
36
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
37
17/08/22 16:20:41 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
38
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-site.xml:0+5511
39
17/08/22 16:20:41 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
40
17/08/22 16:20:41 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
41
17/08/22 16:20:41 INFO mapred.MapTask: soft limit at 83886080
42
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
43
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
44
17/08/22 16:20:41 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
45
17/08/22 16:20:41 INFO mapred.LocalJobRunner:
46
17/08/22 16:20:41 INFO mapred.MapTask: Starting flush of map output
47
17/08/22 16:20:41 INFO mapred.Task: Task:attempt_local598548963_0001_m_000001_0 is done. And is in the process of committing
48
17/08/22 16:20:41 INFO mapred.LocalJobRunner: map
49
17/08/22 16:20:41 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000001_0' done.
50
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000001_0
51
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000002_0
52
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
53
17/08/22 16:20:41 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
54
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/capacity-scheduler.xml:0+4436
55
17/08/22 16:20:41 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
56
17/08/22 16:20:41 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
57
17/08/22 16:20:41 INFO mapred.MapTask: soft limit at 83886080
58
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
59
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
60
17/08/22 16:20:41 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
61
17/08/22 16:20:41 INFO mapred.LocalJobRunner:
62
17/08/22 16:20:41 INFO mapred.MapTask: Starting flush of map output
63
17/08/22 16:20:41 INFO mapred.Task: Task:attempt_local598548963_0001_m_000002_0 is done. And is in the process of committing
64
17/08/22 16:20:41 INFO mapred.LocalJobRunner: map
65
17/08/22 16:20:41 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000002_0' done.
66
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000002_0
67
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000003_0
68
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
69
17/08/22 16:20:41 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
70
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-acls.xml:0+3518
71
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
72
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
73
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
74
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
75
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
76
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
77
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
78
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
79
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000003_0 is done. And is in the process of committing
80
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
81
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000003_0' done.
82
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000003_0
83
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000004_0
84
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
85
17/08/22 16:20:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
86
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hdfs-site.xml:0+1133
87
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
88
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
89
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
90
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
91
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
92
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
93
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
94
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
95
17/08/22 16:20:42 INFO mapred.MapTask: Spilling map output
96
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufend = 84; bufvoid = 104857600
97
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
98
17/08/22 16:20:42 INFO mapred.MapTask: Finished spill 0
99
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000004_0 is done. And is in the process of committing
100
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
101
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000004_0' done.
102
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000004_0
103
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000005_0
104
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
105
17/08/22 16:20:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
106
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/core-site.xml:0+1070
107
17/08/22 16:20:42 INFO mapreduce.Job: Job job_local598548963_0001 running in uber mode : false
108
17/08/22 16:20:42 INFO mapreduce.Job:  map 100% reduce 0%
109
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
110
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
111
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
112
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
113
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
114
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
115
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
116
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
117
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000005_0 is done. And is in the process of committing
118
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
119
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000005_0' done.
120
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000005_0
121
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000006_0
122
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
123
17/08/22 16:20:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
124
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/yarn-site.xml:0+690
125
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
126
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
127
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
128
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
129
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
130
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
131
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
132
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
133
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000006_0 is done. And is in the process of committing
134
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
135
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000006_0' done.
136
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000006_0
137
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000007_0
138
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
139
17/08/22 16:20:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
140
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-site.xml:0+620
141
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
142
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
143
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
144
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
145
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
146
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
147
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
148
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
149
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000007_0 is done. And is in the process of committing
150
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
151
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000007_0' done.
152
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000007_0
153
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map task executor complete.
154
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Waiting for reduce tasks
155
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_r_000000_0
156
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
157
17/08/22 16:20:42 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
158
17/08/22 16:20:42 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4d998798
159
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
160
17/08/22 16:20:42 INFO reduce.EventFetcher: attempt_local598548963_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
161
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
162
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000001_0
163
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
164
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
165
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000002_0
166
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
167
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
168
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000005_0
169
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
170
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
171
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000003_0
172
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
173
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
174
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000006_0
175
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
176
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
177
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local598548963_0001_m_000000_0
178
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->31
179
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
180
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000007_0
181
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
182
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000004_0 decomp: 92 len: 96 to MEMORY
183
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 92 bytes from map-output for attempt_local598548963_0001_m_000004_0
184
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 92, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->125
185
17/08/22 16:20:42 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
186
17/08/22 16:20:42 INFO mapred.LocalJobRunner: 8 / 8 copied.
187
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: finalMerge called with 8 in-memory map-outputs and 0 on-disk map-outputs
188
17/08/22 16:20:42 INFO mapred.Merger: Merging 8 sorted segments
189
17/08/22 16:20:42 INFO mapred.Merger: Down to the last merge-pass, with 2 segments leftof total size: 78 bytes
190
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: Merged 8 segments, 125 bytes to disk tosatisfy reduce memory limit
191
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
192
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memoryinto reduce
193
17/08/22 16:20:42 INFO mapred.Merger: Merging 1 sorted segments
194
17/08/22 16:20:42 INFO mapred.Merger: Down to the last merge-pass, with 1 segments leftof total size: 87 bytes
195
17/08/22 16:20:42 INFO mapred.LocalJobRunner: 8 / 8 copied.
196
17/08/22 16:20:42 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
197
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_r_000000_0 is done. And is in the process of committing
198
17/08/22 16:20:42 INFO mapred.LocalJobRunner: 8 / 8 copied.
199
17/08/22 16:20:42 INFO mapred.Task: Task attempt_local598548963_0001_r_000000_0 is allowed to commit now
200
17/08/22 16:20:42 INFO output.FileOutputCommitter: Saved output of task 'attempt_local598548963_0001_r_000000_0' to hdfs://localhost:9000/user/root/grep-temp-1006666638/_temporary/0/task_local598548963_0001_r_000000
201
17/08/22 16:20:42 INFO mapred.LocalJobRunner: reduce > reduce
202
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_r_000000_0' done.
203
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_r_000000_0
204
17/08/22 16:20:42 INFO mapred.LocalJobRunner: reduce task executor complete.
205
17/08/22 16:20:43 INFO mapreduce.Job:  map 100% reduce 100%
206
17/08/22 16:20:43 INFO mapreduce.Job: Job job_local598548963_0001 completed successfully
207
17/08/22 16:20:43 INFO mapreduce.Job: Counters: 35
208
        File System Counters
209
                FILE: Number of bytes read=2501359
210
                FILE: Number of bytes written=4984808
211
                FILE: Number of read operations=0
212
                FILE: Number of large read operations=0
213
                FILE: Number of write operations=0
214
                HDFS: Number of bytes read=196650
215
                HDFS: Number of bytes written=219
216
                HDFS: Number of read operations=118
217
                HDFS: Number of large read operations=0
218
                HDFS: Number of write operations=11
219
        Map-Reduce Framework
220
                Map input records=765
221
                Map output records=4
222
                Map output bytes=101
223
                Map output materialized bytes=157
224
                Input split bytes=957
225
                Combine input records=4
226
                Combine output records=4
227
                Reduce input groups=4
228
                Reduce shuffle bytes=157
229
                Reduce input records=4
230
                Reduce output records=4
231
                Spilled Records=8
232
                Shuffled Maps =8
233
                Failed Shuffles=0
234
                Merged Map outputs=8
235
                GC time elapsed (ms)=318
236
                Total committed heap usage (bytes)=1380876288
237
        Shuffle Errors
238
                BAD_ID=0
239
                CONNECTION=0
240
                IO_ERROR=0
241
                WRONG_LENGTH=0
242
                WRONG_MAP=0
243
                WRONG_REDUCE=0
244
        File Input Format Counters
245
                Bytes Read=26661
246
        File Output Format Counters
247
                Bytes Written=219
248
17/08/22 16:20:43 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
249
17/08/22 16:20:43 INFO input.FileInputFormat: Total input paths to process : 1
250
17/08/22 16:20:43 INFO mapreduce.JobSubmitter: number of splits:1
251
17/08/22 16:20:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local992066578_0002
252
17/08/22 16:20:43 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
253
17/08/22 16:20:43 INFO mapreduce.Job: Running job: job_local992066578_0002
254
17/08/22 16:20:43 INFO mapred.LocalJobRunner: OutputCommitter set in config null
255
17/08/22 16:20:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
256
17/08/22 16:20:43 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
257
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Waiting for map tasks
258
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Starting task: attempt_local992066578_0002_m_000000_0
259
17/08/22 16:20:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
260
17/08/22 16:20:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
261
17/08/22 16:20:43 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/root/grep-temp-1006666638/part-r-00000:0+219
262
17/08/22 16:20:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
263
17/08/22 16:20:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
264
17/08/22 16:20:43 INFO mapred.MapTask: soft limit at 83886080
265
17/08/22 16:20:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
266
17/08/22 16:20:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
267
17/08/22 16:20:43 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
268
17/08/22 16:20:43 INFO mapred.LocalJobRunner:
269
17/08/22 16:20:43 INFO mapred.MapTask: Starting flush of map output
270
17/08/22 16:20:43 INFO mapred.MapTask: Spilling map output
271
17/08/22 16:20:43 INFO mapred.MapTask: bufstart = 0; bufend = 101; bufvoid = 104857600
272
17/08/22 16:20:43 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
273
17/08/22 16:20:43 INFO mapred.MapTask: Finished spill 0
274
17/08/22 16:20:43 INFO mapred.Task: Task:attempt_local992066578_0002_m_000000_0 is done. And is in the process of committing
275
17/08/22 16:20:43 INFO mapred.LocalJobRunner: map
276
17/08/22 16:20:43 INFO mapred.Task: Task 'attempt_local992066578_0002_m_000000_0' done.
277
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local992066578_0002_m_000000_0
278
17/08/22 16:20:43 INFO mapred.LocalJobRunner: map task executor complete.
279
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Waiting for reduce tasks
280
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Starting task: attempt_local992066578_0002_r_000000_0
281
17/08/22 16:20:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
282
17/08/22 16:20:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
283
17/08/22 16:20:43 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5eeb4365
284
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
285
17/08/22 16:20:43 INFO reduce.EventFetcher: attempt_local992066578_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
286
17/08/22 16:20:43 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local992066578_0002_m_000000_0 decomp: 111 len: 115 to MEMORY
287
17/08/22 16:20:43 INFO reduce.InMemoryMapOutput: Read 111 bytes from map-output for attempt_local992066578_0002_m_000000_0
288
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 111, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->111
289
17/08/22 16:20:43 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
290
17/08/22 16:20:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
291
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
292
17/08/22 16:20:43 INFO mapred.Merger: Merging 1 sorted segments
293
17/08/22 16:20:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments leftof total size: 101 bytes
294
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: Merged 1 segments, 111 bytes to disk tosatisfy reduce memory limit
295
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
296
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memoryinto reduce
297
17/08/22 16:20:43 INFO mapred.Merger: Merging 1 sorted segments
298
17/08/22 16:20:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments leftof total size: 101 bytes
299
17/08/22 16:20:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
300
17/08/22 16:20:43 INFO mapred.Task: Task:attempt_local992066578_0002_r_000000_0 is done. And is in the process of committing
301
17/08/22 16:20:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
302
17/08/22 16:20:43 INFO mapred.Task: Task attempt_local992066578_0002_r_000000_0 is allowed to commit now
303
17/08/22 16:20:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local992066578_0002_r_000000_0' to hdfs://localhost:9000/user/hadoop/output/_temporary/0/task_local992066578_0002_r_000000
304
17/08/22 16:20:43 INFO mapred.LocalJobRunner: reduce > reduce
305
17/08/22 16:20:43 INFO mapred.Task: Task 'attempt_local992066578_0002_r_000000_0' done.
306
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local992066578_0002_r_000000_0
307
17/08/22 16:20:43 INFO mapred.LocalJobRunner: reduce task executor complete.
308
17/08/22 16:20:44 INFO mapreduce.Job: Job job_local992066578_0002 running in uber mode : false
309
17/08/22 16:20:44 INFO mapreduce.Job:  map 100% reduce 100%
310
17/08/22 16:20:44 INFO mapreduce.Job: Job job_local992066578_0002 completed successfully
311
17/08/22 16:20:44 INFO mapreduce.Job: Counters: 35
312
        File System Counters
313
                FILE: Number of bytes read=1108352
314
                FILE: Number of bytes written=2210539
315
                FILE: Number of read operations=0
316
                FILE: Number of large read operations=0
317
                FILE: Number of write operations=0
318
                HDFS: Number of bytes read=53760
319
                HDFS: Number of bytes written=515
320
                HDFS: Number of read operations=67
321
                HDFS: Number of large read operations=0
322
                HDFS: Number of write operations=16
323
        Map-Reduce Framework
324
                Map input records=4
325
                Map output records=4
326
                Map output bytes=101
327
                Map output materialized bytes=115
328
                Input split bytes=130
329
                Combine input records=0
330
                Combine output records=0
331
                Reduce input groups=1
332
                Reduce shuffle bytes=115
333
                Reduce input records=4
334
                Reduce output records=4
335
                Spilled Records=8
336
                Shuffled Maps =1
337
                Failed Shuffles=0
338
                Merged Map outputs=1
339
                GC time elapsed (ms)=42
340
                Total committed heap usage (bytes)=263462912
341
        Shuffle Errors
342
                BAD_ID=0
343
                CONNECTION=0
344
                IO_ERROR=0
345
                WRONG_LENGTH=0
346
                WRONG_MAP=0
347
                WRONG_REDUCE=0
348
        File Input Format Counters
349
                Bytes Read=219
350
        File Output Format Counters
351
                Bytes Written=77

6.5 查看运行结果

上述例子完成后的结果保存在HDFS中,通过下面命令查看结果:
/usr/local/hadoop/bin/hdfs dfs -cat /user/hadoop/output/*
如果运行成功,可以看到如下结果:

1
1       dfsadmin
2
1       dfs.replication
3
1       dfs.namenode.name.dir
4
1       dfs.datanode.data.dir

6.6 删除HDFS上的输出结果

删除HDFS中的结果目录:
/usr/local/hadoop/bin/hdfs dfs -rm -r /user/hadoop/output
运行Hadoop程序时,为了防止覆盖结果,程序指定的输出目录不能存在,否则会提示错误,因此在下次运行前需要先删除输出目录。

1
[root@VM_36_253_centos hadoop]# /usr/local/hadoop/bin/hdfs dfs -rm -r /user/hadoop/output
2
17/08/22 16:24:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
17/08/22 16:24:55 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
4
Deleted /user/hadoop/output

6.7 关闭Hadoop进程

关闭Hadoop进程:
/usr/local/hadoop/sbin/stop-dfs.sh
再起启动只需要执行下面命令:
/usr/local/hadoop/sbin/start-dfs.sh

0x02.后记

随手htop

emmm……
今天是怎么了?