emmm……

0x00.前言

0x01.引用

1.0 教程环境和说明

软硬件环境

  • CentOS 7.2 64位
  • OpenJDK-1.7
  • Hadoop-2.7.1

1.1 关于本教程的说明

云实验室云主机自动使用root账户登录系统,因此本教程中所有的操作都是以root用户来执行的。若要在自己的云主机上进行本教程的实验,为了系统安全,建议新建一个账户登录后再进行后续操作。

2.0 安装SSH客户端

2.1 安装SSH

安装SSH
sudo yum install openssh-clients openssh-server
安装完成后,可以使用下面命令进行测试:
ssh localhost
输入root账户的密码,如果可以正常登录,则说明SSH安装没有问题。测试正常后使用exit命令退出ssh

3.0 安装JAVA环境

3.1 安装JDK

使用yum来安装1.7版本OpenJDK
sudo yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel
安装完成后,输入javajavac命令,如果能输出对应的命令帮助,则表明jdk已正确安装。

3.2 配置JAVA环境变量

执行命令:
编辑~/.bashrc,在结尾追加:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk
保存文件后执行下面命令使JAVA_HOME环境变量生效:
source ~/.bashrc
为了检测系统中JAVA环境是否已经正确配置并生效,可以分别执行下面命令:
java -version
$JAVA_HOME/bin/java -version
若两条命令输出的结果一致,且都为我们前面安装的openjdk-1.7.0的版本,则表明JDK环境已经正确安装并配置。

1
2
3
4
5
6
7
8
[root@VM_36_253_centos ~]# java -version
java version "1.7.0_141"
OpenJDK Runtime Environment (rhel-2.6.10.1.el7_3-x86_64 u141-b02)
OpenJDK 64-Bit Server VM (build 24.141-b02, mixed mode)
[root@VM_36_253_centos ~]# $JAVA_HOME/bin/java -version
java version "1.7.0_141"
OpenJDK Runtime Environment (rhel-2.6.10.1.el7_3-x86_64 u141-b02)
OpenJDK 64-Bit Server VM (build 24.141-b02, mixed mode)

4.0 安装Hadoop

4.1 下载Hadoop

本教程使用hadoop-2.7.1版本,使用wget工具在线下载:
wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

4.2 安装Hadoop

Hadoop安装到/usr/local目录下:
tar -zxf hadoop-2.7.1.tar.gz -C /usr/local
对安装的目录进行重命名,便于后续操作方便:
cd /usr/local
mv ./hadoop-2.7.1/ ./hadoop
检查Hadoop是否已经正确安装:
/usr/local/hadoop/bin/hadoop version
如果成功输出hadoop的版本信息,表明hadoop已经成功安装。

1
2
3
4
5
6
7
[root@VM_36_253_centos local]# /usr/local/hadoop/bin/hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

5.0 Hadoop伪分布式环境配置

Hadoop伪分布式模式使用多个守护线程模拟分布的伪分布运行模式。

5.1 设置Hadoop的环境变量

编辑~/.bashrc,在结尾追加如下内容:

1
2
3
4
5
6
7
8
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

使Hadoop环境变量配置生效:
source ~/.bashrc

5.2 修改 Hadoop 的配置文件

Hadoop的配置文件位于安装目录的/etc/hadoop目录下,在本教程中即位于/url/local/hadoop/etc/hadoop目录下,需要修改的配置文件为如下两个:
/usr/local/hadoop/etc/hadoop/core-site.xml
/usr/local/hadoop/etc/hadoop/hdfs-site.xml
编辑core-site.xml,修改<configuration></configuration>节点的内容为如下所示:

1
2
3
4
5
6
7
8
9
10
11
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
<description>location to store temporary files</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

同理,编辑hdfs-site.xml,修改<configuration></configuration>节点的内容为如下所示:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/tmp/dfs/data</value>
</property>
</configuration>

5.3 格式化NameNode

格式化NameNode:
/usr/local/hadoop/bin/hdfs namenode -format
在输出信息中看到如下信息,则表示格式化成功:

1
2
Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.
Exiting with status 0

5.4 启动NameNodeDataNode守护进程

启动NameNodeDataNode进程:
/usr/local/hadoop/sbin/start-dfs.sh
执行过程中会提示输入用户密码,输入root用户密码即可。另外,启动时ssh会显示警告提示是否继续连接,输入yes即可。

实测启动时需输入三次密码

检查NameNodeDataNode是否正常启动:
jps
如果NameNodeDataNode已经正常启动,会显示NameNodeDataNodeSecondaryNameNode的进程信息:

1
2
3
4
5
[hadoop@VM_80_152_centos ~]$ jps
3689 SecondaryNameNode
3520 DataNode
3800 Jps
3393 NameNode

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@VM_36_253_centos local]# /usr/local/hadoop/sbin/start-dfs.sh
17/08/22 16:11:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for y
our platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
root@localhost's password:localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-VM_
36_253_centos.outroot@localhost's password:
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-VM_36_253_centos.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 22:49:b2:5c:7c:8f:73:56:89:29:8a:bd:56:49:74:66.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
root@0.0.0.0's password:
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-VM_36_253_centos.out
17/08/22 16:11:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for y
our platform... using builtin-java classes where applicable
[root@VM_36_253_centos local]# jps
3355 SecondaryNameNode
3472 Jps
3051 NameNode
3191 DataNode

6.0 运行Hadoop伪分布式实例

Hadoop自带了丰富的例子,包括wordcountgrepsort 等。下面我们将以grep例子为教程,输入一批文件,从中筛选出符合正则表达式dfs[a-z.]+的单词并统计出现的次数。

6.1 查看Hadoop自带的例子

Hadoop附带了丰富的例子, 执行下面命令可以查看:
cd /usr/local/hadoop
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the
input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram o
f the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of
Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of thelength of the words in the input files.

6.2 在 HDFS 中创建用户目录

在 HDFS 中创建用户目录 hadoop:
/usr/local/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop


1
2
3
4
[root@VM_36_253_centos hadoop]# /usr/local/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop
17/08/22 16:15:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@VM_36_253_centos hadoop]# /usr/local/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop
17/08/22 16:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

这就很尴尬了……

6.3 准备实验数据

本教程中,我们将以Hadoop所有的xml配置文件作为输入数据来完成实验。执行下面命令在HDFS中新建一个input文件夹并将 hadoop配置文件上传到该文件夹下:

1
2
3
cd /usr/local/hadoop
./bin/hdfs dfs -mkdir /user/hadoop/input
./bin/hdfs dfs -put ./etc/hadoop/*.xml /user/hadoop/input

使用下面命令可以查看刚刚上传到HDFS的文件:
/usr/local/hadoop/bin/hdfs dfs -ls /user/hadoop/input


emmm……翻车了
1
2
3
4
[root@VM_36_253_centos hadoop]# ./bin/hdfs dfs -mkdir /user/hadoop/input
17/08/22 16:18:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@VM_36_253_centos hadoop]# ./bin/hdfs dfs -put ./etc/hadoop/*.xml /user/hadoop/input
17/08/22 16:18:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

6.4 运行实验

运行实验:
cd /usr/local/hadoop
./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep /user/hadoop/input /user/hadoop/output 'dfs[a-z.]+'
上述命令以HDFS文件系统中的input为输入数据来运行Hadoop自带的grep程序,提取其中符合正则表达式dfs[a-z.]+的数据并进行次数统计,将结果输出到HDFS文件系统的output文件夹下。


提醒,下方高能(误)版面较长……
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
[root@VM_36_253_centos hadoop]# cd /usr/local/hadoop
[root@VM_36_253_centos hadoop]# ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep /user/hadoop/input /user/hadoop/output 'dfs[a-z.]+'
17/08/22 16:20:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/22 16:20:39 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/08/22 16:20:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/08/22 16:20:39 INFO input.FileInputFormat: Total input paths to process : 8
17/08/22 16:20:39 INFO mapreduce.JobSubmitter: number of splits:8
17/08/22 16:20:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local598548963_0001
17/08/22 16:20:41 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/08/22 16:20:41 INFO mapreduce.Job: Running job: job_local598548963_0001
17/08/22 16:20:41 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:41 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Waiting for map tasks
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000000_0
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:41 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-policy.xml:0+9683
17/08/22 16:20:41 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:41 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:41 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:41 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:41 INFO mapred.LocalJobRunner:
17/08/22 16:20:41 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:41 INFO mapred.MapTask: Spilling map output
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
17/08/22 16:20:41 INFO mapred.MapTask: Finished spill 0
17/08/22 16:20:41 INFO mapred.Task: Task:attempt_local598548963_0001_m_000000_0 is done. And is in the process of committing
17/08/22 16:20:41 INFO mapred.LocalJobRunner: map
17/08/22 16:20:41 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000000_0' done.
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000000_0
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000001_0
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:41 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-site.xml:0+5511
17/08/22 16:20:41 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:41 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:41 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:41 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:41 INFO mapred.LocalJobRunner:
17/08/22 16:20:41 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:41 INFO mapred.Task: Task:attempt_local598548963_0001_m_000001_0 is done. And is in the process of committing
17/08/22 16:20:41 INFO mapred.LocalJobRunner: map
17/08/22 16:20:41 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000001_0' done.
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000001_0
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000002_0
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:41 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/capacity-scheduler.xml:0+4436
17/08/22 16:20:41 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:41 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:41 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:41 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:41 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:41 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:41 INFO mapred.LocalJobRunner:
17/08/22 16:20:41 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:41 INFO mapred.Task: Task:attempt_local598548963_0001_m_000002_0 is done. And is in the process of committing
17/08/22 16:20:41 INFO mapred.LocalJobRunner: map
17/08/22 16:20:41 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000002_0' done.
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000002_0
17/08/22 16:20:41 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000003_0
17/08/22 16:20:41 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:41 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:41 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-acls.xml:0+3518
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000003_0 is done. And is in the process of committing
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000003_0' done.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000003_0
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000004_0
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hdfs-site.xml:0+1133
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:42 INFO mapred.MapTask: Spilling map output
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufend = 84; bufvoid = 104857600
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
17/08/22 16:20:42 INFO mapred.MapTask: Finished spill 0
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000004_0 is done. And is in the process of committing
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000004_0' done.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000004_0
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000005_0
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/core-site.xml:0+1070
17/08/22 16:20:42 INFO mapreduce.Job: Job job_local598548963_0001 running in uber mode : false
17/08/22 16:20:42 INFO mapreduce.Job: map 100% reduce 0%
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000005_0 is done. And is in the process of committing
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000005_0' done.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000005_0
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000006_0
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/yarn-site.xml:0+690
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000006_0 is done. And is in the process of committing
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000006_0' done.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000006_0
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_m_000007_0
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:42 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-site.xml:0+620
17/08/22 16:20:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:42 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:42 INFO mapred.LocalJobRunner:
17/08/22 16:20:42 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_m_000007_0 is done. And is in the process of committing
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_m_000007_0' done.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_m_000007_0
17/08/22 16:20:42 INFO mapred.LocalJobRunner: map task executor complete.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Starting task: attempt_local598548963_0001_r_000000_0
17/08/22 16:20:42 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:42 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4d998798
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/08/22 16:20:42 INFO reduce.EventFetcher: attempt_local598548963_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000001_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000002_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000005_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000003_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000006_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local598548963_0001_m_000000_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->31
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local598548963_0001_m_000007_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33
17/08/22 16:20:42 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local598548963_0001_m_000004_0 decomp: 92 len: 96 to MEMORY
17/08/22 16:20:42 INFO reduce.InMemoryMapOutput: Read 92 bytes from map-output for attempt_local598548963_0001_m_000004_0
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 92, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->125
17/08/22 16:20:42 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/08/22 16:20:42 INFO mapred.LocalJobRunner: 8 / 8 copied.
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: finalMerge called with 8 in-memory map-outputs and 0 on-disk map-outputs
17/08/22 16:20:42 INFO mapred.Merger: Merging 8 sorted segments
17/08/22 16:20:42 INFO mapred.Merger: Down to the last merge-pass, with 2 segments leftof total size: 78 bytes
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: Merged 8 segments, 125 bytes to disk tosatisfy reduce memory limit
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
17/08/22 16:20:42 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memoryinto reduce
17/08/22 16:20:42 INFO mapred.Merger: Merging 1 sorted segments
17/08/22 16:20:42 INFO mapred.Merger: Down to the last merge-pass, with 1 segments leftof total size: 87 bytes
17/08/22 16:20:42 INFO mapred.LocalJobRunner: 8 / 8 copied.
17/08/22 16:20:42 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
17/08/22 16:20:42 INFO mapred.Task: Task:attempt_local598548963_0001_r_000000_0 is done. And is in the process of committing
17/08/22 16:20:42 INFO mapred.LocalJobRunner: 8 / 8 copied.
17/08/22 16:20:42 INFO mapred.Task: Task attempt_local598548963_0001_r_000000_0 is allowed to commit now
17/08/22 16:20:42 INFO output.FileOutputCommitter: Saved output of task 'attempt_local598548963_0001_r_000000_0' to hdfs://localhost:9000/user/root/grep-temp-1006666638/_temporary/0/task_local598548963_0001_r_000000
17/08/22 16:20:42 INFO mapred.LocalJobRunner: reduce > reduce
17/08/22 16:20:42 INFO mapred.Task: Task 'attempt_local598548963_0001_r_000000_0' done.
17/08/22 16:20:42 INFO mapred.LocalJobRunner: Finishing task: attempt_local598548963_0001_r_000000_0
17/08/22 16:20:42 INFO mapred.LocalJobRunner: reduce task executor complete.
17/08/22 16:20:43 INFO mapreduce.Job: map 100% reduce 100%
17/08/22 16:20:43 INFO mapreduce.Job: Job job_local598548963_0001 completed successfully
17/08/22 16:20:43 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=2501359
FILE: Number of bytes written=4984808
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=196650
HDFS: Number of bytes written=219
HDFS: Number of read operations=118
HDFS: Number of large read operations=0
HDFS: Number of write operations=11
Map-Reduce Framework
Map input records=765
Map output records=4
Map output bytes=101
Map output materialized bytes=157
Input split bytes=957
Combine input records=4
Combine output records=4
Reduce input groups=4
Reduce shuffle bytes=157
Reduce input records=4
Reduce output records=4
Spilled Records=8
Shuffled Maps =8
Failed Shuffles=0
Merged Map outputs=8
GC time elapsed (ms)=318
Total committed heap usage (bytes)=1380876288
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=26661
File Output Format Counters
Bytes Written=219
17/08/22 16:20:43 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
17/08/22 16:20:43 INFO input.FileInputFormat: Total input paths to process : 1
17/08/22 16:20:43 INFO mapreduce.JobSubmitter: number of splits:1
17/08/22 16:20:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local992066578_0002
17/08/22 16:20:43 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/08/22 16:20:43 INFO mapreduce.Job: Running job: job_local992066578_0002
17/08/22 16:20:43 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/08/22 16:20:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:43 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Waiting for map tasks
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Starting task: attempt_local992066578_0002_m_000000_0
17/08/22 16:20:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:43 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:43 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/root/grep-temp-1006666638/part-r-00000:0+219
17/08/22 16:20:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/08/22 16:20:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/08/22 16:20:43 INFO mapred.MapTask: soft limit at 83886080
17/08/22 16:20:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/08/22 16:20:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/08/22 16:20:43 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/08/22 16:20:43 INFO mapred.LocalJobRunner:
17/08/22 16:20:43 INFO mapred.MapTask: Starting flush of map output
17/08/22 16:20:43 INFO mapred.MapTask: Spilling map output
17/08/22 16:20:43 INFO mapred.MapTask: bufstart = 0; bufend = 101; bufvoid = 104857600
17/08/22 16:20:43 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
17/08/22 16:20:43 INFO mapred.MapTask: Finished spill 0
17/08/22 16:20:43 INFO mapred.Task: Task:attempt_local992066578_0002_m_000000_0 is done. And is in the process of committing
17/08/22 16:20:43 INFO mapred.LocalJobRunner: map
17/08/22 16:20:43 INFO mapred.Task: Task 'attempt_local992066578_0002_m_000000_0' done.
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local992066578_0002_m_000000_0
17/08/22 16:20:43 INFO mapred.LocalJobRunner: map task executor complete.
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Starting task: attempt_local992066578_0002_r_000000_0
17/08/22 16:20:43 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/08/22 16:20:43 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
17/08/22 16:20:43 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5eeb4365
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/08/22 16:20:43 INFO reduce.EventFetcher: attempt_local992066578_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/08/22 16:20:43 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local992066578_0002_m_000000_0 decomp: 111 len: 115 to MEMORY
17/08/22 16:20:43 INFO reduce.InMemoryMapOutput: Read 111 bytes from map-output for attempt_local992066578_0002_m_000000_0
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 111, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->111
17/08/22 16:20:43 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/08/22 16:20:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
17/08/22 16:20:43 INFO mapred.Merger: Merging 1 sorted segments
17/08/22 16:20:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments leftof total size: 101 bytes
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: Merged 1 segments, 111 bytes to disk tosatisfy reduce memory limit
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
17/08/22 16:20:43 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memoryinto reduce
17/08/22 16:20:43 INFO mapred.Merger: Merging 1 sorted segments
17/08/22 16:20:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments leftof total size: 101 bytes
17/08/22 16:20:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/08/22 16:20:43 INFO mapred.Task: Task:attempt_local992066578_0002_r_000000_0 is done. And is in the process of committing
17/08/22 16:20:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
17/08/22 16:20:43 INFO mapred.Task: Task attempt_local992066578_0002_r_000000_0 is allowed to commit now
17/08/22 16:20:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local992066578_0002_r_000000_0' to hdfs://localhost:9000/user/hadoop/output/_temporary/0/task_local992066578_0002_r_000000
17/08/22 16:20:43 INFO mapred.LocalJobRunner: reduce > reduce
17/08/22 16:20:43 INFO mapred.Task: Task 'attempt_local992066578_0002_r_000000_0' done.
17/08/22 16:20:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local992066578_0002_r_000000_0
17/08/22 16:20:43 INFO mapred.LocalJobRunner: reduce task executor complete.
17/08/22 16:20:44 INFO mapreduce.Job: Job job_local992066578_0002 running in uber mode : false
17/08/22 16:20:44 INFO mapreduce.Job: map 100% reduce 100%
17/08/22 16:20:44 INFO mapreduce.Job: Job job_local992066578_0002 completed successfully
17/08/22 16:20:44 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=1108352
FILE: Number of bytes written=2210539
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=53760
HDFS: Number of bytes written=515
HDFS: Number of read operations=67
HDFS: Number of large read operations=0
HDFS: Number of write operations=16
Map-Reduce Framework
Map input records=4
Map output records=4
Map output bytes=101
Map output materialized bytes=115
Input split bytes=130
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=115
Reduce input records=4
Reduce output records=4
Spilled Records=8
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=42
Total committed heap usage (bytes)=263462912
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=219
File Output Format Counters
Bytes Written=77

6.5 查看运行结果

上述例子完成后的结果保存在HDFS中,通过下面命令查看结果:
/usr/local/hadoop/bin/hdfs dfs -cat /user/hadoop/output/*
如果运行成功,可以看到如下结果:

1
2
3
4
1       dfsadmin
1 dfs.replication
1 dfs.namenode.name.dir
1 dfs.datanode.data.dir

6.6 删除HDFS上的输出结果

删除HDFS中的结果目录:
/usr/local/hadoop/bin/hdfs dfs -rm -r /user/hadoop/output
运行Hadoop程序时,为了防止覆盖结果,程序指定的输出目录不能存在,否则会提示错误,因此在下次运行前需要先删除输出目录。

1
2
3
4
[root@VM_36_253_centos hadoop]# /usr/local/hadoop/bin/hdfs dfs -rm -r /user/hadoop/output
17/08/22 16:24:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/22 16:24:55 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /user/hadoop/output

6.7 关闭Hadoop进程

关闭Hadoop进程:
/usr/local/hadoop/sbin/stop-dfs.sh
再起启动只需要执行下面命令:
/usr/local/hadoop/sbin/start-dfs.sh

0x02.后记

随手htop


emmm……
今天是怎么了?