當前位置：首頁 > server > anz > 詳細頁面

Hadoop部署的基礎設施操作詳解

時間：2023-03-01來源：系統城裝機大師作者：佚名

hadoop.apache.org/docs/r2.6.5…

支持最好的平臺：GNU/Linux
依賴的軟件：
- Java™ must be installed. Recommended Java versions are described at HadoopJavaVersions.Hadoop基于Java開發，Java的移動性好
- ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.
部署模式：
- Local (Standalone) Mode 單機模式，主要用于debug
- Pseudo-Distributed Mode 偽分布式，單節點多角色
- Fully-Distributed Mode 完全分布式，多節點多角色

基礎設施

操作系統、環境、網絡、必須軟件

設置IP及主機名
關閉防火墻&selinux
設置hosts映射
時間同步
安裝jdk
設置SSH免秘鑰

設置IP及主機名

可以正常聯網查看IP

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37 [root@localhost ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.118 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::3db1:f589:4820:5457 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:18:10:17 txqueuelen 1000 (Ethernet)
RX packets 462062 bytes 670399544 (639.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 166351 bytes 11096211 (10.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 84 bytes 13940 (13.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 84 bytes 13940 (13.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:5a:69:e1 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
// 配置
DEVICE=eth0
#HWADDR=00:0C:29:42:15:C2
TYPE=Ethernet
ONBOOT=yes
NM_TROLLED=yes
BOOTPROTO=static
IPADDR=192.168.150.11
NETMASK=255.255.255.0
GATEWAY=192.168.150.2
DNS1=223.5.5.5
DNS2=114.114.114.114

修改配置文件設置主機名

1
2
3 [root@localhost ~]# vi /etc/sysconfig/network
// 設置主機名并保存退出
HOSTNAME=node01

關閉防火墻&selinux

1
2
3
4
5
6
7
8
9
10 //關閉防火墻
[root@localhost ~]# systemctl stop firewalld
//關閉開機啟動防火墻
[root@localhost ~]# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@localhost ~]#
[root@localhost ~]# vi /etc/selinux/config
//將配置文件中的SELINUX設置為disabled
SELINUX=disabled

設置hosts映射

增加映射關系，因為在現在搭建環境的時候，一般會做解耦，不會再軟件配置文件中寫ip地址，主要使用主機名，就和dns解析一樣，避免ip改變時要修改很多配置文件

1
2
3
4 [root@localhost ~]# vi /etc/hosts
// 增加映射關系
192.168.0.118 node01
192.168.0.119 node02

時間同步

所有節點時間需同步一致，不然未來做心跳的時候會有問題

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20 [root@localhost ~]# date
Fri Dec 30 22:23:45 CST 2022
// 安裝ntp
[root@localhost ~]# yum install ntp -y
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* extras: mirrors.163.com
* updates: mirrors.163.com
Package ntp-4.2.6p5-29.el7.centos.2.x86_64 already installed and latest version
Nothing to do
[root@localhost ~]# vi /etc/ntp.conf
// 增加配置
server ntp1.aliyun.com
// 啟動ntpd
[root@localhost ~]# systemctl start ntpd.service
// 配置開機啟動ntpd
[root@localhost ~]# systemctl enable ntpd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@localhost ~]#

安裝jdk

設置環境變量的時候要記得設置兩次，一次在安裝jdk的時候設置，一次在搭建Hadoop的時候設置

去官網下載jdk，最好是rpm包，然后上傳到服務器節點中（使用jdk壓縮包解壓安裝需要做很多其他的配置）

這里使用過windows下載，通過scp命令上傳到118節點中

1
2
3
4 F:\workspace\demo>scp C:\Users\43866\Downloads\jdk-8u351-linux-x64.rpm root@192.168.0.118:/usr/jdk
root@192.168.0.118's password:
jdk-8u351-linux-aarch64.rpm 100% 59MB 70.1MB/s 00:00
F:\workspace\demo>

在118節點中使用命令安裝jdk

1
2
3
4
5
6
7
8
9
10
11
12 [root@node01 jdk]# rpm -i jdk-8u351-linux-x64.rpm
warning: jdk-8u351-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
[root@node01 jdk]#

設置JAVA_HOME環境變量

1
2
3
4 [root@node01 default]# vi /etc/profile
// 追加環境變量并保存文件
export JAVA_HOME=/usr/java/default
export PATH=$PATH:$JAVA_HOME/bin

設置SSH免秘鑰

想要使用Hadoop腳本就必須設置ssh免密

1、檢查是否已設置ssh免密

1
2
3
4
5
6
7 [root@node01 default]# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:rDsunaro1f5LyKkEeNohXfDD9oEglNXtviD/JZtA1dw.
ECDSA key fingerprint is MD5:51:38:8d:fc:e4:d2:c7:ce:66:09:c8:f2:e6:9e:a7:f2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
root@localhost's password:

通過ssh localhost命令檢查發現需要輸入密碼，說明還未設置ssh免密，而且還可以自動生成.ssh文件，這個文件要創建的話比較麻煩

2、生成秘鑰和公鑰

如果118節點想免密登錄到118節點（自己）：

118節點就要生成公鑰和秘鑰，且將公鑰放到 ~/.ssh/authorized_keys中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23 [root@node01 .ssh]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
SHA256:GHhKB7GeoYYlNPWJpCZ+VoABUyIgDbdYrjjyAdgui1k root@node01
The key's randomart image is:
+---[DSA 1024]----+
|@BB+o. |
|=X=.++. |
|*o=.=++ |
|=B +.* o |
|B.Eo+ . S |
|oOo. |
|+ . |
| |
| |
+----[SHA256]-----+
[root@node01 .ssh]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@node01 .ssh]#
[root@node01 .ssh]# ssh 192.168.0.118
Last login: Sat Dec 31 13:52:49 2022 from 192.168.0.117
[root@node01 ~]#