一、实验说明
1、本实验将使用DNS而不是hosts文件解析主机名;
2、使用NFS共享密钥文件,而不是逐个手工拷贝添加密钥;
3、复制Hadoop时使用批量拷贝脚本而不是逐台复制。
测试环境:
Hostname | IP | Hadoop版本 | Hadoop | 功能 | 系统 |
hadoop1 | 192.168.1.161 | 0.20.0 | namenode | nfs服务器端 | rhel5.4x86 |
hadoop2 | 192.168.1.162 | 0.20.0 | datanode | dns+nfs客服端 | rhel5.4 x86 |
hadoop3 | 192.168.1.163 | 0.20.0 | datanode | nfs客户端 | rhel5.4 x86 |
二、DNS的安装与配置
1、上传dns目录:
1 [root@hadoop2 dns]# ls2 dnsmasq.conf dnsmasq.hosts dnsmasq.resolv.conf pid start.sh stop.sh
2、修改dns目录中的文件:
----dnsmasq.conf为dnsmasq的配置文件---- [root@hadoop2 dns]# cat dnsmasq.conf cache-size=50000dns-forward-max=1000resolv-file=/dns/dnsmasq.resolv.confaddn-hosts=/dns/dnsmasq.hosts----dnsmasq缓存下来的域名,不使用/etc/hosts---- [root@hadoop2 dns]# cat dnsmasq.hosts 192.168.1.161 hadoop1192.168.1.162 hadoop2192.168.1.163 hadoop3----在dnsmasq.resolv.conf添加上游dns的地址---- [root@hadoop2 dns]# cat dnsmasq.resolv.conf ### /etc/resolv.conf file autogenerated by netconfig!## Before you change this file manually, consider to define the# static DNS configuration using the following variables in the# /etc/sysconfig/network/config file:# NETCONFIG_DNS_STATIC_SEARCHLIST# NETCONFIG_DNS_STATIC_SERVERS# NETCONFIG_DNS_FORWARDER# or disable DNS configuration updates via netconfig by setting:# NETCONFIG_DNS_POLICY=''## See also the netconfig(8) manual page and other documentation.## Note: Manual change of this file disables netconfig too, but# may get lost when this file contains comments or empty lines# only, the netconfig settings are same with settings in this# file and in case of a "netconfig update -f" call.#nameserver 218.108.248.228 nameserver 218.108.248.200 [root@hadoop2 dns]# cat start.sh #!/bin/shkillall dnsmasqdnsmasq --port=53 --pid-file=/dns/pid --conf-file=/dns/dnsmasq.conf[root@hadoop2 dns]# cat stop.sh #!/bin/shkillall dnsmasq
3、启动dns,并在hadoop2上进行测试:
[root@hadoop2 dns]# dig @hadoop2 www.qq.com; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> @hadoop2 www.qq.com; (1 server found);; global options: printcmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41272;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 7;; QUESTION SECTION:;www.qq.com. IN A;; ANSWER SECTION:www.qq.com. 5 IN A 182.254.8.146;; AUTHORITY SECTION:www.qq.com. 5 IN NS ns-cmn1.qq.com.www.qq.com. 5 IN NS ns-cnc1.qq.com.www.qq.com. 5 IN NS ns-os1.qq.com.;; ADDITIONAL SECTION:ns-os1.qq.com. 5 IN A 184.105.66.196ns-os1.qq.com. 5 IN A 202.55.2.226ns-os1.qq.com. 5 IN A 202.55.2.230ns-os1.qq.com. 5 IN A 114.134.85.106ns-cmn1.qq.com. 5 IN A 120.204.202.200ns-cnc1.qq.com. 5 IN A 125.39.127.27ns-cnc1.qq.com. 5 IN A 61.135.167.182;; Query time: 33 msec;; SERVER: 192.168.1.162#53(192.168.1.162);; WHEN: Sat Aug 24 19:40:25 2013;; MSG SIZE rcvd: 221
4、配置hadoop1和hadoop3的/etc/resolv.conf
----dns客户端需要对/etc/resolv.conf文件中注释掉search localdomain,否则无法解析dnsmasq缓存下来的域名---- [hadoop@hadoop3 ~]$ cat /etc/resolv.conf; generated by /sbin/dhclient-script#search localdomain#nameserver 192.168.11.2 nameserver 192.168.1.162[root@hadoop3 ~]# dig @hadoop2 www.qq.com; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5 <<>> @hadoop2 www.qq.com; (1 server found);; global options: printcmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2061;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 7;; QUESTION SECTION:;www.qq.com. IN A;; ANSWER SECTION:www.qq.com. 5 IN A 182.254.8.146;; AUTHORITY SECTION:www.qq.com. 5 IN NS ns-cnc1.qq.com.www.qq.com. 5 IN NS ns-cmn1.qq.com.www.qq.com. 5 IN NS ns-os1.qq.com.;; ADDITIONAL SECTION:ns-os1.qq.com. 5 IN A 202.55.2.226ns-os1.qq.com. 5 IN A 202.55.2.230ns-os1.qq.com. 5 IN A 114.134.85.106ns-os1.qq.com. 5 IN A 184.105.66.196ns-cmn1.qq.com. 5 IN A 120.204.202.200ns-cnc1.qq.com. 5 IN A 61.135.167.182ns-cnc1.qq.com. 5 IN A 125.39.127.27;; Query time: 24 msec;; SERVER: 192.168.1.162#53(192.168.1.162);; WHEN: Sat Aug 24 19:44:43 2013;; MSG SIZE rcvd: 221
三、配置NFS
1、查看nfs是否已经安装
[root@hadoop1 ~]# rpm -qa |grep nfsnfs-utils-1.0.9-42.el5nfs-utils-lib-1.0.8-7.6.el5
2、编辑/etc/exports
[root@hadoop1 ~]# cat /etc/exports /home/hadoop/.ssh/ *(rw,sync,no_root_squash)
3、创建hadoop用户
[root@hadoop1 ~]# useradd hadoop[root@hadoop1 ~]# passwd hadoopChanging password for user hadoop.New UNIX password: BAD PASSWORD: it is based on a dictionary wordRetype new UNIX password: passwd: all authentication tokens updated successfully.
4、生成ssh密钥
[hadoop@hadoop1 ~]$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'.Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa.Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.The key fingerprint is:11:a6:28:73:db:0b:c2:47:fc:c9:8d:1c:0c:b4:6e:00 hadoop@hadoop1
5、修改挂载点的属性
[root@hadoop1 ~]# chmod 777 /home/hadoop/.ssh/
6、重启nfs
[root@hadoop1 ~]# service nfs restartShutting down NFS mountd: [FAILED]Shutting down NFS daemon: [FAILED]Shutting down NFS quotas: [FAILED]Shutting down NFS services: [FAILED]Starting NFS services: [ OK ]Starting NFS quotas: [ OK ]Starting NFS daemon: [ OK ]Starting NFS mountd: [ OK ]
7、在本机挂载测试
[root@hadoop1 ~]# mount 192.168.1.161:/home/hadoop/.ssh /mnt[root@hadoop1 ~]# mount/dev/sda1 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)nfsd on /proc/fs/nfsd type nfsd (rw)192.168.1.161:/home/hadoop/.ssh on /mnt type nfs (rw,addr=192.168.1.161)[root@hadoop1 ~]# ll /home/hadoop/.ssh/total 8-rw------- 1 hadoop hadoop 1675 Aug 25 10:59 id_rsa-rw-r--r-- 1 hadoop hadoop 396 Aug 25 10:59 id_rsa.pub[root@hadoop1 ~]# ll /mnttotal 8-rw------- 1 hadoop hadoop 1675 Aug 25 10:59 id_rsa-rw-r--r-- 1 hadoop hadoop 396 Aug 25 10:59 id_rsa.pub
四、nfs整合ssh密钥
1、先将id_rsa.pub拷贝成authorized_keys
[hadoop@hadoop1 ~]$ cp .ssh/id_rsa.pub .ssh/authorized_keys
2、再登陆hadoop2和hadoop3创建hadoop用户并用hadoop登陆,然后生成每个机器的ssh的rsa密钥
----hadoop2和hadoop3操作一样---- [root@hadoop2 dns]# useradd hadoop[root@hadoop2 dns]# passwd hadoopChanging password for user hadoop.New UNIX password: BAD PASSWORD: it is based on a dictionary wordRetype new UNIX password: passwd: all authentication tokens updated successfully.[root@hadoop2 dns]# su - hadoop[hadoop@hadoop2 ~]$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'.Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa.Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.The key fingerprint is:3c:9d:07:2a:7d:3d:e3:d3:22:0c:0e:8b:5d:96:93:e1 hadoop@hadoop2
3、在hadoop2和hadoop3上挂载nfs
[root@hadoop2 dns]# mount 192.168.1.161:/home/hadoop/.ssh /mnt[root@hadoop2 dns]# ll /mnttotal 12-rw-r--r-- 1 hadoop hadoop 396 Aug 25 11:04 authorized_keys-rw------- 1 hadoop hadoop 1675 Aug 25 10:59 id_rsa-rw-r--r-- 1 hadoop hadoop 396 Aug 25 10:59 id_rsa.pub[root@hadoop2 dns]# mount/dev/sda1 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)192.168.1.161:/home/hadoop/.ssh on /mnt type nfs (rw,addr=192.168.1.161)
4、把hadoop2和hadoop3的公钥id_rsa_pub都添加到/mnt/authorized_keys里
[root@hadoop2 dns]# cat /home/hadoop/.ssh/id_rsa.pub >> /mnt/authorized_keys [root@hadoop3 ~]# cat /home/hadoop/.ssh/id_rsa.pub >> /mnt/authorized_keys
5、查看authorized_keys内容
[hadoop@hadoop1 ~]$ cat /mnt/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA32vNwXv/23k0yF7QeITb61J5uccudHB3gQBtqCnB7wsOtIhdUsVIfcxGmPnWp6S9V+Ob+b73Vrl2xsxP4i0N8Cu1l2ZcU9jevc+o37yX4nW2oTBFVEP31y9E9fXkYf3cKiF0UrvunL59qgNnVUbq8qRtFr5QPAx6lGY0TYZiPaPr+POwNKF1IZvToqABsOnNimv0DNmAhbd3QyM7GaR/ZRQKOCMF8NYljo6exoDk9xPq/wCHC/rBnAU3gUlwi7Kn/tk2dirwvYZuqP3VO+w5zd6sYxscD8+UNK99XdOARzTlc8/iEPHy+JSBa6sQI2hOAOCAuHBtTymoJFUDH9YqXQ== hadoop@hadoop1ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4lTx6JTZlhoLI4Yyo0a6YeDmIgz60pYwYKwVL+p4wfp9OWB2/sEyf9iCsK8i94mnWMfNsRehqAG2ucPmWz1s/Kufxu/6uc8hJjDlOOMUOE7ENyN0Zre5MHj8jauDRhY4y37Rh3Crx86wzq79isDqJOWnKyjPQDjUH45780Hvtk87ckwNNSFhwuRgTFKhz0bQloJuHazU1/W924wmicqeEUSGhUFEkXUeJu7FqQjJcPjoRNqyTEuCHiYVh9HjOrUPdosfYqmQfuZ/x2gmsGRUdfTl32rkoZW43ay8CFV/MKqAFucEOiiHW7xttmm3zJgcyLptGhjo7NtvAQwKkPfG6w== hadoop@hadoop2ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs7fkzQMR6yVqLBVAnJqTxFPO9NNngrmYDNZMbWXDz6V8J4Z7zC46odUERe3CNjC+v3X8rwvUWlALYtvMNonQwhnpvqe2s0CpDithSFkOt5fQarRYP5JtAjHvF5b22NqcyltF+ywLT4zKAg4tjgGV5nLafI2hsNjgljUOXkRjpwSSUpLmLayWnepLIwioCPPGIkM40balUOEWEASzaI4DaPoywmoVUrByou71i1F1VizXpbhIWW+LE2cANAy1xmP0zYBa+/O4mvpgZjWLtLpKFR/1nRZPh1emy+OB6RcoJl3Awmhcsyyjd4Q8jfOYsH78PKpnwJfyhtUEIENrzUV63w== hadoop@hadoop3
6、做软连接(此步骤不需要在nfs服务器端做,只在客户端做)
[hadoop@hadoop2 ~]$ ln -s /mnt/authorized_keys /home/hadoop/.ssh/authorized_keys[hadoop@hadoop2 ~]$ ll /home/hadoop/.ssh/authorized_keyslrwxrwxrwx 1 hadoop hadoop 20 Aug 25 11:14 /home/hadoop/.ssh/authorized_keys -> /mnt/authorized_keys[hadoop@hadoop3 ~]$ ln -s /mnt/authorized_keys /home/hadoop/.ssh/authorized_keys[hadoop@hadoop3 ~]$ ll /home/hadoop/.ssh/authorized_keyslrwxrwxrwx 1 hadoop hadoop 20 Aug 25 11:15 /home/hadoop/.ssh/authorized_keys -> /mnt/authorized_keys
7、修改权限
[hadoop@hadoop1 ~]$ chmod 700 /home/hadoop/.ssh/
备注:如果不修改的话,在进行登陆的时候会出现需要密码。
8、测试是否实验无密码登陆
[hadoop@hadoop1 ~]$ ssh hadoop2The authenticity of host 'hadoop2 (192.168.1.162)' can't be established.RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'hadoop2,192.168.1.162' (RSA) to the list of known hosts.[hadoop@hadoop2 ~]$ ssh hadoop3The authenticity of host 'hadoop3 (192.168.1.163)' can't be established.RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'hadoop3,192.168.1.163' (RSA) to the list of known hosts.[hadoop@hadoop3 ~]$
五、批量安装Hadoop
1、先在hadoop1上把namenode安装完成,安装hadoop分布式可以参考:[hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp hadoop-0.20.2 hadoop@"$1":/home/hadoop/"}' > scp.sh[hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp temp hadoop@"$1":/home/hadoop/"}' >> scp.sh[hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp user hadoop@"$1":/home/hadoop/"}' >> scp.sh[hadoop@hadoop1 ~]$ cat hadoop-0.20.2/conf/slaves | awk '{print "scp -rp jdk1.7 hadoop@"$1":/home/hadoop/"}' >> scp.sh[hadoop@hadoop1 ~]$ lshadoop-0.20.2 jdk1.7 scp.sh temp user[hadoop@hadoop1 ~]$ cat scp.sh scp -rp hadoop-0.20.2 hadoop@192.168.1.162:/home/hadoop/scp -rp hadoop-0.20.2 hadoop@192.168.1.163:/home/hadoop/scp -rp temp hadoop@192.168.1.162:/home/hadoop/scp -rp temp hadoop@192.168.1.163:/home/hadoop/scp -rp user hadoop@192.168.1.162:/home/hadoop/scp -rp user hadoop@192.168.1.163:/home/hadoop/scp -rp jdk1.7 hadoop@192.168.1.162:/home/hadoop/scp -rp jdk1.7 hadoop@192.168.1.163:/home/hadoop/
2、格式化namenode
[hadoop@hadoop1 ~]$ hadoop-0.20.2/bin/hadoop namenode -format13/08/25 11:52:39 INFO namenode.NameNode: STARTUP_MSG: /************************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = hadoop1/192.168.1.161STARTUP_MSG: args = [-format]STARTUP_MSG: version = 0.20.2STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010************************************************************/Re-format filesystem in /home/hadoop/user/name ? (Y or N) Y13/08/25 11:52:46 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop13/08/25 11:52:46 INFO namenode.FSNamesystem: supergroup=supergroup13/08/25 11:52:46 INFO namenode.FSNamesystem: isPermissionEnabled=true13/08/25 11:52:47 INFO common.Storage: Image file of size 96 saved in 0 seconds.13/08/25 11:52:48 INFO common.Storage: Storage directory /home/hadoop/user/name has been successfully formatted.13/08/25 11:52:48 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************SHUTDOWN_MSG: Shutting down NameNode at hadoop1/192.168.1.161************************************************************/
3、启动hadoop
[hadoop@hadoop1 ~]$ hadoop-0.20.2/bin/start-all.shstarting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-hadoop1.out192.168.1.163: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-hadoop3.out192.168.1.162: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-hadoop2.outThe authenticity of host '192.168.1.161 (192.168.1.161)' can't be established.RSA key fingerprint is ca:9a:7e:19:ee:a1:35:44:7e:9d:d4:09:5c:fc:c5:0a.Are you sure you want to continue connecting (yes/no)? yes192.168.1.161: Warning: Permanently added '192.168.1.161' (RSA) to the list of known hosts.192.168.1.161: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-hadoop1.outstarting jobtracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-hadoop1.out192.168.1.162: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-hadoop2.out192.168.1.163: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-hadoop3.out
4、查看各个节点
[hadoop@hadoop1 ~]$ jdk1.7/bin/jps4416 Jps4344 JobTracker4306 SecondaryNameNode4157 NameNode[hadoop@hadoop2 ~]$ jdk1.7/bin/jps3699 TaskTracker3636 DataNode3752 Jps[hadoop@hadoop3 ~]$ jdk1.7/bin/jps4763 TaskTracker4834 Jps4653 DataNode
六、重点说明
1、如果重启以后无法自动挂载nfs,可以在/etc/rc.d/rc.local文件中添加:
/bin/mount -a
2、如果IP是自动获取的,请在DNS主机的/etc/rc.d/rc.local文件添加:
/bin/cat /app/resolv.conf > /etc/resolv.conf
[root@node1 ~]# cat /app/resolv.conf ; generated by /sbin/dhclient-script#search localdomain#nameserver 192.168.1.151
其它主机的/etc/rc.d/rc.local添加:
/bin/cat /app/resolv.conf > /etc/resolv.conf
[root@node2 ~]# cat /app/resolv.conf ; generated by /sbin/dhclient-script#search localdomainnameserver 192.168.1.151