Hadoop + HBase 集群
最好在各个服务器的/etc/hosts里面,配置机器名:IP对,以下所有配置文件,都用机器名
192.168.2.79 webdev
192.168.5.11 TEST-A
192.168.5.12 TEST-B (后来新增)
下载最新的HBase
http://labs.renren.com/apache-mirror/hadoop/hbase/hbase-0.20.3/
安装步骤
http://hadoop.apache.org/hbase/docs/r0.20.3/api/overview-summary.html#overview_description
2.79
cp hbase-0.20.3.tar.gz /home/iic/
cd /home/iic
gzip -d hbase-0.20.3.tar.gz
tar xvf hbase-0.20.3.tar
cd hbase-0.20.3
chmod 700 bin/*
vi conf/hbase-env.sh: export JAVA_HOME=/home/bmb/jdk1.6.0_16
修改2.79的conf/hadoop-env.sh,加入对HBASE类库的引用
export HBASE_HOME=/home/iic/hbase-0.20.3
(此处的配置要加上$HADOOP_CLASSPATH:,不然会影响Hive的启动)
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.20.3.jar:$HBASE_HOME/hbase-0.20.3-test.jar:$HBASE_HOME/conf:${HBASE_HOME}/lib/zookeeper-3.3.0.jar
5.11
scp hbase-0.20.3.tar iic@192.168.5.11:/home/iic/
vi conf/hbase-env.sh: export JAVA_HOME=/home/iic/jdk1.6.0_16
1:提高ulimit数量
ulimit -n 2048
2:所有机器的时间必须同步
date -s 10:54:12
date -s 100412
3:修改 ${HBASE_HOME}/conf/hbase-site.xml,指向2.79 Hadoop集群 (/hbase必须自动创建)
<property>
<name>hbase.rootdir</name>
<value>hdfs://webdev:9000/hbase</value>
<description>The directory shared by region servers.
</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>webdev,TEST-A</value>
<description>Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed modes
of operation. For a fully-distributed setup, this should be set to a full
list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
this is the list of servers which we will start/stop ZooKeeper on.
</description>
</property>
4:修改2.79 ${HBASE_HOME}/conf/regionservers
webdev
TEST-A
TEST-B
5:HBase 依赖 ZooKeeper 集群 (本例子使用默认的Hbase启动内嵌的Zookeeper,而不是使用单独Zookeeper集群)
${HBASE_HOME}/conf/hbase-env.sh的变量HBASE_MANAGES_ZK 默认true. tells HBase whether to start/stop the ZooKeeper quorum servers alongside the rest of the servers.
5:让HBase看到Hadoop的HDFS client configuration
Add a pointer to your HADOOP_CONF_DIR
to CLASSPATH
in hbase-env.sh
.
Add a copy of hdfs-site.xml
(or hadoop-site.xml
) to ${HBASE_HOME}/conf
, or
if only a small set of HDFS client configurations, add them to hbase-site.xml
.
此功能很重要,比如Hadoop指定了replication是2,如果不按照上面的操作,则Hbase不会跟Hadoop一致。
HBase默认的replication是3
cd /home/iic/hbase-0.20.3
vi conf/hbase-env.sh
export HBASE_CLASSPATH=/home/iic/hadoop-0.20.2
启动Hbase集群
/home/iic/hbase-0.20.3
启动后,日志hbase-iic-master-webdev.log出现错误:
Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://192.168.2.79:9000/hbase, expected: hdfs://webdev:9000
修改:hbase-site.xml
重启后,还是出错:
尝试把“hbase.zookeeper.quorum”的IP“192.168.2.79,192.168.5.11”,改为机器名“webdev,TEST-A”
在/etc/hosts中互相添加映射
192.168.2.79 webdev
192.168.5.11 TEST-A
重启后,5.11的zookeeper日志出现错误:org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
最后发现5.11的/etc/hosts是如下配置,
127.0.0.1 TEST-A localhost.localdomain localhost
由于以上的配置,导致2.79的master.log
Updated ZNode /hbase/rs/1271048379831 with data 127.0.0.1:60020
Updated ZNode /hbase/rs/1271048380554 with data 192.168.2.79:60020
而5.11想再次注册Test-A时,发现已经存在相同的“ data 127.0.0.1:60020”
修改/etc/hosts成:
127.0.0.1 localhost.localdomain localhost
192.168.5.11 TEST-A
重启后,RegionServer运行正常,但是5.11的Zookeeper日志有错误:
Got exception when processing sessionid:0x124827089980006 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
此错误不影响HBase的运行,临时解决方案是采用单ZK Server
清空HDFS /hbase目录,可以重新生成Zookeeper节点信息:
bin/hadoop fs -rmr /hbase
修改 <value>webdev,TEST-A</value>为 <value>webdev</value>,但QuarmServer
但是,运行一段时间,仍然出现以上错误。
升级ZooKeeper版本,把hbase-0.20.3自带的zookeeper-3.2.2.jar,升级到最新版的zookeeper-3.3.0.jar。
rm -rf /tmp/hbase-iic/zookeeper/
bin/hadoop fs -rmr /hbase
出现“2010-04-13 09:40:02,685 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x127f4d2acc70001 type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/hbase Error:KeeperErrorCode = NodeExists for /hbase”
从日志看,这个异常的日志级别是INFO,可能是正常的情况,不影响Hbase运行。
但是升级Zookeeper以后,带来新的问题,Hadoop不能Map Reduce(是因为换了ZooKeeper,没有重新启动Hadoop?hadoop-env.sh引用旧版的ZooKeeper):
java.io.IOException: java.io.IOException: Could not find requested method, the usual cause is a version mismatch between client and server.
ps -ef|grep hadoop 得到的NameNode中,Classpath指向/home/iic/hbase-0.20.3/lib/zookeeper-3.2.2.jar org.apache.hadoop.hdfs.server.namenode.NameNode。修改hadoop
vi conf/hadoop-env.sh
查看Master
http://192.168.2.79:60010/master.jsp
查看Region Server
http://webdev:60030/regionserver.jsp
http://test-a:60030/regionserver.jsp
查看ZK Tree
http://192.168.2.79:60010/zk.jsp
批量删除Region Server进程
kill -9 `ps -ef |grep hbase |grep -v grep |awk '{print $2}' `
kill -9 `ps -ef |grep hbase.regionserver |grep -v grep |awk '{print $2}' `
bin/hbase-daemon.sh stop zookeeper
bin/hbase-daemon.sh start zookeeper
----------------------------------------------------------------
测试Hbase按照是否正确
bin/hbase shell
create 'scores', 'grade', 'course'
list
describe 'scores'
put 'scores', 'Tom', 'grade:', '1'
put 'scores', 'Tom', 'course:math', '87'
put 'scores', 'Tom', 'course:art', '97'
put 'scores', 'Jerry', 'grade:', '2'
put 'scores', 'Jerry', 'course:math', '100'
put 'scores', 'Jerry', 'course:art', '80'
get 'scores', 'Tom'
get 'scores', 'Jerry'
scan 'scores'
scan 'scores', ['course:']
----------------------------------------------------------------
Hbase Client:D:\7g\Projects\BMB\Hadoop-Projects\Hbase-Learning
项目中包含hbase-site.xml
同时覆盖三个参数:<name>hbase.master</name>,<name>hbase.rootdir</name>,<name>hbase.zookeeper.quorum</name>
分享到:
相关推荐
徐老师大数据培训Hadoop+HBase+ZooKeeper+Spark+Kafka+Scala+Ambari
1、内容概要:Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+Elasticsearch+Redash等大数据集群及组件搭建指南(详细搭建步骤+实践过程问题总结)。 2、适合人群:大数据运维、大数据相关技术及组件初学者。 3、...
hadoop+hbase集群搭建 详细手册
数据仓库hadoop+zookeeper+hbase集群安装方法记录,自己搭建纯手写的记录。相关软件请自行下载
hadoop集群配置流程以及用到的配置文件,hadoop2.8.4、hbase2.1.0、zookeeper3.4.12
jdk1.8.0_131、apache-zookeeper-3.8.0、hadoop-3.3.2、hbase-2.4.12 mysql5.7.38、mysql jdbc驱动mysql-connector-java-8.0.8-dmr-bin.jar、 apache-hive-3.1.3 2.本文软件均安装在自建的目录/export/server/下 ...
七月在线七月在线## Note, this file is written by cloud-init on first boot of an instance
Hadoop HA高可用集群搭建(Hadoop+Zookeeper+HBase)
Hadoop+Zookeeper+Hbase+Hive部署
hadoop+hbase分布式集群安装方法。
hadoop+hbase+hive集群搭建
从零开始hadoop+zookeeper+hbase+hive集群安装搭建,内附详细配置、测试、常见error等图文,按照文档一步一步搭建肯定能成功。(最好用有道云打开笔记)
Hadoop2.2+Zookeeper3.4.5+HBase0.96集群环境搭建
通过本文档,你将掌握搭建hadoop和hbase的方法,深入浅出,易学上手快。
hadoop+HBase+Kylin集群启动停止命令
hadoop+spark+hive Linux centos大数据集群搭建,简单易懂,从0到1搭建大数据集群
Docker(Hadoop_3.3.1+HBase_2.4.16+Zookeeper_3.7.1+Hive_3.1.3 )配置文件 搭建集群环境
Hadoop2.6+HA+Zookeeper3.4.6+Hbase1.0.0 集群安装详细步骤
apache-phoenix-4.13.0-HBase-1.3-bin.tar.gz hadoop-2.7.4.tar.gz hbase-1.3.1-bin.tar.gz jdk-8u144-linux-x64.tar.gz kafka_2.12-1.0.0.tgz scala-2.12.4.tar.gz scala-2.12.4.tgz spark-2.2.0-bin-hadoop2.7.tgz...
Hadoop+Zookeeper+HBase环境搭建,详细步骤和实例,从零开始搭建Hadoop集群