在自己的机器上操作的,之前因为网络出问题强行关闭虚拟机。再次开启时发现HMaster进程启动不了(HRegionServer正常启动,而HMaster启动几秒后挂了),查看日志发现如下:
2019-09-20 20:32:05,519 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase/splitWAL/WALs%2Fchdp13%2C16020%2C1568877911961-splitting%2Fchdp13%252C16020%252C1568877911961.meta.1568906734035.meta entered state: ERR chdp13,16020,1568982710483
2019-09-20 20:32:05,519 WARN [main-EventThread] coordination.SplitLogManagerCoordination: Error splitting /hbase/splitWAL/WALs%2Fchdp13%2C16020%2C1568877911961-splitting%2Fchdp13%252C16020%252C1568877911961.meta.1568906734035.meta
2019-09-20 20:32:05,519 WARN [chdp11:16000.activeMasterManager] master.SplitLogManager: error while splitting logs in [hdfs://chdp11:9000/HBase/WALs/chdp13,16020,1568877911961-splitting] installed = 1 but only 0 done
2019-09-20 20:32:05,520 FATAL [chdp11:16000.activeMasterManager] master.HMaster: Failed to become active master
java.io.IOException: error or interrupted while splitting logs in [hdfs://chdp11:9000/HBase/WALs/chdp13,16020,1568877911961-splitting] Task = installed = 1 done = 0 error = 1at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:402)at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:312)at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:303)at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1067)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:771)at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:198)at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1868)at java.lang.Thread.run(Thread.java:748)
2019-09-20 20:32:05,521 FATAL [chdp11:16000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
2019-09-20 20:32:05,521 FATAL [chdp11:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.io.IOException: error or interrupted while splitting logs in [hdfs://chdp11:9000/HBase/WALs/chdp13,16020,1568877911961-splitting] Task = installed = 1 done = 0 error = 1at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:402)at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:312)at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:303)at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1067)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:771)at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:198)at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1868)at java.lang.Thread.run(Thread.java:748)
2019-09-20 20:32:05,521 INFO [chdp11:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
关键信息
error or interrupted while splitting logs in [hdfs://chdp11:9000/HBase/WALs/chdp13,16020,1568877911961-splitting] Task = installed = 1 done = 0 error = 1
meta数据出问题了.
打开hdfs webUI发现:
有数据丢失了。。。
参考这位大佬博客:https://www.cnblogs.com/yueweimian/p/6183264.html
因为不是生产环境,干脆一点就把HBase在hdfs上的数据全部删除了,若删错了可以从trash中找回来(若配置了trash)。。。
#我电脑上配置的hbase元数据存储路径为hdfs://chdp11:9000/hbase
hadoop fs -rm -r /hbase #删库警告
重新启动,问题解决,但hbase数据全部丢失,因为是学习环境所以也就没什么的了,没必要花太多的时间去恢复不重要的数据。