?【hive & Hadoop】踩坑 记录

# 【hive & Hadoop】踩坑 记录
*平台部署知识*
本文记录的配置 [ hive](https://hive.apache.org/ " hive") 以及 [ Hadoop](https://hadoop.apache.org/ " Hadoop") 时可能会出现的问题以及解决方案。

## 目录

[TOC]

![文章的封面](https://img1.baidu.com/it/u=2125663575,1655506439&fm=253&fmt=auto&app=138&f=JPEG?w=864&h=376 "文章的封面")

## Hadoop 记录
在这里您可以查询到已知的 Hadoop 的错误记录信息！

### Hadoop 启动/终止报错 JAVA_HOME is not set and could not be found.

本次发生的异常信息，如果出现了类似下面的错误信息，则可能可以在本章找到解决方案！
```
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/sbin# ./stop-all.sh 
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Stopping namenodes on [liming141]
liming141: ERROR: JAVA_HOME is not set and could not be found.
Stopping datanodes
localhost: ERROR: JAVA_HOME is not set and could not be found.
Stopping secondary namenodes [liming-virtual-machine]
liming-virtual-machine: ERROR: JAVA_HOME is not set and could not be found.
Stopping nodemanagers
localhost: ERROR: JAVA_HOME is not set and could not be found.
```

#### 原因解释
在 Hadoop 启动与停止的脚本调用的时候，Hadoop会先调用 `hadoop-env.sh` ，然后会使用 Java相关的命令，因此 如果我们没有在 `hadoop-env.sh` 中配置 `JAVA_HOME` 参数，则最终会导致它无法调用 Java 命令（即使配置了环境变量也不行）。

#### 本次的解决方案

解决方案就是修改 hadoop-env.sh 文件，确保我们在这个文件中写入了下面这样的代码，这代表我们告知 Hadoop Java 要去哪里找！
```
# 在 hadoop-env.sh里面直接插入这个行就可以啦
export JAVA_HOME="/opt/software/jdk1.8.0_202"
```


### Hadoop 启动/终止报错 Cannot set priority of datanode process xxxx

如果您在启动 Hadoop 或者进行一些操作的时候，出现了类似下面的错误的时候，您或许可通过本章找到答案！
```
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/etc/hadoop# ../../sbin/start-all.sh 
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [liming-virtual-machine]
liming-virtual-machine: ERROR: Cannot set priority of namenode process 1576
Starting datanodes
localhost: ERROR: Cannot set priority of datanode process 1696
2024-03-07 17:30:31,107 ERROR conf.Configuration: error parsing conf core-site.xml
com.ctc.wstx.exc.WstxParsingException: Unexpected '<' character in element (missing closing '>'?)
 at [row,col,system-id]: [20,9,"file:/opt/software/hadoop-3.2.1/etc/hadoop/core-site.xml"]
        at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
        at com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:3088)
        at com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:3043)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2919)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3320)
        at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1366)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:174)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:156)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:361)
```


#### 原因解释

着段异常信息描述的就是 Hadoop 中的一些进程启动失败了，大部分情况是因为配置文件导致的这种错误，您可以在日志中查看到完整异常信息！
```
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [liming-virtual-machine]
liming-virtual-machine: ERROR: Cannot set priority of namenode process 1576
Starting datanodes
localhost: ERROR: Cannot set priority of datanode process 1696
2024-03-07 17:30:31,107 ERROR conf.Configuration: error parsing conf core-site.xml
com.ctc.wstx.exc.WstxParsingException: Unexpected '<' character in element (missing closing '>'?)
 at [row,col,system-id]: [20,9,"file:/opt/software/hadoop-3.2.1/etc/hadoop/core-site.xml"]
        at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
        at com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:3088)
        at com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:3043)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2919)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3320)
        at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1366)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:174)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:156)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:361)
Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Unexpected '<' character in element (missing closing '>'?)
 at [row,col,system-id]: [20,9,"file:/opt/software/hadoop-3.2.1/etc/hadoop/core-site.xml"]
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3024)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1366)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:174)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:156)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:361)
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected '<' character in element (missing closing '>'?)
 at [row,col,system-id]: [20,9,"file:/opt/software/hadoop-3.2.1/etc/hadoop/core-site.xml"]
        at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
        at com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:3088)
        at com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:3043)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2919)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3320)
        at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
        ... 12 more
```
通过日志文件中的描述我们知道了这是配置文件`core-site.xml`中的格式错误，当然，期间如果出现了其它配置错误的情况也是可以通过这样的操作来进行排查。

#### 本次解决方案

既然日志文件中提到错误是由于核心配置文件中的语法问题，那么我们就先来看一下配置文件中的语法，下面就是配置文件的错误部分展示
```
<configuration>
	<!-- 这里出现了错误，缺少了一个 ‘>’ 符号 -->
    <property
        <name>fs.defaultFS</name>
        <value>hdfs://liming141:8020</value>
    </property>
	......
</configuration>
```
只需要将 `>` 符号写上，然后重启Hadoop 就可以啦！

### Hadoop HDFS操作报错 Call From liming-virtual-machine/127.0.1.1 to liming141:8020 failed on connection exception...

下面是我们操作的完整的日志。
```
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/etc/hadoop# hadoop dfs -ls /
WARNING: Use of this script to execute dfs is deprecated.
WARNING: Attempting to execute replacement "hdfs dfs" instead.

ls: Call From liming-virtual-machine/127.0.1.1 to liming141:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
```

#### 原因解释
当 HDFS 相关的命令执行之后，Hadoop先去与name node的节点进行通信，所以出现这个情况一般就是因为name Note并没有启动。
我们可以先调用一下 `jps` 命令来判断name node是否启动了。
```
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/etc/hadoop# jps
2514 DataNode
2934 ResourceManager
2713 SecondaryNameNode
3068 NodeManager
3903 Jps
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/etc/hadoop# 
```
#### 本次解决方案
如果我们发现 `jps` 执行之后没有 namenode 就代表主节点没启动，在这里我们需要将 namenode 进行启动，下面就是启动 namenode 的命令！
```
hadoop-daemon.sh start namenode
```
启动之后，我们再调用 HDFS 就发现已经正常了！
```
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/sbin# ./hadoop-daemon.sh start namenode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/sbin# jps
2514 DataNode
4244 NameNode
2934 ResourceManager
2713 SecondaryNameNode
4314 Jps
3068 NodeManager
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/sbin# hadoop dfs -ls /
WARNING: Use of this script to execute dfs is deprecated.
WARNING: Attempting to execute replacement "hdfs dfs" instead.

Found 1 items
drwxrwx---   - root supergroup          0 2024-03-02 09:01 /tmp
root@liming-virtual-machine:/opt/software/hadoop-3.2.1/sbin# 
```

## Hive 记录

在这里您将可以查询到 hive 相关的异常信息！

### hive 启动报错 Permission denied Unable to determine Hadoop version information.

本次发生的异常信息，如果出现了类似下面的错误信息，则可能可以在本章找到解决方案！
```
/opt/apache-hive-2.3.9/bin/hive: line 247: /tmp/gunan/stderr: Permission denied
Unable to determine Hadoop version information.
'hadoop version' returned:
Hadoop 2.7.7 Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac Compiled by stevel on 2018-07-18T22:47Z Compiled with protoc 2.5.0 From source with checksum 792e15d20b12c74bd6f19a1fb886490 This command was run using /opt/hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar
```
#### 原因解释

Hive在尝试访问`/tmp/gunan/stderr`时遇到了`Permission denied`错误。这个错误通常意味着运行Hive的用户没有足够的权限来写入`/tmp/gunan/stderr`文件或目录。同时，错误消息还提到无法确定Hadoop的版本信息，但这可能是由于Hive无法正确执行`hadoop version`命令造成的。

#### 本次的解决方案

##### 检查目录权限：
首先，您需要确保/tmp/gunan/目录存在，并且运行Hive的用户有足够的权限来在其中创建和写入文件。您可以使用以下命令来检查和更改权限：
```
# 检查目录是否存在
ls -ld /tmp/gunan/

# 如果目录不存在，创建它
mkdir -p /tmp/gunan/

# 更改目录的权限，使Hive用户有读写权限
chown -R hive:hive /tmp/gunan/
chmod -R 755 /tmp/gunan/
请将hive:hive替换为实际运行Hive的用户和组。

```
##### 检查Hadoop安装：
由于Hive依赖于Hadoop，您需要确保Hadoop已经正确安装并且hadoop命令可以在Hive的上下文中执行。您可以尝试手动运行hadoop version来检查是否能正确获取版本信息。
```
/opt/hadoop-2.7.7/bin/hadoop version
```
如果这个命令成功执行并返回了Hadoop的版本信息，那么说明Hadoop的安装和配置是正确的。如果命令执行失败，您可能需要检查Hadoop的配置和环境变量。

##### 检查Hive配置：
检查Hive的配置文件（如hive-site.xml），确保其中关于Hadoop的路径和配置是正确的。特别是检查`hadoop.home.dir`属性，它应该指向Hadoop的安装目录。
最终发现是缺少此配置项目！！！


### hive 操作报错 Missing Hive Execution Jar: /xxx/apache-hive-2.3.3-bin/lib/hive-exec-*.jar
本次发生的异常信息，如果出现了类似下面的错误信息，则可能可以在本章找到解决方案！
```
/opt/apache-hive-2.3.9/bin/hive: line 247: /tmp/gunan/stderr: Permission denied
Unable to determine Hadoop version information.
'hadoop version' returned:
Hadoop 2.7.7 Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac Compiled by stevel on 2018-07-18T22:47Z Compiled with protoc 2.5.0 From source with checksum 792e15d20b12c74bd6f19a1fb886490 This command was run using /opt/hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar
```

#### 原因解释
这是因为 hive 的环境变量不正确，我们只需要设置好 hive 的环境变量即可。

#### 本次的解决方案
> 在这里的是我们的一个环境变量示例，请您按照自己的设备情况来进行修改。

```
# 开始编辑环境变量
vim /etc/profile
# 下面是环境变量
export JAVA_HOME="/opt/software/jdk1.8.0_202"
export HADOOP_HOME="/opt/software/hadoop-3.2.1"
export HIVE_HOME="/opt/software/apache-hive-3.1.2-bin"
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HIVE_HOME}/bin:$PATH
```

### hive 操作报错 java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
本次发生的异常信息，如果出现了类似下面的错误信息，则可能可以在本章找到解决方案！
```
/opt/apache-hive-2.3.9/bin/hive: line 247: /tmp/gunan/stderr: Permission denied
Unable to determine Hadoop version information.
'hadoop version' returned:
Hadoop 2.7.7 Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac Compiled by stevel on 2018-07-18T22:47Z Compiled with protoc 2.5.0 From source with checksum 792e15d20b12c74bd6f19a1fb886490 This command was run using /opt/hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar
```

#### 原因解释
这是因为 hive 在启动和进行一些操作的时候需要使用到 guava 这个jar包，但是这个hive中的lib下的包版本与 Hadoop 不匹配，因此我们只需要将 hive 中的版本删掉 替换为 Hadoop 中的 guava 包即可！

#### 本次的解决方案
> 在这里的是我们的一个示例，请您按照自己的设备情况来进行 HIVE_HOME 的配置。

```
# 解决日志冲突
mv $HIVE_HOME/lib/log4j-slf4j-impl-2.10.0.jar $HIVE_HOME/lib/log4j-slf4j-impl-2.10.0.bak
# 解决 HDFS 操作冲突
rm -rf ${HIVE_HOME}/lib/guava-19.0.jar
cp ${HADOOP_HOME}/share/hadoop/common/lib/guava-27.0-jre.jar ${HIVE_HOME}/lib/
```


------
***操作记录***
作者：[root](https://www.lingyuzhao.top//index.html?search=1 "root")
操作时间：2024-05-03 11:42:57 星期五
事件描述备注：保存/发布
 中国 天津 
[](如果不需要此记录可以手动删除，每次保存都会自动的追加记录)