当前位置: 代码迷 >> 综合 >> flume-ng-sql-source读取数据库延迟8小时以上报错
  详细解决方案

flume-ng-sql-source读取数据库延迟8小时以上报错

热度:27   发布时间:2023-09-29 09:46:28.0

https://github.com/keedio/flume-ng-sql-source这个项目读取数据库的时候run.query.delay设置的时间过长,导致java.sql.SQLRecoverableException报错,

MySql默认一个connection(连接)空闲超过8小时自动断开,而程序认为这个连接还是可用的,这个时候用这个过时connection去连接MySql时就会报这个错误,oracle也有这个问题,

SQL Error: 17002, SQLState: 08006
2019/07/30 10:28:57,095
IO Error: Connection timed out
2019/07/30 10:28:57,095
[c3p0] A PooledConnection that has already signalled a Connection error is still in use!
2019/07/30 10:28:57,095
[c3p0] Another error has occurred [ java.sql.SQLRecoverableException: Closed Connection ] which will not be reported to listeners!
java.sql.SQLRecoverableException: Closed Connectionat oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:1828)at com.mchange.v2.c3p0.impl.NewProxyConnection.getAutoCommit(NewProxyConnection.java:938)at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.isAutoCommit(LogicalConnectionImpl.java:322)at org.hibernate.engine.transaction.internal.TransactionCoordinatorImpl.afterNonTransactionalQuery(TransactionCoordinatorImpl.java:199)at org.hibernate.internal.SessionImpl.afterOperation(SessionImpl.java:503)at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:1916)at org.hibernate.internal.AbstractSessionImpl.list(AbstractSessionImpl.java:311)at org.hibernate.internal.SQLQueryImpl.list(SQLQueryImpl.java:141)at org.keedio.flume.source.HibernateHelper.executeQuery(HibernateHelper.java:122)at org.keedio.flume.source.DBSQLSource.process(DBSQLSource.java:102)at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:129)at java.lang.Thread.run(Thread.java:748)
2019/07/30 10:28:57,096
SQL Error: 17008, SQLState: 08003
2019/07/30 10:28:57,096
Closed Connection
2019/07/30 10:28:57,096
Exception thrown, resetting connection.
org.hibernate.exception.JDBCConnectionException: could not inspect JDBC autocommit modeat org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132)at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:126)at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:112)at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.isAutoCommit(LogicalConnectionImpl.java:325)at 

问题一:这么解决?

c3p0里有一个配置maxIdleTime默认是0将这个配置设置的比数据库里自动断开连接的时间短就好了。


<!--最大空闲时间,60秒内未使用则连接被丢弃。若为0则永不丢弃。Default: 0 -->
<property name="maxIdleTime">60</property>

问题二:这个项目里用的hibernate-c3p0没有提供那个设置的入口

<dependency><groupId>org.hibernate</groupId><artifactId>hibernate-c3p0</artifactId><version>4.3.10.Final</version></dependency>
agent.sources.sqlSource.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider

 hibernate配置如下

#hibernate.c3p0.max_size 2
#hibernate.c3p0.min_size 2
#hibernate.c3p0.timeout 5000
#hibernate.c3p0.max_statements 100
#hibernate.c3p0.idle_test_period 3000
#hibernate.c3p0.acquire_increment 2
#hibernate.c3p0.validate false

 配置里指定了C3P0ConnectionProvider 这个类那就只能看这个代码了,在这个位置

flume-ng-sql-source读取数据库延迟8小时以上报错

第113行发现hibernate是用timeout这个参数替换了c3p0的maxIdleTime这个名字,所以设置下hibernate.c3p0.timeout的值就好了。

 

flume-ng-sql-source读取数据库延迟8小时以上报错在网上发现很多人都说hibernate.c3p0.timeout这个是获得连接的超时时间,如果超过这个时间,会抛出异常,单位秒,是错误的

最后竟然发现百度百科里错了 https://baike.baidu.com/item/c3p0/3719378?fr=aladdin

flume-ng-sql-source读取数据库延迟8小时以上报错

在官方文档里找到了,准确的信息 https://www.mchange.com/projects/c3p0/index.html#hibernate-specific

flume-ng-sql-source读取数据库延迟8小时以上报错

这两个是对应的 

  相关解决方案