1，Tomcat启动报错如下：
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memcachedClient' defined in file [/usr/local/apache-tomcat-6.0.37_6500/webapps/trade_service/WEB-INF/classes/META-INF/spring/springContext_memcached.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] threw exception; nested exception is java.lang.OutOfMemoryError at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:581) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1015) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:911) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322) ... 44 moreCaused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] threw exception; nested exception is java.lang.OutOfMemoryError at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:570) ... 53 moreCaused by: java.lang.OutOfMemoryError
,
2，看到创建memcache clent报错，马上去memcache服务器查看一下，memcache正常运行，内存也足够剩余。
[[email protected] ~]# tail -f /tmp/memcache_error.log <287 0x00 0x00 0x00 0x00<287 0x00 0x00 0x00 0x00<287 0x00 0x00 0x00 0x00>287 Writing bin response:>287 0x81 0x0b 0x00 0x00>287 0x00 0x00 0x00 0x00>287 0x00 0x00 0x00 0x06>287 0x00 0x00 0x00 0x00>287 0x00 0x00 0x00 0x00>287 0x00 0x00 0x00 0x00...[[email protected] ~]# free -m total used free shared buffers cachedMem: 5938 3368 2570 0 195 296-/+ buffers/cache: 2876 3061Swap: 7935 0 7935[[email protected] ~]#问题在哪里？再看看日志里面有误error或者warning信息：
[[email protected] ~]# cat /tmp/memcache_error.log |grep warning[[email protected] ~]# cat /tmp/memcache_error.log |grep error[[email protected] ~]#一点错误信息也没有，判断问题不是出在memcache这一块。
3，去check tomcat服务器
[[email protected] logs]# free -m total used free shared buffers cachedMem: 18022 16943 1078 0 104 675-/+ buffers/cache: 16162 1859Swap: 20031 8012 12019[[email protected] logs]#只剩余1个G的内存，去修改一下tomcat的内存配置：
[[email protected] bin]# vim catalina.sh
#JAVA_OPTS="-Xms2048m -Xmx2048m -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=768m"# add by wangqi on 20140614 JAVA_OPTS="-Xms512m -Xmx512m -Xss128K -XX:PermSize=64m -XX:MaxPermSize=256m"再重启tomcat，看到还是报一样的错误。
4，回到应用本身去查问题
跟开发人员一起排查，看到是trade_service启动了过多的多余的事务，把tomcat的内存耗光了，所以关闭掉多余的事务，再启动tomcat，就不会再报错。
感慨：就跟大部分web应用卡，页面登录不上去，相应过慢很多是出在效率低下的sql上面，tomcat启动不起来，内存溢出大部分也是在java代码上面，大家需要写代码或者sql需要谨慎，且写且谨慎。