现象：
最近几天一个华为RH2285服务器一直不定时自动重启，基本每天一两次，查看系统日志报下面的错误，每秒记录一条错误日志
OS：OEL 6.5
$ more /var/log/message
Jul 21 08:54:32 customerkernel: EDAC MC1: 5486 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:33 customerkernel: EDAC MC1: 11480 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:34 customerkernel: EDAC MC1: 11330 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:35 customerkernel: EDAC MC1: 6584 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:36 customerkernel: EDAC MC1: 27428 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:37 customerkernel: EDAC MC1: 30113 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:38 customerkernel: EDAC MC1: 4453 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:39 customerkernel: EDAC MC1: 6269 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:40 customer kernel:EDAC MC1: 15720 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1 page:0x0offset:0x0 grain:8 syndrome:0x0)
Jul 21 08:54:41 customerkernel: EDAC MC1: 16107 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)
分析解决：
这个是[EDAC (Error Detection AndCorrection)](https://www.kernel.org/doc/Documentation/edac.txt) 的日志.
CE Error 是 Correctable Error 的简称，另外还有 UE(Uncorrectable Error)
按照上面的文档, 找出错误的DIMM:
[[email protected] log]# grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count
/sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch1_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch2_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch1_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch2_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch2_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch2_ce_count:554836518
查到是 /mc1/csrow1/ch2, 根据结构图:
Channel 0 Channel 1
===================================
csrow0 | DIMM_A0 | DIMM_B0 |
csrow1 | DIMM_A0 | DIMM_B0 |
===================================
===================================
csrow2 | DIMM_A1 | DIMM_B1 |
csrow3 | DIMM_A1 | DIMM_B1 |
===================================
然后通过dmidecode查看：
[[email protected] log]# dmidecode -t memory |grep 'Locator: DIMM'
Locator: DIMM_D0
Locator: DIMM_D1
Locator: DIMM_E0
Locator: DIMM_E1
Locator: DIMM_F0
Locator: DIMM_F1
Locator: DIMM_A0
Locator: DIMM_A1
Locator: DIMM_B0
Locator: DIMM_B1
Locator: DIMM_C0
Locator: DIMM_C1
通过服务器控制台查看内存：
主板上内存插槽的分布：
结合报错日志：kernel: EDAC MC1: 16107 CE error on CPU#1Channel#2_DIMM#1 (channel:2slot:1
应该是内存插槽DIMM_F1的问题。
解决：
最后我们要做的就是，把有问题的F1插槽上的内存拔出来或是更换到其它的内存插槽上面，之后系统启动后不再报错。
参考：
http://blog.tankywoo.com/2014/12/02/edac-dimm-ce-error.html
http://serverfault.com/questions/648240/how-can-i-find-which-memory-have-ce-error
版权声明：本文为博主原创文章，未经博主允许不得转载。