案例分享：如何通过JVM crash 的日志和core dump定位和分析Instrument引起的JVM crash

# Problematic frame:
# V  [libjvm.so+0x5bbf05]  instanceKlass::oop_follow_contents(ParCompactionManager*, oopDesc*)+0x2c5
Stack 信息：
Stack: [0x00007fa9482b3000,0x00007fa9483b4000],  sp=0x00007fa9483b2a10,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x5bbf05]  instanceKlass::oop_follow_contents(ParCompactionManager*, oopDesc*)+0x2c5
V  [libjvm.so+0x87504c]  ParCompactionManager::follow_marking_stacks()+0x1ec
V  [libjvm.so+0x85c138]  MarkFromRootsTask::do_it(GCTaskManager*, unsigned int)+0x78
V  [libjvm.so+0x55813f]  GCTaskThread::run()+0x12f
V  [libjvm.so+0x821ca8]  java_start(Thread*)+0x108

• 看到里面的栈信息是GCTaskThread线程，初步判断在执行GC的时候发生了crash，代码段在0x5bbf05，函数是instanceKlass::oop_follow_content。
• InstanceKlass 就是我们常说的class对象，因为是在GC的时候出现问题，具体的代码段通常是在GC部分并不能容易的判断发生了什么，而我们更需要知道的是GC的时候在处理哪个对象出了问题

-XX:+VerifyBeforeGC -XX:+VerifyAfterGC

Failed: 0x000000079ac5fe30 -> 0x0000000410bc55c0

3. SA 工具之CLHSDB

java -cp .:$JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB

java -cp .:$JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB $JAVA_HOME/bin/java 99083

• 版本问题，如果产品上装了多个JVM环境的化，注意core dump要和JVM的分析的版本一致
• SA环境需要root权限

3.2 分析对象

• 地址0x000000079ac5fe30 是属于0x000000079ac5fe08地址的对象的成员，也就是MemberName对象的成员
  
• 通过0x0000000782178ab8的地址分析，这是一个reinvokeTarget的method的地址

 final class More ...MemberName implements Member, Cloneable {
73      private Class<?> clazz;       // class in which the method is defined
74      private String   name;        // may be null if not yet materialized
75      private Object   type;        // may be null if not yet materialized
76      private int      flags;       // modifier bits; see reflect.Modifier
77      //@Injected JVM_Method* vmtarget;
78      //@Injected int         vmindex;
79      private Object   resolution;  // if null, this guy is resolved
}

class VerifyFieldClosure: public OopClosure {protected:template <class T> void do_oop_work(T* p) {guarantee(Universe::heap()->is_in_closed_subset(p), "should be in heap");oop obj = oopDesc::load_decode_heap_oop(p);if (!obj->is_oop_or_null()) {tty->print_cr("Failed: " PTR_FORMAT " -> " PTR_FORMAT, p, (address)obj);Universe::print();guarantee(false, "boom");}}public:virtual void do_oop(oop* p)       { VerifyFieldClosure::do_oop_work(p); }virtual void do_oop(narrowOop* p) { VerifyFieldClosure::do_oop_work(p); }
};

Failed: 0x000000079ac5fe30 -> 0x0000000410bc55c0

inline oop oopDesc::load_decode_heap_oop_not_null(oop* p)       { return *p; }
inline oop oopDesc::load_decode_heap_oop_not_null(narrowOop* p) {return decode_heap_oop_not_null(*p);
}

void java_lang_invoke_MemberName::adjust_vmtarget(oop mname, oop ref) {
mname->address_field_put(_vmtarget_offset, (address)ref);
}

4. MethodHandler

 void MemberNameTable::adjust_method_entries(methodOop* old_methods, methodOop* new_methods,int methods_length, bool *trace_name_printed) {assert(SafepointSynchronize::is_at_safepoint(), "only called at safepoint");
-  // search the MemberNameTable for uses of either obsolete or EMCP methods
+  // For each redefined methodfor (int j = 0; j < methods_length; j++) {methodOop old_method = old_methods[j];methodOop new_method = new_methods[j];
-    oop mem_name = find_member_name_by_method(old_method);
-    if (mem_name != NULL) {
-      java_lang_invoke_MemberName::adjust_vmtarget(mem_name, new_method);
-
-      if (RC_TRACE_IN_RANGE(0x00100000, 0x00400000)) {
-        if (!(*trace_name_printed)) {
-          // RC_TRACE_MESG macro has an embedded ResourceMark
-          RC_TRACE_MESG(("adjust: name=%s",
-                         Klass::cast(old_method->method_holder())->external_name()));
-          *trace_name_printed = true;
-        }
-        // RC_TRACE macro has an embedded ResourceMark
-        RC_TRACE(0x00400000, ("MemberName method update: %s(%s)",
-                              new_method->name()->as_C_string(),
-                              new_method->signature()->as_C_string()));
-      }

5. 如何修复？

mname->address_field_put(_vmtarget_offset, (address)ref);

mname->obj_field_put(_vmtarget_offset, new_method);

 -XX:+UseCompressedOops