The most interesting feature in JDK 7 for me, at least until the recent reappearance of closures, and probably even then, is the JSR 292 invokedynamic support. It is basically the reason why I started my attempt to build OpenJDK 7 on Snow Leopard.
Ostensibly to use the upcoming JSR 292 features it is necessary to apply a number of patches to a standard OpenJDK 7 release. See here for details. However the current revision of the BSD port seems to contain a lot of the necessary code already so I decided to see how far I could get using that rather than a patched version.
JSR 292 is very much a work-in-progress. There was an early-draft release of the specification almost eighteen months ago, but nothing since. You can see what is going on by looking at the code as it is released, but it is still changing so some or all of the following may change.
Basically JSR 292 defines a new JVM opcode
invokedynamic
which is intended to provide VM level support for method invocation in languages other than Java which have different invocation semantics. That is a drastic over-simplification but it will do for the moment.
There are also some features visible in Java itself, one of which is method handles, which can be thought of as pointers to individual methods. To obtain a handle to a given method you need an instance of a MethodType, which defines the return type of the method and the parameters it takes. The classes
MethodHandle
and
MethodType
are defined in the package
java.dyn
To determine how they currently work and what else you can do you need to look at the source code and the accompanying comments.
Making A MethodType
The following code is based on my understanding of the currently available revision of the JSR 292 code in the BSD port. Hopefully makes an instance of MethodType specifying a method which returns void and takes a single argument of type String.
import java.dyn.MethodType;
public final class DynTest
{
public static void main(String[] theArgs)
{
try
{
MethodType type = MethodType.make(Void.TYPE, String.class);
System.out.print("type == ");
System.out.println(type);
}
catch (Throwable t)
{
t.printStackTrace();
}
}
}
Running this code results in the following. Note that although the JSR292 code is present in the current revision of the BSD port it is still deemed to be experimental and has to be unlocked using the command line incantation shown.
Macintosh:tmp simon$ build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
OpenJDK 64-Bit Server VM warning: JSR 292 method handle code is mismatched to this JVM. Disabling support.
type == (java.lang.String)void
An instance of MethodType appears to have been made, but the warning message is not particularly auspicious.
The message originates in the function
JVM_RegisterMethodHandleMethods(JNIEnv, jclass)
which is in the file
$(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp
It looks like this (lightly re-formatted by me)
JVM_ENTRY(void, JVM_RegisterMethodHandleMethods(JNIEnv *env, jclass MHN_class)) {
assert(MethodHandles::spot_check_entry_names(), "entry enum is OK");
// note: this explicit warning-producing stuff will be replaced by auto-detection of the JSR 292 classes
if (!EnableMethodHandles) {
warning("JSR 292 method handles are disabled in this JVM. Use -XX:+UnlockExperimentalVMOptions -XX:+EnableMethodHandles to enable.");
return; // bind nothing
}
bool enable_MH = true;
{
ThreadToNativeFromVM ttnfv(thread);
int status = env->RegisterNatives(MHN_class, methods, sizeof(methods)/sizeof(JNINativeMethod));
if (env->ExceptionOccurred()) {
MethodHandles::set_enabled(false);
warning("JSR 292 method handle code is mismatched to this JVM. Disabling support.");
enable_MH = false;
env->ExceptionClear();
}
}
if (enable_MH) {
KlassHandle MHI_klass = SystemDictionaryHandles::MethodHandleImpl_klass();
if (MHI_klass.not_null()) {
symbolHandle raiseException_name = oopFactory::new_symbol_handle("raiseException", CHECK);
symbolHandle raiseException_sig = oopFactory::new_symbol_handle("(ILjava/lang/Object;Ljava/lang/Object;)V", CHECK);
methodOop raiseException_method = instanceKlass::cast(MHI_klass->as_klassOop())
->find_method(raiseException_name(), raiseException_sig());
if (raiseException_method != NULL && raiseException_method->is_static()) {
MethodHandles::set_raise_exception_method(raiseException_method);
} else {
warning("JSR 292 method handle code is mismatched to this JVM. Disabling support.");
enable_MH = false;
}
}
}
if (enable_MH) {
MethodHandles::set_enabled(true);
}
if (!EnableInvokeDynamic) {
warning("JSR 292 invokedynamic is disabled in this JVM. Use -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic to enable.");
return; // bind nothing
}
{
ThreadToNativeFromVM ttnfv(thread);
int status = env->RegisterNatives(MHN_class, methods2, sizeof(methods2)/sizeof(JNINativeMethod));
if (env->ExceptionOccurred()) {
MethodHandles::set_enabled(false);
env->ExceptionDescribe();
warning("JSR 292 method handle code is mismatched to this JVM. Disabling support.");
env->ExceptionClear();
} else {
MethodHandles::set_enabled(true);
}
}
}
JVM_END
As can be seen it generates the warning message in three cases. Cases one and three are if the initialization of native methods fail. Case two is if the class
sun.dyn.MethodHandleImpl
does not have a method of the form
???? static void raiseException(int arg1, Object arg2, Object arg3)
This is easy enough to check. The class is defined in the file
$(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleImpl.java
In fact it does not.
Although it is not possible to know exactly what the method should do, the only use made of it is a bit opaque,
it is certainly possible to add something like this just in case it actually does get invoked.
public static void raiseException(int arg1, Object arg2, Object arg3)
{
System.out.println("MethodHandleImpl.raiseException(" + arg1 + ", " + arg2 + ", " + arg3 + ")");
throw new RuntimeException("MethodHandleImpl.raiseException(...)");
}
Trying again following the addition
Macintosh:tmp simon$ !!
type == (java.lang.String)void
Getting A MethodHandle
So far so good, so let’s try getting a MethodHandle for a method of that type. There is a suitable one in java.io.PrintStream.
import java.dyn.MethodHandle;
import java.dyn.MethodHandles;
import java.dyn.MethodType;
import java.io.PrintStream;
public final class DynTest
{
public static void main(String[] theArgs)
{
try
{
MethodType type = MethodType.make(Void.TYPE, String.class);
System.out.print("type == ");
System.out.println(type);
MethodHandle handle = MethodHandles.lookup().findVirtual(PrintStream.class, "print", type);
System.out.print("handle == ");
System.out.println(handle);
}
catch (Throwable t)
{
t.printStackTrace();
}
}
}
}
Now what happens ?
Macintosh:tmp simon$ !!
build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
type == (java.lang.String)void
java.lang.InternalError
at sun.dyn.MethodHandleNatives.init(Native Method)
at sun.dyn.DirectMethodHandle.(DirectMethodHandle.java:48)
at sun.dyn.MethodHandleImpl.findMethod(MethodHandleImpl.java:163)
at java.dyn.MethodHandles$Lookup.findVirtual(MethodHandles.java:233)
at DynTest.main(DynTest.java:19)
Not so good.
The class
sun.dyn.MethodHandleNatives
can be found in the file
$(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleNatives.java
The class has four init native methods.
/** Initialize the method handle to adapt the call. */
static native void init(AdapterMethodHandle self, MethodHandle target, int argnum);
/** Initialize the method handle to call the correct method, directly. */
static native void init(BoundMethodHandle self, Object target, int argnum);
/** Initialize the method handle to call as if by an invoke* instruction. */
static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);
/** Initialize a method type, once per form. */
static native void init(MethodType self);
The method is being called from the constructor of the class
sun.dyn.DirectMethodHandle
which can be found in the file
$(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/DirectMethodHandle.java
The method is being passed the instance of sun.dyn.DirectMethodHandle being constructed, so it is evidently this variant that is being invoked.
/** Initialize the method handle to call as if by an invoke* instruction. */
static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);
The CPU neutral native method handle code can be found in the file
$(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp
The native method registration information is defined in the following data structure
static JNINativeMethod methods[] = {
// void init(MemberName self, AccessibleObject ref)
{CC"init", CC"("AMH""MH"I)V", FN_PTR(MHI_init_AMH)},
{CC"init", CC"("BMH""OBJ"I)V", FN_PTR(MHI_init_BMH)},
{CC"init", CC"("DMH""OBJ"Z"CLS")V", FN_PTR(MHI_init_DMH)},
{CC"init", CC"("MT")V", FN_PTR(MHI_init_MT)},
{CC"init", CC"("MEM""OBJ")V", FN_PTR(MHI_init_Mem)},
{CC"expand", CC"("MEM")V", FN_PTR(MHI_expand_Mem)},
{CC"resolve", CC"("MEM""CLS")V", FN_PTR(MHI_resolve_Mem)},
{CC"getTarget", CC"("MH"I)"OBJ, FN_PTR(MHI_getTarget)},
{CC"getConstant", CC"(I)I", FN_PTR(MHI_getConstant)},
// static native int getNamedCon(int which, Object[] name)
{CC"getNamedCon", CC"(I["OBJ")I", FN_PTR(MHI_getNamedCon)},
// static native int getMembers(Class defc, String matchName, String matchSig,
// int matchFlags, Class caller, int skip, MemberName[] results);
{CC"getMembers", CC"("CLS""STRG""STRG"I"CLS"I["MEM")I", FN_PTR(MHI_getMembers)}
};
from which we can deduce, after looking at the macro definitions for CC, DMH, etc., that the function we are interested in is
MHI_init_DMH
This function throws an InternalError exception without an error message in two different places, and calls a function which also throws an InternalError exception without an error message in two different places, so it is not possible to statically determine what has caused the InternalError.
Some experimentation results in the discovery that the InternalError is being thrown from the function invoked by MHI_init_DMH, namely
MethodHandles::init_DirectMethodHandle
at this point
if (me == NULL) { THROW(vmSymbols::java_lang_InternalError()); }
The variable me can be set in four different places in this function
...
me = MethodHandles::entry(MethodHandles::_invokeinterface_mh);
...
me = MethodHandles::entry(MethodHandles::_invokespecial_mh);
...
me = MethodHandles::entry(MethodHandles::_invokestatic_mh);
...
me = MethodHandles::entry(MethodHandles::_invokevirtual_mh);
...
Whichever call to MethodHandles::entry(EntryKind) is being made at runtime is clearly returning NULL.
The method MethodHandles::entry(EntryKind) is defined inline in the file
$(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.hpp
It is doing a straightforward array access indexed by the EntryKind argument
static MethodHandleEntry* entry(EntryKind ek) { assert(ek_valid(ek), "initialized");
return _entries[ek]; }
so it is the element in the _entries array corresponding to the EntryKind argument which is NULL.
The _entries array is static and is not directly accessible, it is declared private, and there is only one method which can be used to set its elements
static void init_entry(EntryKind ek, MethodHandleEntry* me) {
assert(ek_valid(ek), "oob");
assert(_entries[ek] == NULL, "no double initialization");
_entries[ek] = me;
}
This method is only called from one place, the method
MethodHandles::generate_method_handle_stub
which is defined in the file
$(JDK7_ROOT)/hotspot/src/cpu/x86/vm/methodHandles_x86.cpp
This method in turn is only called from one place, the method
StubGenerator::generate_all()
which is defined in the file
$(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_32.cpp
The code in question looks like this
...
if (EnableMethodHandles && SystemDictionary::MethodHandle_klass() != NULL) {
for (MethodHandles::EntryKind ek = MethodHandles::_EK_FIRST;
ek < MethodHandles::_EK_LIMIT;
ek = MethodHandles::EntryKind(1 + (int)ek)) {
StubCodeMark mark(this, "MethodHandle", MethodHandles::entry_name(ek));
MethodHandles::generate_method_handle_stub(_masm, ek);
}
}
...
It is not called from the corresponding version of the method in the file
$(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_64.cpp
which is the version used in the 64-bit version of the JVM that is running the code. Hence the problem. The array has not been initialized and all the entries are NULL.
The code in the 32-bit version does not appear to be 32-bit specific, and the method MethodHandles::generate_method_handle_stub() is defined in a generic x86 file rather than an x86_32 one so what happens if we graft the code segment above into the 64-bit version of StubGenerator::generate_all() ?
Crash, Bang, Wallop …
Macintosh:tmp simon$ !!
build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
testtype == (java.lang.String)void
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000000010281b7d2, pid=13801, tid=4298117120
#
# JRE version: 7.0
# Java VM: OpenJDK 64-Bit Server VM (17.0-b05 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# j sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
#
...
Possibly not such a good idea after all then.
Having said that, at first glance it is not obvious that this problem is a direct result of problems with the stub generation, especially since it seems to be crashing inside a Java method.
The method in question can be found in the file
$(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/AdapterMethodHandle.java
and it looks like this.
public static boolean canPairwiseConvert(MethodType newType, MethodType oldType) {
// same number of args, of course
int len = newType.parameterCount();
if (len != oldType.parameterCount())
return false;
// Check return type. (Not much can be done with it.)
Class exp = newType.returnType();
Class ret = oldType.returnType();
if (!VerifyType.isNullConversion(ret, exp))
return false;
// Check args pairwise.
for (int i = 0; i < len; i++) {
Class src = newType.parameterType(i); // source type
Class dst = oldType.parameterType(i); // destination type
if (!canConvertArgument(src, dst))
return false;
}
return true;
}
It doesn’t look as though it is doing anything particularly foolhardy.
The crash site is identified in terms of the bytecode rather than the source code, so we need actually need that
public static boolean canPairwiseConvert(java.dyn.MethodType, java.dyn.MethodType);
Code:
0: aload_0
1: invokevirtual #9; //Method java/dyn/MethodType.parameterCount:()I
4: istore_2
5: iload_2
6: aload_1
7: invokevirtual #9; //Method java/dyn/MethodType.parameterCount:()I
10: if_icmpeq 15
13: iconst_0
14: ireturn
...
The pc is seven when the crash occurs which is this instruction
7: invokevirtual #9; //Method java/dyn/MethodType.parameterCount:()I
which corresponds to the call
oldType.parameterCount()
so it looks as though there is something drastically wrong with the value of oldType at this point.
Using the stack trace from the error log
j sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
j sun.dyn.AdapterMethodHandle.makePairwiseConvert(Lsun/dyn/Access;Ljava/dyn/MethodType;Ljava/dyn/MethodHandle;)Ljava/dyn/MethodHandle;+18
j sun.dyn.MethodHandleImpl.findMethod(Lsun/dyn/Access;Lsun/dyn/MemberName;ZLjava/lang/Class;)Ljava/dyn/MethodHandle;+107
j java.dyn.MethodHandles$Lookup.findVirtual(Ljava/lang/Class;Ljava/lang/String;Ljava/dyn/MethodType;)Ljava/dyn/MethodHandle;+43
j DynTest.main([Ljava/lang/String;)V+34
we can determine that the MethodType value passed as the oldType argument originates from a call to the type() method on an instance of MethodHandle which has been passed to the calling method makePairwiseConvert()
The MethodHandle instance being passed to makePairwiseConvert is a newly created instance of DirectMethodHandle. The instance of MethodType it contains and which is returned by the call to the type() method, is in theory, the instance of MethodType created by the test program. This has already been printed successfully which implies that at that point it was sufficiently valid, or not sufficiently invalid to cause problems. This assumes of cource that it is the object actually being returned from the call to the type() method.
Given that it is extremely difficult to do anything from inside Java to a Java object which can subsequently cause the VM to crash, the problem is most likely to originate somewhere in native code.
As we already know the DirectMethodHandle constructor is invoking the static native method init() in the class MethodHandlesNative. We also know that the native function being called is actually MHI_init_DMH
This function is predominantly performing checks. It does not actually do anything very much to the MethodHandle instance it is passed. The function that it finishes by invoking
MethodHandles::init_DirectMethodHandle
however, does.
The calls of interest are
...
java_dyn_MethodHandle::init_vmslots(mh());
...
sun_dyn_DirectMethodHandle::set_vmtarget(mh(), vmtarget);
sun_dyn_DirectMethodHandle::set_vmindex(mh(), vmindex);
...
java_dyn_MethodHandle::set_vmentry(mh(), me);
...
Some experimentation reveals that the value in the instance variable type is changed during the call to
java_dyn_MethodHandle::set_vmentry()
In fact it is changed to the value of me. It looks very much as though the type instance variable is being over-written with the MethodEntry pointer which is not a pointer to a Java object at all.
The Magical vmentry Field ?
The vm_entry instance variable is defined in the class sun.dyn.MethodHandleImpl as follows.
private byte vmentry; // adapter stub or method entry point
Given that a CPU-level pointer is being stored in vmentry its declaration as a byte is rather curious.
Looking at the definition of set_vmentry() we can see that it is simply placing the pointer to the MethodEntry at _vmentry_offset within the instance which does not itself explain how the instance variable is capable of storing a 32-bit pointer let alone 64-bit one.
Maybe there is something magical about this particular instance variable ?
_vmentry_offset is itself set in the method java_dyn_MethodHandle::compute_offsets() along with the type and vmtarget instance variable offsets
void java_dyn_MethodHandle::compute_offsets() {
klassOop k = SystemDictionary::MethodHandle_klass();
if (k != NULL && EnableMethodHandles) {
compute_offset(_type_offset, k, vmSymbols::type_name(), vmSymbols::java_dyn_MethodType_signature(), true);
compute_offset(_vmtarget_offset, k, vmSymbols::vmtarget_name(), vmSymbols::object_signature(), true);
compute_offset(_vmentry_offset, k, vmSymbols::vmentry_name(), vmSymbols::machine_word_signature(), true);
// Note: MH.vmslots (if it is present) is a hoisted copy of MH.type.form.vmslots.
// It is optional pending experiments to keep or toss.
compute_optional_offset(_vmslots_offset, k, vmSymbols::vmslots_name(), vmSymbols::int_signature(), true);
}
}
Looking at the values of these offsets at runtime shows them to be as follows.
_type_offset == 24
_vmtarget_offset == 12
_vmentry_offset == 24
_vmslots_offset == 0
The type and vmentry offsets are the same so set_vmentry() is indeed clobbering the value in the type instance variable. Something
has gone wrong with the computation of the instance variable offsets, or the instance variable vmentry is insufficently magical in the 64-bit case.
The compute_offset() function simply looks up the field in the class and returns the offset as held by the field descriptor.
The offsets of the fields in an instance of a Java class are computed in the method
ClassFileParser::parseClassFile()
which is defined in the file
$(JDK7_ROOT)/hotspot/src/share/vm/classfile/classFileParser.cpp
This a long and complicated method, its almost a thousand lines long, but approximately half-way down is this
...
// adjust the vmentry field declaration in java.dyn.MethodHandle
if (EnableMethodHandles && class_name() == vmSymbols::sun_dyn_MethodHandleImpl() && class_loader.is_null()) {
java_dyn_MethodHandle_fix_pre(cp, &fields, &fac, CHECK_(nullHandle));
}
...
The Magical vmentry Field Explained
The ClassFileParser::java_dyn_MethodHandle_fix_pre() method is what makes the instance variable vmentry magic. It redefines
its type as either a Java int or long depending upon whether it is a 32-bit or a 64-bit VM.
As an aside despite the name the method is actually operating on the class sun.dyn.MethodHandleImpl not on the class java.dyn.MethodHandle.
It begins by looking for a particular field in the constant pool of the sun.dyn.MethodHandleImpl class.
...
int word_sig_index = 0;
const int cp_size = cp->length();
for (int index = 1; index < cp_size; index++) {
if (cp->tag_at(index).is_utf8() &&
cp->symbol_at(index) == vmSymbols::machine_word_signature()) {
word_sig_index = index;
break;
}
}
...
The method vmSymbols::machine_word_signature() is defined in the file
$(JDK7_ROOT)/hotspot/src/share/vm/classfile/vmSymbols.hpp
and its definition is dependent upon whether it is a 32 or a 64 bit field. In the 32 bit case it returns the signature
I
and in the 64 bit case
J
If a field with the right signature cannot be found a VirtualMachineError is thrown.
If an appropriate field is found the method then looks for a non-static field of type byte with the name vmentry. If it finds the field, it then re-defines
its type.
...
const int n = (*fields_ptr)()->length();
for (int i = 0; i < n; i += instanceKlass::next_offset) {
int name_index = (*fields_ptr)->ushort_at(i + instanceKlass::name_index_offset);
int sig_index = (*fields_ptr)->ushort_at(i + instanceKlass::signature_index_offset);
int acc_flags = (*fields_ptr)->ushort_at(i + instanceKlass::access_flags_offset);
symbolOop f_name = cp->symbol_at(name_index);
symbolOop f_sig = cp->symbol_at(sig_index);
if (f_sig == vmSymbols::byte_signature() &&
f_name == vmSymbols::vmentry_name() &&
(acc_flags & JVM_ACC_STATIC) == 0) {
// Adjust the field type from byte to an unmanaged pointer.
assert(fac_ptr->nonstatic_byte_count > 0, "");
fac_ptr->nonstatic_byte_count -= 1;
(*fields_ptr)->ushort_at_put(
i + instanceKlass::signature_index_offset,
word_sig_index);
if (wordSize == jintSize) {
fac_ptr->nonstatic_word_count += 1;
} else {
fac_ptr->nonstatic_double_count += 1;
}
FieldAllocationType atype = (FieldAllocationType) (*fields_ptr)->ushort_at(i+4);
assert(atype == NONSTATIC_BYTE, "");
FieldAllocationType new_atype = NONSTATIC_WORD;
if (wordSize > jintSize) {
if (Universe::field_type_should_be_aligned(T_LONG)) {
atype = NONSTATIC_ALIGNED_DOUBLE;
} else {
atype = NONSTATIC_DOUBLE;
}
}
(*fields_ptr)->ushort_at_put(i+4, new_atype);
found_vmentry = true;
break;
}
}
...
except that in this version of the code there is a bug in the 64-bit case. I didn’t spot it straight away but I did get it eventually, albeit after a lengthy excursion into the mechanics of laying out instance variables in Hotspot.
Found it yet ?
It’s here.
...
FieldAllocationType new_atype = NONSTATIC_WORD;
if (wordSize > jintSize) {
if (Universe::field_type_should_be_aligned(T_LONG)) {
atype = NONSTATIC_ALIGNED_DOUBLE;
} else {
atype = NONSTATIC_DOUBLE;
}
}
(*fields_ptr)->ushort_at_put(i+4, new_atype);
...
In the 64-bit case the field type is not correctly updated, which causes the subsequent field offset problem.
With this fixed the problem goes away.
Macintosh:tmp simon$ !!
build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
type == (java.lang.String)void
handle == print(java.io.PrintStream,java.lang.String)void
Now to work out how to actually invoke the method via the MethodHandle, assuming that it is even possible with this version of the code.
Copyright (c) 2009 By Simon Lewis. All Rights Reserved.
Just An Application 