InvokeDynamic And MethodHandles On Snow Leopard: Part Four

The Even More Magical java.dyn.InvokeDynamic

java.dyn.InvokeDynamic is a final class with a private constructor and no other no declared methods, but you can do this with it

    package test;

    import java.dyn.InvokeDynamic;

    public final class Bar
    {
        void foo()
        {
            System.out.println(InvokeDynamic.bar("bar invoked dynamically"));
        }
    }

and the compiler will compile it to this

    Macintosh:tmp simon$ build/bsd-i586/bin/javap -c test.Bar
    Compiled from "Bar.java"
    public final class test.Bar extends java.lang.Object {
      public test.Bar();
        Code:
          0: aload_0       
          1: invokespecial #1                  // Method java/lang/Object."":()V
          4: return        

      void foo();
        Code:
          0: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
          3: ldc           #3                  // String bar invoked dynamically
          5: invokedynamic #4,  0              // NameAndType bar:(Ljava/lang/String;)Ljava/lang/String;
         10: invokevirtual #5                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
         13: return        
    }

java.dyn.InvokeDynamic is a Java language level marker for the JVM instruction formerly known as

    xxxunusedxxx

and now known as

    invokedynamic

The invokedynamic Instruction

The instruction is five bytes long. The one byte opcode, 186 (0xBA), is followed by two bytes which give the index into the constant pool of a name and type, in this case,

    bar

and

    (Ljava/lang/String;)Ljava/lang/String

followed by two zero bytes.

And that is it. Unlike the original invoke instructions

• invokevirtual
• invokespecial
• invokestatic
• invokeinterface

there is not enough information in the instruction to enable the VM to determine what it should be invoking or how.

Hence, if you run this,

    package test;

    public final class InvokeDynamicTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                Bar bar = new Bar();
			
                bar.foo();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

this happens

    Macintosh:tmp simon$ build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.InvokeDynamicTest
    java.dyn.InvokeDynamicBootstrapError: class has no bootstrap method: class test.Bar
	    at sun.dyn.CallSiteImpl.makeSite(CallSiteImpl.java:63)
	    at test.Bar.foo(Bar.java:10)
	    at test.InvokeDynamicTest.main(InvokeDynamicTest.java:13)

The Bootstrap Method

The first time the VM encounters an invokedynamic instruction it looks for the bootstrap method for the class of the caller.

The bootstrap method is a static method of the form

    static CallSite bootstrapMethod(Class theCallerClass, String theMethodName, MethodType theMethodType)

The bootstrap method for a class is registered either by name or as a MethodHandle using a static method on the class

    java.dyn.Linkage

For example

    ...
	
    static void init()
    {
        Linkage.registerBootstrapMethod(
                    Bar.class, 
                    MethodHandles.
                        lookup().
                            findStatic(
                                Bootstrap.class,
                                "bootstrap",
                                MethodType.make(
                                               CallSite.class, 
                                               Class.class, 
                                               String.class, 
                                               MethodType.class)));
    }
	
    ...

registers the static method

     private static CallSite bootstrap(Class theClass, String theName, MethodType theType)

of the class Bootstrap as the bootstrap method for the class Bar.

The CallSite

The bootstrap method is passed the class of the caller, and the name and type of the method specified by the instance of the invokedynamic instruction, and is responsible for returning an instance of the class

    java.dyn.CallSite

The returned CallSite instance contains a MethodHandle. The JVM uses the method referenced by the MethodHandle as the method to invoke.

Using this as the bootstrap method.

    private static CallSite bootstrap(Class<?> theClass, String theName, MethodType theType)
    {
        MethodHandle mh = MethodHandles.lookup().findStatic(Methods.class, "generic", theType);
        CallSite     cs = new CallSite(theClass, theName, mh.type());
		
        cs.setTarget(mh);
        return cs;
    }

this as the definition of the class Methods

    package test;

    final class Methods
    {
        static String generic(String theArgument)
        {
            return theArgument + ": brought to you by Methods.generic()";
        }
    }

and invoking Bar.foo() like this

    package test;


    public final class InvokeDynamicTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                Bootstrap.init();
			
                Bar bar = new Bar();
			
                bar.foo();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

gets you this.

    Macintosh:tmp simon$ build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.InvokeDynamicTest
    bar invoked dynamically: brought to you by Methods.generic()

While this is not of itself a particularly exciting example there are few things worth noting

• most importantly, what happens when the method Bar.foo() is invoked is determined entirely at runtime, i.e., dynamically

• the whole thing was done in Java. Systems which generate bytecode themselves can obviously generate invokedynamic instructions directly as well as generating bytecode on the basis of the behaviour which results

• in the example the bootstrap method produces a single CallSite. In practice bootstrap methods would produce different instances of CallSite depending upon their arguments, and would also use different MethodHandles depending on their arguments.

• the CallSite class can be sub-classed, which enables further specialization of certain aspects of its behavior

• in the example the method actually invoked does not do anything particularly interesting. In practice the behaviour of invoked methods can be dependent on their argument(s), and in the case of systems generating bytecode dynamically, invoked methods can feedback information to the code generation stage.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

InvokeDynamic And MethodHandles On Snow Leopard: Part Three

A Functioning MethodHandle

Switching to the 32-bit VM does indeed result in a fully functioning MethodHandle.

    Macintosh:tmp simon$ build/bsd-i586/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void
    Hello World via a MethodHandle !

One thing to note is the presence of the void qualifier on the call to the invoke() method.

    handle.<void>invoke(System.out, "Hello World via a MethodHandle !\n");

This specifies the return type of the method being invoked and is necessary.

If it is not specified it defaults to Object and in this case you get this.

    Macintosh:tmp simon$ build/bsd-i586/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void
    java.dyn.WrongMethodTypeException: (Ljava/lang/String;)V cannot be called as (Ljava/io/PrintStream;Ljava/lang/String;)Ljava/lang/Object;
	    at DynTest.main(DynTest.java:24)

Equally it is an error if the wrong type is specified, for example,

    handle.<boolean>invoke(System.out, "Hello World via a MethodHandle !\n");

gets you this

    ...

    java.dyn.WrongMethodTypeException: (Ljava/lang/String;)V cannot be called as (Ljava/io/PrintStream;Ljava/lang/String;)Z
	    at DynTest.main(DynTest.java:24)

It is also an error to call a method via a MethodHandle with the wrong number of arguments, or the wrong argument types. In short MethodHandles are type-safe at runtime.

I believe that MethodHandles are also intended to be access-controlled, but at the current time they do not appear to be or not in the ways I would expect them to be. However, this is probably just a function of the state of my particular build and/or the current state of the implementation.

For example, given this class definition

    package other;

    public final class Foo
    {
        public Foo()
        {
        }
	
        //
	
        void localMethod()
        {
            System.out.println("Foo.localMethod()");
        }
	
        private void privateMethod()
        {
            System.out.println("Foo.privateMethod()");
        }
	
        private static void privateStaticMethod()
        {
            System.out.println("Foo.privateStaticMethod()");
        }
    }

I would not have expected this to work.

    package test;

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import other.Foo;

    public final class MethodHandleAccessTestOne
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE);
			
                System.out.print("type == ");
                System.out.println(type);
			
                MethodHandle handle = MethodHandles.lookup().findStatic(Foo.class, "privateStaticMethod", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
			
                handle.invoke();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

but it does

    Macintosh:tmp simon$ build/bsd-i586/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.MethodHandleAccessTestOne
    type == ()void
    handle == privateStaticMethod()void
    Foo.privateStaticMethod()

This does not work

    package test;

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import other.Foo;

    public final class MethodHandleAccessTestTwo
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE);
			
                System.out.print("type == ");
                System.out.println(type);
			
                MethodHandle handle = MethodHandles.lookup().findVirtual(Foo.class, "localMethod", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
			
                Foo foo = new Foo();
			
                System.out.print("foo == ");
                System.out.println(foo);
                handle.invoke(foo);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

but on the basis of the error message I am not really sure if it is not working for the right reason

    Macintosh:tmp simon$ /Users/simon/Scratch/xper/jdk7-bsd/build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.MethodHandleAccessTestTwo
    type == ()void
    handle == Adapted[localMethod](java.lang.Object)void
    foo == other.Foo@19efb05
    java.lang.NoClassDefFoundError: other/Foo
	    at test.MethodHandleAccessTestTwo.main(MethodHandleAccessTestTwo.java:30)

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

InvokeDynamic And MethodHandles On Snow Leopard: Part Two

MethodHandles Considered Magical

Invoking the method referenced by a MethodHandle is suprisingly straight forward. Call its invoke() method.

But its only got one publically accessible method and that is

    public MethodType type()

True, but it turns out that a MethodHandle is sufficiently advanced to be indistinguishable from magic, and automatically acquires an invoke() method with a signature corresponding to the method it references !

So you can do this

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import java.io.PrintStream;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
			
                MethodHandle handle = MethodHandles.lookup().findVirtual(PrintStream.class, "print", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
			
                handle.<void>invoke(System.out, "Hello World via a MethodHandle !\n");
			
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

But Not In The 64-Bit VM

Except that if you try it using the 64-bit VM, you get this.

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void
    Exception in thread "main" java.lang.AbstractMethodError
        at DynTest.main(DynTest.java:31)

The method

    InterpreterGenerator::generate_abstract_entry(void)

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/interpreter_x86_64.cpp

looks as though it may be something to do with this, especially given this method which is defined in the same file

    // Method handle invoker
    // Dispatch a method of the form java.dyn.MethodHandles::invoke(...)
    address InterpreterGenerator::generate_method_handle_entry(void) {
        if (!EnableMethodHandles) {
            return generate_abstract_entry();
        }
        return generate_abstract_entry(); //6815692//
    }

The 32-bit version defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/interpreter_x86_32.cpp

is a bit different.

    // Method handle invoker
    // Dispatch a method of the form java.dyn.MethodHandles::invoke(...)
    address InterpreterGenerator::generate_method_handle_entry(void) {
        if (!EnableMethodHandles) {
            return generate_abstract_entry();
        }

        address entry_point = MethodHandles::generate_method_handle_interpreter_entry(_masm);

        return entry_point;
    }

Replacing the 64-bit version with the 32-bit version in a continuing spirit of gung-ho optimism confirms that this is the indeed the code involved in invoking the method. Unfortunately it does so by crashing.

Its obviously time to build a 32-bit JVM.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

InvokeDynamic And MethodHandles On Snow Leopard

The most interesting feature in JDK 7 for me, at least until the recent reappearance of closures, and probably even then, is the JSR 292 invokedynamic support. It is basically the reason why I started my attempt to build OpenJDK 7 on Snow Leopard.

Ostensibly to use the upcoming JSR 292 features it is necessary to apply a number of patches to a standard OpenJDK 7 release. See here for details. However the current revision of the BSD port seems to contain a lot of the necessary code already so I decided to see how far I could get using that rather than a patched version.

JSR 292 is very much a work-in-progress. There was an early-draft release of the specification almost eighteen months ago, but nothing since. You can see what is going on by looking at the code as it is released, but it is still changing so some or all of the following may change.

Basically JSR 292 defines a new JVM opcode

    invokedynamic

which is intended to provide VM level support for method invocation in languages other than Java which have different invocation semantics. That is a drastic over-simplification but it will do for the moment.

There are also some features visible in Java itself, one of which is method handles, which can be thought of as pointers to individual methods. To obtain a handle to a given method you need an instance of a MethodType, which defines the return type of the method and the parameters it takes. The classes

    MethodHandle

and

    MethodType

are defined in the package

    java.dyn

To determine how they currently work and what else you can do you need to look at the source code and the accompanying comments.

Making A MethodType

The following code is based on my understanding of the currently available revision of the JSR 292 code in the BSD port. Hopefully makes an instance of MethodType specifying a method which returns void and takes a single argument of type String.

    import java.dyn.MethodType;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

Running this code results in the following. Note that although the JSR292 code is present in the current revision of the BSD port it is still deemed to be experimental and has to be unlocked using the command line incantation shown.

    Macintosh:tmp simon$ build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    OpenJDK 64-Bit Server VM warning: JSR 292 method handle code is mismatched to this JVM.  Disabling support.
    type == (java.lang.String)void

An instance of MethodType appears to have been made, but the warning message is not particularly auspicious.

The message originates in the function

    JVM_RegisterMethodHandleMethods(JNIEnv, jclass)

which is in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp

It looks like this (lightly re-formatted by me)

    JVM_ENTRY(void, JVM_RegisterMethodHandleMethods(JNIEnv *env, jclass MHN_class)) {
        assert(MethodHandles::spot_check_entry_names(), "entry enum is OK");

        // note: this explicit warning-producing stuff will be replaced by auto-detection of the JSR 292 classes

        if (!EnableMethodHandles) {
            warning("JSR 292 method handles are disabled in this JVM.  Use -XX:+UnlockExperimentalVMOptions -XX:+EnableMethodHandles to enable.");
            return;  // bind nothing
        }

        bool enable_MH = true;

        {
            ThreadToNativeFromVM ttnfv(thread);

            int status = env->RegisterNatives(MHN_class, methods, sizeof(methods)/sizeof(JNINativeMethod));

            if (env->ExceptionOccurred()) {
                MethodHandles::set_enabled(false);
                warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                enable_MH = false;
                env->ExceptionClear();
            }
        }

        if (enable_MH) {
	
            KlassHandle MHI_klass = SystemDictionaryHandles::MethodHandleImpl_klass();
		
            if (MHI_klass.not_null()) {
      
                symbolHandle raiseException_name   = oopFactory::new_symbol_handle("raiseException", CHECK);
                symbolHandle raiseException_sig    = oopFactory::new_symbol_handle("(ILjava/lang/Object;Ljava/lang/Object;)V", CHECK);
                methodOop    raiseException_method = instanceKlass::cast(MHI_klass->as_klassOop())
                                                         ->find_method(raiseException_name(), raiseException_sig());
													 
                if (raiseException_method != NULL && raiseException_method->is_static()) {
                    MethodHandles::set_raise_exception_method(raiseException_method);
                } else {
                    warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                    enable_MH = false;
                }
            }
        }

        if (enable_MH) {
            MethodHandles::set_enabled(true);
        }

        if (!EnableInvokeDynamic) {
            warning("JSR 292 invokedynamic is disabled in this JVM.  Use -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic to enable.");
            return;  // bind nothing
        }

        {
            ThreadToNativeFromVM ttnfv(thread);

            int status = env->RegisterNatives(MHN_class, methods2, sizeof(methods2)/sizeof(JNINativeMethod));

            if (env->ExceptionOccurred()) {
                MethodHandles::set_enabled(false);
                env->ExceptionDescribe();
                warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                env->ExceptionClear();
            } else {
                MethodHandles::set_enabled(true);
            }
        }
    }
    JVM_END

As can be seen it generates the warning message in three cases. Cases one and three are if the initialization of native methods fail. Case two is if the class

	sun.dyn.MethodHandleImpl

does not have a method of the form

    ???? static void raiseException(int arg1, Object arg2, Object arg3)

This is easy enough to check. The class is defined in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleImpl.java

In fact it does not.

Although it is not possible to know exactly what the method should do, the only use made of it is a bit opaque,
it is certainly possible to add something like this just in case it actually does get invoked.

    public static void raiseException(int arg1, Object arg2, Object arg3)
    {
        System.out.println("MethodHandleImpl.raiseException(" + arg1 + ", " + arg2 + ", " + arg3 + ")");
        throw new RuntimeException("MethodHandleImpl.raiseException(...)");
    }

Trying again following the addition

    Macintosh:tmp simon$ !!
    type == (java.lang.String)void

Getting A MethodHandle

So far so good, so let’s try getting a MethodHandle for a method of that type. There is a suitable one in java.io.PrintStream.

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import java.io.PrintStream;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
				
                MethodHandle handle = MethodHandles.lookup().findVirtual(PrintStream.class, "print", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }
}

Now what happens ?

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    java.lang.InternalError
        at sun.dyn.MethodHandleNatives.init(Native Method)
        at sun.dyn.DirectMethodHandle.(DirectMethodHandle.java:48)
        at sun.dyn.MethodHandleImpl.findMethod(MethodHandleImpl.java:163)
        at java.dyn.MethodHandles$Lookup.findVirtual(MethodHandles.java:233)
        at DynTest.main(DynTest.java:19)

Not so good.

The class

    sun.dyn.MethodHandleNatives

can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleNatives.java

The class has four init native methods.

    /** Initialize the method handle to adapt the call. */
    static native void init(AdapterMethodHandle self, MethodHandle target, int argnum);
    /** Initialize the method handle to call the correct method, directly. */
    static native void init(BoundMethodHandle self, Object target, int argnum);
    /** Initialize the method handle to call as if by an invoke* instruction. */
    static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);

    /** Initialize a method type, once per form. */
    static native void init(MethodType self);

The method is being called from the constructor of the class

    sun.dyn.DirectMethodHandle

which can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/DirectMethodHandle.java

The method is being passed the instance of sun.dyn.DirectMethodHandle being constructed, so it is evidently this variant that is being invoked.

     /** Initialize the method handle to call as if by an invoke* instruction. */
     static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);

The CPU neutral native method handle code can be found in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp

The native method registration information is defined in the following data structure

static JNINativeMethod methods[] = {
  // void init(MemberName self, AccessibleObject ref)
  {CC"init",                    CC"("AMH""MH"I)V",              FN_PTR(MHI_init_AMH)},
  {CC"init",                    CC"("BMH""OBJ"I)V",             FN_PTR(MHI_init_BMH)},
  {CC"init",                    CC"("DMH""OBJ"Z"CLS")V",        FN_PTR(MHI_init_DMH)},
  {CC"init",                    CC"("MT")V",                    FN_PTR(MHI_init_MT)},
  {CC"init",                    CC"("MEM""OBJ")V",              FN_PTR(MHI_init_Mem)},
  {CC"expand",                  CC"("MEM")V",                   FN_PTR(MHI_expand_Mem)},
  {CC"resolve",                 CC"("MEM""CLS")V",              FN_PTR(MHI_resolve_Mem)},
  {CC"getTarget",               CC"("MH"I)"OBJ,                 FN_PTR(MHI_getTarget)},
  {CC"getConstant",             CC"(I)I",                       FN_PTR(MHI_getConstant)},
  //  static native int getNamedCon(int which, Object[] name)
  {CC"getNamedCon",             CC"(I["OBJ")I",                 FN_PTR(MHI_getNamedCon)},
  //  static native int getMembers(Class defc, String matchName, String matchSig,
  //          int matchFlags, Class caller, int skip, MemberName[] results);
  {CC"getMembers",              CC"("CLS""STRG""STRG"I"CLS"I["MEM")I",  FN_PTR(MHI_getMembers)}
};

from which we can deduce, after looking at the macro definitions for CC, DMH, etc., that the function we are interested in is

    MHI_init_DMH

This function throws an InternalError exception without an error message in two different places, and calls a function which also throws an InternalError exception without an error message in two different places, so it is not possible to statically determine what has caused the InternalError.

Some experimentation results in the discovery that the InternalError is being thrown from the function invoked by MHI_init_DMH, namely

    MethodHandles::init_DirectMethodHandle

at this point

    if (me == NULL) { THROW(vmSymbols::java_lang_InternalError()); }

The variable me can be set in four different places in this function

    ...

    me = MethodHandles::entry(MethodHandles::_invokeinterface_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokespecial_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokestatic_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokevirtual_mh);

    ...

Whichever call to MethodHandles::entry(EntryKind) is being made at runtime is clearly returning NULL.

The method MethodHandles::entry(EntryKind) is defined inline in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.hpp

It is doing a straightforward array access indexed by the EntryKind argument

    static MethodHandleEntry* entry(EntryKind ek) { assert(ek_valid(ek), "initialized");
                                                    return _entries[ek]; }

so it is the element in the _entries array corresponding to the EntryKind argument which is NULL.

The _entries array is static and is not directly accessible, it is declared private, and there is only one method which can be used to set its elements

    static void init_entry(EntryKind ek, MethodHandleEntry* me) {
        assert(ek_valid(ek), "oob");
        assert(_entries[ek] == NULL, "no double initialization");
    
        _entries[ek] = me;
    }

This method is only called from one place, the method

    MethodHandles::generate_method_handle_stub

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/methodHandles_x86.cpp

This method in turn is only called from one place, the method

    StubGenerator::generate_all()

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_32.cpp

The code in question looks like this

    ...
	
    if (EnableMethodHandles && SystemDictionary::MethodHandle_klass() != NULL) {
        for (MethodHandles::EntryKind ek = MethodHandles::_EK_FIRST;
             ek < MethodHandles::_EK_LIMIT;
             ek = MethodHandles::EntryKind(1 + (int)ek)) {
            StubCodeMark mark(this, "MethodHandle", MethodHandles::entry_name(ek));
            MethodHandles::generate_method_handle_stub(_masm, ek);
            }
    }

    ...

It is not called from the corresponding version of the method in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_64.cpp

which is the version used in the 64-bit version of the JVM that is running the code. Hence the problem. The array has not been initialized and all the entries are NULL.

The code in the 32-bit version does not appear to be 32-bit specific, and the method MethodHandles::generate_method_handle_stub() is defined in a generic x86 file rather than an x86_32 one so what happens if we graft the code segment above into the 64-bit version of StubGenerator::generate_all() ?

Crash, Bang, Wallop …

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    testtype == (java.lang.String)void
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  SIGSEGV (0xb) at pc=0x000000010281b7d2, pid=13801, tid=4298117120
    #
    # JRE version: 7.0
    # Java VM: OpenJDK 64-Bit Server VM (17.0-b05 mixed mode bsd-amd64 compressed oops)
    # Problematic frame:
    # j  sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
    #
    ...

Possibly not such a good idea after all then.

Having said that, at first glance it is not obvious that this problem is a direct result of problems with the stub generation, especially since it seems to be crashing inside a Java method.

The method in question can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/AdapterMethodHandle.java

and it looks like this.

    public static boolean canPairwiseConvert(MethodType newType, MethodType oldType) {
        // same number of args, of course
        int len = newType.parameterCount();
        if (len != oldType.parameterCount())
            return false;

        // Check return type.  (Not much can be done with it.)
        Class exp = newType.returnType();
        Class ret = oldType.returnType();
        if (!VerifyType.isNullConversion(ret, exp))
            return false;

        // Check args pairwise.
        for (int i = 0; i < len; i++) {
            Class src = newType.parameterType(i); // source type
            Class dst = oldType.parameterType(i); // destination type
            if (!canConvertArgument(src, dst))
                return false;
        }

        return true;
    }

It doesn’t look as though it is doing anything particularly foolhardy.

The crash site is identified in terms of the bytecode rather than the source code, so we need actually need that

    public static boolean canPairwiseConvert(java.dyn.MethodType, java.dyn.MethodType);
    Code:
        0:	aload_0
        1:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I
        4:	istore_2
        5:	iload_2
        6:	aload_1
        7:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I
        10:	if_icmpeq	15
        13:	iconst_0
        14:	ireturn
   
        ...

The pc is seven when the crash occurs which is this instruction

    7:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I

which corresponds to the call

    oldType.parameterCount()

so it looks as though there is something drastically wrong with the value of oldType at this point.

Using the stack trace from the error log

    j  sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
    j  sun.dyn.AdapterMethodHandle.makePairwiseConvert(Lsun/dyn/Access;Ljava/dyn/MethodType;Ljava/dyn/MethodHandle;)Ljava/dyn/MethodHandle;+18
    j  sun.dyn.MethodHandleImpl.findMethod(Lsun/dyn/Access;Lsun/dyn/MemberName;ZLjava/lang/Class;)Ljava/dyn/MethodHandle;+107
    j  java.dyn.MethodHandles$Lookup.findVirtual(Ljava/lang/Class;Ljava/lang/String;Ljava/dyn/MethodType;)Ljava/dyn/MethodHandle;+43
    j  DynTest.main([Ljava/lang/String;)V+34

we can determine that the MethodType value passed as the oldType argument originates from a call to the type() method on an instance of MethodHandle which has been passed to the calling method makePairwiseConvert()

The MethodHandle instance being passed to makePairwiseConvert is a newly created instance of DirectMethodHandle. The instance of MethodType it contains and which is returned by the call to the type() method, is in theory, the instance of MethodType created by the test program. This has already been printed successfully which implies that at that point it was sufficiently valid, or not sufficiently invalid to cause problems. This assumes of cource that it is the object actually being returned from the call to the type() method.

Given that it is extremely difficult to do anything from inside Java to a Java object which can subsequently cause the VM to crash, the problem is most likely to originate somewhere in native code.

As we already know the DirectMethodHandle constructor is invoking the static native method init() in the class MethodHandlesNative. We also know that the native function being called is actually MHI_init_DMH

This function is predominantly performing checks. It does not actually do anything very much to the MethodHandle instance it is passed. The function that it finishes by invoking

    MethodHandles::init_DirectMethodHandle

however, does.

The calls of interest are

    ...
	
    java_dyn_MethodHandle::init_vmslots(mh());

    ...

    sun_dyn_DirectMethodHandle::set_vmtarget(mh(), vmtarget);
    sun_dyn_DirectMethodHandle::set_vmindex(mh(),  vmindex);
	
    ...

    java_dyn_MethodHandle::set_vmentry(mh(), me);
   
    ...

Some experimentation reveals that the value in the instance variable type is changed during the call to

    java_dyn_MethodHandle::set_vmentry()

In fact it is changed to the value of me. It looks very much as though the type instance variable is being over-written with the MethodEntry pointer which is not a pointer to a Java object at all.

The Magical vmentry Field ?

The vm_entry instance variable is defined in the class sun.dyn.MethodHandleImpl as follows.

    private byte       vmentry;    // adapter stub or method entry point

Given that a CPU-level pointer is being stored in vmentry its declaration as a byte is rather curious.

Looking at the definition of set_vmentry() we can see that it is simply placing the pointer to the MethodEntry at _vmentry_offset within the instance which does not itself explain how the instance variable is capable of storing a 32-bit pointer let alone 64-bit one.

Maybe there is something magical about this particular instance variable ?

_vmentry_offset is itself set in the method java_dyn_MethodHandle::compute_offsets() along with the type and vmtarget instance variable offsets

    void java_dyn_MethodHandle::compute_offsets() {

        klassOop k = SystemDictionary::MethodHandle_klass();
  
        if (k != NULL && EnableMethodHandles) {
            compute_offset(_type_offset,      k, vmSymbols::type_name(),      vmSymbols::java_dyn_MethodType_signature(), true);
            compute_offset(_vmtarget_offset,  k, vmSymbols::vmtarget_name(),  vmSymbols::object_signature(), true);
            compute_offset(_vmentry_offset,   k, vmSymbols::vmentry_name(),   vmSymbols::machine_word_signature(), true);

            // Note:  MH.vmslots (if it is present) is a hoisted copy of MH.type.form.vmslots.
            // It is optional pending experiments to keep or toss.
            compute_optional_offset(_vmslots_offset, k, vmSymbols::vmslots_name(), vmSymbols::int_signature(), true);
        }
   }

Looking at the values of these offsets at runtime shows them to be as follows.

    _type_offset     == 24
    _vmtarget_offset == 12
    _vmentry_offset  == 24
    _vmslots_offset  == 0

The type and vmentry offsets are the same so set_vmentry() is indeed clobbering the value in the type instance variable. Something
has gone wrong with the computation of the instance variable offsets, or the instance variable vmentry is insufficently magical in the 64-bit case.

The compute_offset() function simply looks up the field in the class and returns the offset as held by the field descriptor.

The offsets of the fields in an instance of a Java class are computed in the method

    ClassFileParser::parseClassFile()

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/classfile/classFileParser.cpp

This a long and complicated method, its almost a thousand lines long, but approximately half-way down is this

    ...
	
    // adjust the vmentry field declaration in java.dyn.MethodHandle
    if (EnableMethodHandles && class_name() == vmSymbols::sun_dyn_MethodHandleImpl() && class_loader.is_null()) {
      java_dyn_MethodHandle_fix_pre(cp, &fields, &fac, CHECK_(nullHandle));
    }
	
    ...

The Magical vmentry Field Explained

The ClassFileParser::java_dyn_MethodHandle_fix_pre() method is what makes the instance variable vmentry magic. It redefines
its type as either a Java int or long depending upon whether it is a 32-bit or a 64-bit VM.

As an aside despite the name the method is actually operating on the class sun.dyn.MethodHandleImpl not on the class java.dyn.MethodHandle.

It begins by looking for a particular field in the constant pool of the sun.dyn.MethodHandleImpl class.

  ...
  
  int       word_sig_index = 0;
  const int cp_size        = cp->length();
  
  for (int index = 1; index < cp_size; index++) {
      if (cp->tag_at(index).is_utf8() &&
          cp->symbol_at(index) == vmSymbols::machine_word_signature()) {
         word_sig_index = index;
         break;
      }
  }

  ...

The method vmSymbols::machine_word_signature() is defined in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/classfile/vmSymbols.hpp

and its definition is dependent upon whether it is a 32 or a 64 bit field. In the 32 bit case it returns the signature

    I

and in the 64 bit case

    J

If a field with the right signature cannot be found a VirtualMachineError is thrown.

If an appropriate field is found the method then looks for a non-static field of type byte with the name vmentry. If it finds the field, it then re-defines
its type.

    ...

    const int n = (*fields_ptr)()->length();
  
    for (int i = 0; i < n; i += instanceKlass::next_offset) {
        int       name_index = (*fields_ptr)->ushort_at(i + instanceKlass::name_index_offset);
        int       sig_index  = (*fields_ptr)->ushort_at(i + instanceKlass::signature_index_offset);
        int       acc_flags  = (*fields_ptr)->ushort_at(i + instanceKlass::access_flags_offset);
        symbolOop f_name     = cp->symbol_at(name_index);
        symbolOop f_sig      = cp->symbol_at(sig_index);
		
        if (f_sig == vmSymbols::byte_signature() &&
            f_name == vmSymbols::vmentry_name() &&
            (acc_flags & JVM_ACC_STATIC) == 0) {
            // Adjust the field type from byte to an unmanaged pointer.
            assert(fac_ptr->nonstatic_byte_count > 0, "");
			
            fac_ptr->nonstatic_byte_count -= 1;
            (*fields_ptr)->ushort_at_put(
                               i + instanceKlass::signature_index_offset,
                               word_sig_index);
            if (wordSize == jintSize) {
                fac_ptr->nonstatic_word_count += 1;
            } else {
                fac_ptr->nonstatic_double_count += 1;
            }

            FieldAllocationType atype = (FieldAllocationType) (*fields_ptr)->ushort_at(i+4);
			
            assert(atype == NONSTATIC_BYTE, "");
			
            FieldAllocationType new_atype = NONSTATIC_WORD;
      
            if (wordSize > jintSize) {
                if (Universe::field_type_should_be_aligned(T_LONG)) {
                    atype = NONSTATIC_ALIGNED_DOUBLE;
                } else {
                    atype = NONSTATIC_DOUBLE;
                }
            }
            (*fields_ptr)->ushort_at_put(i+4, new_atype);

            found_vmentry = true;
            break;
        }
    }
	
	...

except that in this version of the code there is a bug in the 64-bit case. I didn’t spot it straight away but I did get it eventually, albeit after a lengthy excursion into the mechanics of laying out instance variables in Hotspot.

Found it yet ?

It’s here.

            ...
			
            FieldAllocationType new_atype = NONSTATIC_WORD;
      
            if (wordSize > jintSize) {
                if (Universe::field_type_should_be_aligned(T_LONG)) {
                    atype = NONSTATIC_ALIGNED_DOUBLE;
                } else {
                    atype = NONSTATIC_DOUBLE;
                }
            }
            (*fields_ptr)->ushort_at_put(i+4, new_atype);
			
            ...

In the 64-bit case the field type is not correctly updated, which causes the subsequent field offset problem.

With this fixed the problem goes away.

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void

Now to work out how to actually invoke the method via the MethodHandle, assuming that it is even possible with this version of the code.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.