InvokeDynamic And MethodHandles On Snow Leopard: Part Four

The Even More Magical java.dyn.InvokeDynamic

java.dyn.InvokeDynamic is a final class with a private constructor and no other no declared methods, but you can do this with it

    package test;

    import java.dyn.InvokeDynamic;

    public final class Bar
    {
        void foo()
        {
            System.out.println(InvokeDynamic.bar("bar invoked dynamically"));
        }
    }

and the compiler will compile it to this

    Macintosh:tmp simon$ build/bsd-i586/bin/javap -c test.Bar
    Compiled from "Bar.java"
    public final class test.Bar extends java.lang.Object {
      public test.Bar();
        Code:
          0: aload_0       
          1: invokespecial #1                  // Method java/lang/Object."":()V
          4: return        

      void foo();
        Code:
          0: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
          3: ldc           #3                  // String bar invoked dynamically
          5: invokedynamic #4,  0              // NameAndType bar:(Ljava/lang/String;)Ljava/lang/String;
         10: invokevirtual #5                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
         13: return        
    }

java.dyn.InvokeDynamic is a Java language level marker for the JVM instruction formerly known as

    xxxunusedxxx

and now known as

    invokedynamic

The invokedynamic Instruction

The instruction is five bytes long. The one byte opcode, 186 (0xBA), is followed by two bytes which give the index into the constant pool of a name and type, in this case,

    bar

and

    (Ljava/lang/String;)Ljava/lang/String

followed by two zero bytes.

And that is it. Unlike the original invoke instructions

• invokevirtual
• invokespecial
• invokestatic
• invokeinterface

there is not enough information in the instruction to enable the VM to determine what it should be invoking or how.

Hence, if you run this,

    package test;

    public final class InvokeDynamicTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                Bar bar = new Bar();
			
                bar.foo();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

this happens

    Macintosh:tmp simon$ build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.InvokeDynamicTest
    java.dyn.InvokeDynamicBootstrapError: class has no bootstrap method: class test.Bar
	    at sun.dyn.CallSiteImpl.makeSite(CallSiteImpl.java:63)
	    at test.Bar.foo(Bar.java:10)
	    at test.InvokeDynamicTest.main(InvokeDynamicTest.java:13)

The Bootstrap Method

The first time the VM encounters an invokedynamic instruction it looks for the bootstrap method for the class of the caller.

The bootstrap method is a static method of the form

    static CallSite bootstrapMethod(Class theCallerClass, String theMethodName, MethodType theMethodType)

The bootstrap method for a class is registered either by name or as a MethodHandle using a static method on the class

    java.dyn.Linkage

For example

    ...
	
    static void init()
    {
        Linkage.registerBootstrapMethod(
                    Bar.class, 
                    MethodHandles.
                        lookup().
                            findStatic(
                                Bootstrap.class,
                                "bootstrap",
                                MethodType.make(
                                               CallSite.class, 
                                               Class.class, 
                                               String.class, 
                                               MethodType.class)));
    }
	
    ...

registers the static method

     private static CallSite bootstrap(Class theClass, String theName, MethodType theType)

of the class Bootstrap as the bootstrap method for the class Bar.

The CallSite

The bootstrap method is passed the class of the caller, and the name and type of the method specified by the instance of the invokedynamic instruction, and is responsible for returning an instance of the class

    java.dyn.CallSite

The returned CallSite instance contains a MethodHandle. The JVM uses the method referenced by the MethodHandle as the method to invoke.

Using this as the bootstrap method.

    private static CallSite bootstrap(Class<?> theClass, String theName, MethodType theType)
    {
        MethodHandle mh = MethodHandles.lookup().findStatic(Methods.class, "generic", theType);
        CallSite     cs = new CallSite(theClass, theName, mh.type());
		
        cs.setTarget(mh);
        return cs;
    }

this as the definition of the class Methods

    package test;

    final class Methods
    {
        static String generic(String theArgument)
        {
            return theArgument + ": brought to you by Methods.generic()";
        }
    }

and invoking Bar.foo() like this

    package test;


    public final class InvokeDynamicTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                Bootstrap.init();
			
                Bar bar = new Bar();
			
                bar.foo();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

gets you this.

    Macintosh:tmp simon$ build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.InvokeDynamicTest
    bar invoked dynamically: brought to you by Methods.generic()

While this is not of itself a particularly exciting example there are few things worth noting

• most importantly, what happens when the method Bar.foo() is invoked is determined entirely at runtime, i.e., dynamically

• the whole thing was done in Java. Systems which generate bytecode themselves can obviously generate invokedynamic instructions directly as well as generating bytecode on the basis of the behaviour which results

• in the example the bootstrap method produces a single CallSite. In practice bootstrap methods would produce different instances of CallSite depending upon their arguments, and would also use different MethodHandles depending on their arguments.

• the CallSite class can be sub-classed, which enables further specialization of certain aspects of its behavior

• in the example the method actually invoked does not do anything particularly interesting. In practice the behaviour of invoked methods can be dependent on their argument(s), and in the case of systems generating bytecode dynamically, invoked methods can feedback information to the code generation stage.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

InvokeDynamic And MethodHandles On Snow Leopard: Part Three

A Functioning MethodHandle

Switching to the 32-bit VM does indeed result in a fully functioning MethodHandle.

    Macintosh:tmp simon$ build/bsd-i586/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void
    Hello World via a MethodHandle !

One thing to note is the presence of the void qualifier on the call to the invoke() method.

    handle.<void>invoke(System.out, "Hello World via a MethodHandle !\n");

This specifies the return type of the method being invoked and is necessary.

If it is not specified it defaults to Object and in this case you get this.

    Macintosh:tmp simon$ build/bsd-i586/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void
    java.dyn.WrongMethodTypeException: (Ljava/lang/String;)V cannot be called as (Ljava/io/PrintStream;Ljava/lang/String;)Ljava/lang/Object;
	    at DynTest.main(DynTest.java:24)

Equally it is an error if the wrong type is specified, for example,

    handle.<boolean>invoke(System.out, "Hello World via a MethodHandle !\n");

gets you this

    ...

    java.dyn.WrongMethodTypeException: (Ljava/lang/String;)V cannot be called as (Ljava/io/PrintStream;Ljava/lang/String;)Z
	    at DynTest.main(DynTest.java:24)

It is also an error to call a method via a MethodHandle with the wrong number of arguments, or the wrong argument types. In short MethodHandles are type-safe at runtime.

I believe that MethodHandles are also intended to be access-controlled, but at the current time they do not appear to be or not in the ways I would expect them to be. However, this is probably just a function of the state of my particular build and/or the current state of the implementation.

For example, given this class definition

    package other;

    public final class Foo
    {
        public Foo()
        {
        }
	
        //
	
        void localMethod()
        {
            System.out.println("Foo.localMethod()");
        }
	
        private void privateMethod()
        {
            System.out.println("Foo.privateMethod()");
        }
	
        private static void privateStaticMethod()
        {
            System.out.println("Foo.privateStaticMethod()");
        }
    }

I would not have expected this to work.

    package test;

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import other.Foo;

    public final class MethodHandleAccessTestOne
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE);
			
                System.out.print("type == ");
                System.out.println(type);
			
                MethodHandle handle = MethodHandles.lookup().findStatic(Foo.class, "privateStaticMethod", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
			
                handle.invoke();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

but it does

    Macintosh:tmp simon$ build/bsd-i586/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.MethodHandleAccessTestOne
    type == ()void
    handle == privateStaticMethod()void
    Foo.privateStaticMethod()

This does not work

    package test;

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import other.Foo;

    public final class MethodHandleAccessTestTwo
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE);
			
                System.out.print("type == ");
                System.out.println(type);
			
                MethodHandle handle = MethodHandles.lookup().findVirtual(Foo.class, "localMethod", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
			
                Foo foo = new Foo();
			
                System.out.print("foo == ");
                System.out.println(foo);
                handle.invoke(foo);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

but on the basis of the error message I am not really sure if it is not working for the right reason

    Macintosh:tmp simon$ /Users/simon/Scratch/xper/jdk7-bsd/build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.MethodHandleAccessTestTwo
    type == ()void
    handle == Adapted[localMethod](java.lang.Object)void
    foo == other.Foo@19efb05
    java.lang.NoClassDefFoundError: other/Foo
	    at test.MethodHandleAccessTestTwo.main(MethodHandleAccessTestTwo.java:30)

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

Bootstrapping OpenJDK 7 On Snow Leopard: The 32-Bit Version

The process described here builds a 64-bit version of OpenJDK 7.

The same process can also be used to build a 32-bit version but there are a couple of important differences.

Simply changing the value of

    ARCH_DATA_MODEL

to

    32

will trigger a number of compilation issues whilest building Hotspot. They are only warnings, but the build is set-up to treat all warnings as fatal.

The issues can be fixed such that the warnings go away and the build completes, but unlike the 64-bit version the resulting SDK does not work properly, and is pretty much unusable.

There is, however an easy fix. Using earlier versions of gcc and g++ than the Snow Leopard default versions results in a working SDK.

The versions can be specified using variables passed to make in the build script as follows

    make \
        ARCH_DATA_MODEL=32                         \
        LANG=C                                     \
        ALT_BOOTDIR=$JDK6_HOME                     \
        ALT_FREETYPE_HEADERS_PATH=/usr/X11/include \
        ALT_FREETYPE_LIB_PATH=/usr/X11/lib         \
        NO_DOCS=true                               \
        CC=gcc-4.0                                 \
        CXX=g++-4.0                                \
        $@

where

    $JDK6_HOME

specifies the location of the ersatz JDK6 directory.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

InvokeDynamic And MethodHandles On Snow Leopard: Part Two

MethodHandles Considered Magical

Invoking the method referenced by a MethodHandle is suprisingly straight forward. Call its invoke() method.

But its only got one publically accessible method and that is

    public MethodType type()

True, but it turns out that a MethodHandle is sufficiently advanced to be indistinguishable from magic, and automatically acquires an invoke() method with a signature corresponding to the method it references !

So you can do this

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import java.io.PrintStream;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
			
                MethodHandle handle = MethodHandles.lookup().findVirtual(PrintStream.class, "print", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
			
                handle.<void>invoke(System.out, "Hello World via a MethodHandle !\n");
			
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

But Not In The 64-Bit VM

Except that if you try it using the 64-bit VM, you get this.

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void
    Exception in thread "main" java.lang.AbstractMethodError
        at DynTest.main(DynTest.java:31)

The method

    InterpreterGenerator::generate_abstract_entry(void)

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/interpreter_x86_64.cpp

looks as though it may be something to do with this, especially given this method which is defined in the same file

    // Method handle invoker
    // Dispatch a method of the form java.dyn.MethodHandles::invoke(...)
    address InterpreterGenerator::generate_method_handle_entry(void) {
        if (!EnableMethodHandles) {
            return generate_abstract_entry();
        }
        return generate_abstract_entry(); //6815692//
    }

The 32-bit version defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/interpreter_x86_32.cpp

is a bit different.

    // Method handle invoker
    // Dispatch a method of the form java.dyn.MethodHandles::invoke(...)
    address InterpreterGenerator::generate_method_handle_entry(void) {
        if (!EnableMethodHandles) {
            return generate_abstract_entry();
        }

        address entry_point = MethodHandles::generate_method_handle_interpreter_entry(_masm);

        return entry_point;
    }

Replacing the 64-bit version with the 32-bit version in a continuing spirit of gung-ho optimism confirms that this is the indeed the code involved in invoking the method. Unfortunately it does so by crashing.

Its obviously time to build a 32-bit JVM.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

InvokeDynamic And MethodHandles On Snow Leopard

The most interesting feature in JDK 7 for me, at least until the recent reappearance of closures, and probably even then, is the JSR 292 invokedynamic support. It is basically the reason why I started my attempt to build OpenJDK 7 on Snow Leopard.

Ostensibly to use the upcoming JSR 292 features it is necessary to apply a number of patches to a standard OpenJDK 7 release. See here for details. However the current revision of the BSD port seems to contain a lot of the necessary code already so I decided to see how far I could get using that rather than a patched version.

JSR 292 is very much a work-in-progress. There was an early-draft release of the specification almost eighteen months ago, but nothing since. You can see what is going on by looking at the code as it is released, but it is still changing so some or all of the following may change.

Basically JSR 292 defines a new JVM opcode

    invokedynamic

which is intended to provide VM level support for method invocation in languages other than Java which have different invocation semantics. That is a drastic over-simplification but it will do for the moment.

There are also some features visible in Java itself, one of which is method handles, which can be thought of as pointers to individual methods. To obtain a handle to a given method you need an instance of a MethodType, which defines the return type of the method and the parameters it takes. The classes

    MethodHandle

and

    MethodType

are defined in the package

    java.dyn

To determine how they currently work and what else you can do you need to look at the source code and the accompanying comments.

Making A MethodType

The following code is based on my understanding of the currently available revision of the JSR 292 code in the BSD port. Hopefully makes an instance of MethodType specifying a method which returns void and takes a single argument of type String.

    import java.dyn.MethodType;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

Running this code results in the following. Note that although the JSR292 code is present in the current revision of the BSD port it is still deemed to be experimental and has to be unlocked using the command line incantation shown.

    Macintosh:tmp simon$ build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    OpenJDK 64-Bit Server VM warning: JSR 292 method handle code is mismatched to this JVM.  Disabling support.
    type == (java.lang.String)void

An instance of MethodType appears to have been made, but the warning message is not particularly auspicious.

The message originates in the function

    JVM_RegisterMethodHandleMethods(JNIEnv, jclass)

which is in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp

It looks like this (lightly re-formatted by me)

    JVM_ENTRY(void, JVM_RegisterMethodHandleMethods(JNIEnv *env, jclass MHN_class)) {
        assert(MethodHandles::spot_check_entry_names(), "entry enum is OK");

        // note: this explicit warning-producing stuff will be replaced by auto-detection of the JSR 292 classes

        if (!EnableMethodHandles) {
            warning("JSR 292 method handles are disabled in this JVM.  Use -XX:+UnlockExperimentalVMOptions -XX:+EnableMethodHandles to enable.");
            return;  // bind nothing
        }

        bool enable_MH = true;

        {
            ThreadToNativeFromVM ttnfv(thread);

            int status = env->RegisterNatives(MHN_class, methods, sizeof(methods)/sizeof(JNINativeMethod));

            if (env->ExceptionOccurred()) {
                MethodHandles::set_enabled(false);
                warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                enable_MH = false;
                env->ExceptionClear();
            }
        }

        if (enable_MH) {
	
            KlassHandle MHI_klass = SystemDictionaryHandles::MethodHandleImpl_klass();
		
            if (MHI_klass.not_null()) {
      
                symbolHandle raiseException_name   = oopFactory::new_symbol_handle("raiseException", CHECK);
                symbolHandle raiseException_sig    = oopFactory::new_symbol_handle("(ILjava/lang/Object;Ljava/lang/Object;)V", CHECK);
                methodOop    raiseException_method = instanceKlass::cast(MHI_klass->as_klassOop())
                                                         ->find_method(raiseException_name(), raiseException_sig());
													 
                if (raiseException_method != NULL && raiseException_method->is_static()) {
                    MethodHandles::set_raise_exception_method(raiseException_method);
                } else {
                    warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                    enable_MH = false;
                }
            }
        }

        if (enable_MH) {
            MethodHandles::set_enabled(true);
        }

        if (!EnableInvokeDynamic) {
            warning("JSR 292 invokedynamic is disabled in this JVM.  Use -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic to enable.");
            return;  // bind nothing
        }

        {
            ThreadToNativeFromVM ttnfv(thread);

            int status = env->RegisterNatives(MHN_class, methods2, sizeof(methods2)/sizeof(JNINativeMethod));

            if (env->ExceptionOccurred()) {
                MethodHandles::set_enabled(false);
                env->ExceptionDescribe();
                warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                env->ExceptionClear();
            } else {
                MethodHandles::set_enabled(true);
            }
        }
    }
    JVM_END

As can be seen it generates the warning message in three cases. Cases one and three are if the initialization of native methods fail. Case two is if the class

	sun.dyn.MethodHandleImpl

does not have a method of the form

    ???? static void raiseException(int arg1, Object arg2, Object arg3)

This is easy enough to check. The class is defined in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleImpl.java

In fact it does not.

Although it is not possible to know exactly what the method should do, the only use made of it is a bit opaque,
it is certainly possible to add something like this just in case it actually does get invoked.

    public static void raiseException(int arg1, Object arg2, Object arg3)
    {
        System.out.println("MethodHandleImpl.raiseException(" + arg1 + ", " + arg2 + ", " + arg3 + ")");
        throw new RuntimeException("MethodHandleImpl.raiseException(...)");
    }

Trying again following the addition

    Macintosh:tmp simon$ !!
    type == (java.lang.String)void

Getting A MethodHandle

So far so good, so let’s try getting a MethodHandle for a method of that type. There is a suitable one in java.io.PrintStream.

    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import java.io.PrintStream;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
				
                MethodHandle handle = MethodHandles.lookup().findVirtual(PrintStream.class, "print", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }
}

Now what happens ?

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    java.lang.InternalError
        at sun.dyn.MethodHandleNatives.init(Native Method)
        at sun.dyn.DirectMethodHandle.(DirectMethodHandle.java:48)
        at sun.dyn.MethodHandleImpl.findMethod(MethodHandleImpl.java:163)
        at java.dyn.MethodHandles$Lookup.findVirtual(MethodHandles.java:233)
        at DynTest.main(DynTest.java:19)

Not so good.

The class

    sun.dyn.MethodHandleNatives

can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleNatives.java

The class has four init native methods.

    /** Initialize the method handle to adapt the call. */
    static native void init(AdapterMethodHandle self, MethodHandle target, int argnum);
    /** Initialize the method handle to call the correct method, directly. */
    static native void init(BoundMethodHandle self, Object target, int argnum);
    /** Initialize the method handle to call as if by an invoke* instruction. */
    static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);

    /** Initialize a method type, once per form. */
    static native void init(MethodType self);

The method is being called from the constructor of the class

    sun.dyn.DirectMethodHandle

which can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/DirectMethodHandle.java

The method is being passed the instance of sun.dyn.DirectMethodHandle being constructed, so it is evidently this variant that is being invoked.

     /** Initialize the method handle to call as if by an invoke* instruction. */
     static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);

The CPU neutral native method handle code can be found in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp

The native method registration information is defined in the following data structure

static JNINativeMethod methods[] = {
  // void init(MemberName self, AccessibleObject ref)
  {CC"init",                    CC"("AMH""MH"I)V",              FN_PTR(MHI_init_AMH)},
  {CC"init",                    CC"("BMH""OBJ"I)V",             FN_PTR(MHI_init_BMH)},
  {CC"init",                    CC"("DMH""OBJ"Z"CLS")V",        FN_PTR(MHI_init_DMH)},
  {CC"init",                    CC"("MT")V",                    FN_PTR(MHI_init_MT)},
  {CC"init",                    CC"("MEM""OBJ")V",              FN_PTR(MHI_init_Mem)},
  {CC"expand",                  CC"("MEM")V",                   FN_PTR(MHI_expand_Mem)},
  {CC"resolve",                 CC"("MEM""CLS")V",              FN_PTR(MHI_resolve_Mem)},
  {CC"getTarget",               CC"("MH"I)"OBJ,                 FN_PTR(MHI_getTarget)},
  {CC"getConstant",             CC"(I)I",                       FN_PTR(MHI_getConstant)},
  //  static native int getNamedCon(int which, Object[] name)
  {CC"getNamedCon",             CC"(I["OBJ")I",                 FN_PTR(MHI_getNamedCon)},
  //  static native int getMembers(Class defc, String matchName, String matchSig,
  //          int matchFlags, Class caller, int skip, MemberName[] results);
  {CC"getMembers",              CC"("CLS""STRG""STRG"I"CLS"I["MEM")I",  FN_PTR(MHI_getMembers)}
};

from which we can deduce, after looking at the macro definitions for CC, DMH, etc., that the function we are interested in is

    MHI_init_DMH

This function throws an InternalError exception without an error message in two different places, and calls a function which also throws an InternalError exception without an error message in two different places, so it is not possible to statically determine what has caused the InternalError.

Some experimentation results in the discovery that the InternalError is being thrown from the function invoked by MHI_init_DMH, namely

    MethodHandles::init_DirectMethodHandle

at this point

    if (me == NULL) { THROW(vmSymbols::java_lang_InternalError()); }

The variable me can be set in four different places in this function

    ...

    me = MethodHandles::entry(MethodHandles::_invokeinterface_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokespecial_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokestatic_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokevirtual_mh);

    ...

Whichever call to MethodHandles::entry(EntryKind) is being made at runtime is clearly returning NULL.

The method MethodHandles::entry(EntryKind) is defined inline in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.hpp

It is doing a straightforward array access indexed by the EntryKind argument

    static MethodHandleEntry* entry(EntryKind ek) { assert(ek_valid(ek), "initialized");
                                                    return _entries[ek]; }

so it is the element in the _entries array corresponding to the EntryKind argument which is NULL.

The _entries array is static and is not directly accessible, it is declared private, and there is only one method which can be used to set its elements

    static void init_entry(EntryKind ek, MethodHandleEntry* me) {
        assert(ek_valid(ek), "oob");
        assert(_entries[ek] == NULL, "no double initialization");
    
        _entries[ek] = me;
    }

This method is only called from one place, the method

    MethodHandles::generate_method_handle_stub

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/methodHandles_x86.cpp

This method in turn is only called from one place, the method

    StubGenerator::generate_all()

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_32.cpp

The code in question looks like this

    ...
	
    if (EnableMethodHandles && SystemDictionary::MethodHandle_klass() != NULL) {
        for (MethodHandles::EntryKind ek = MethodHandles::_EK_FIRST;
             ek < MethodHandles::_EK_LIMIT;
             ek = MethodHandles::EntryKind(1 + (int)ek)) {
            StubCodeMark mark(this, "MethodHandle", MethodHandles::entry_name(ek));
            MethodHandles::generate_method_handle_stub(_masm, ek);
            }
    }

    ...

It is not called from the corresponding version of the method in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_64.cpp

which is the version used in the 64-bit version of the JVM that is running the code. Hence the problem. The array has not been initialized and all the entries are NULL.

The code in the 32-bit version does not appear to be 32-bit specific, and the method MethodHandles::generate_method_handle_stub() is defined in a generic x86 file rather than an x86_32 one so what happens if we graft the code segment above into the 64-bit version of StubGenerator::generate_all() ?

Crash, Bang, Wallop …

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    testtype == (java.lang.String)void
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  SIGSEGV (0xb) at pc=0x000000010281b7d2, pid=13801, tid=4298117120
    #
    # JRE version: 7.0
    # Java VM: OpenJDK 64-Bit Server VM (17.0-b05 mixed mode bsd-amd64 compressed oops)
    # Problematic frame:
    # j  sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
    #
    ...

Possibly not such a good idea after all then.

Having said that, at first glance it is not obvious that this problem is a direct result of problems with the stub generation, especially since it seems to be crashing inside a Java method.

The method in question can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/AdapterMethodHandle.java

and it looks like this.

    public static boolean canPairwiseConvert(MethodType newType, MethodType oldType) {
        // same number of args, of course
        int len = newType.parameterCount();
        if (len != oldType.parameterCount())
            return false;

        // Check return type.  (Not much can be done with it.)
        Class exp = newType.returnType();
        Class ret = oldType.returnType();
        if (!VerifyType.isNullConversion(ret, exp))
            return false;

        // Check args pairwise.
        for (int i = 0; i < len; i++) {
            Class src = newType.parameterType(i); // source type
            Class dst = oldType.parameterType(i); // destination type
            if (!canConvertArgument(src, dst))
                return false;
        }

        return true;
    }

It doesn’t look as though it is doing anything particularly foolhardy.

The crash site is identified in terms of the bytecode rather than the source code, so we need actually need that

    public static boolean canPairwiseConvert(java.dyn.MethodType, java.dyn.MethodType);
    Code:
        0:	aload_0
        1:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I
        4:	istore_2
        5:	iload_2
        6:	aload_1
        7:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I
        10:	if_icmpeq	15
        13:	iconst_0
        14:	ireturn
   
        ...

The pc is seven when the crash occurs which is this instruction

    7:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I

which corresponds to the call

    oldType.parameterCount()

so it looks as though there is something drastically wrong with the value of oldType at this point.

Using the stack trace from the error log

    j  sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
    j  sun.dyn.AdapterMethodHandle.makePairwiseConvert(Lsun/dyn/Access;Ljava/dyn/MethodType;Ljava/dyn/MethodHandle;)Ljava/dyn/MethodHandle;+18
    j  sun.dyn.MethodHandleImpl.findMethod(Lsun/dyn/Access;Lsun/dyn/MemberName;ZLjava/lang/Class;)Ljava/dyn/MethodHandle;+107
    j  java.dyn.MethodHandles$Lookup.findVirtual(Ljava/lang/Class;Ljava/lang/String;Ljava/dyn/MethodType;)Ljava/dyn/MethodHandle;+43
    j  DynTest.main([Ljava/lang/String;)V+34

we can determine that the MethodType value passed as the oldType argument originates from a call to the type() method on an instance of MethodHandle which has been passed to the calling method makePairwiseConvert()

The MethodHandle instance being passed to makePairwiseConvert is a newly created instance of DirectMethodHandle. The instance of MethodType it contains and which is returned by the call to the type() method, is in theory, the instance of MethodType created by the test program. This has already been printed successfully which implies that at that point it was sufficiently valid, or not sufficiently invalid to cause problems. This assumes of cource that it is the object actually being returned from the call to the type() method.

Given that it is extremely difficult to do anything from inside Java to a Java object which can subsequently cause the VM to crash, the problem is most likely to originate somewhere in native code.

As we already know the DirectMethodHandle constructor is invoking the static native method init() in the class MethodHandlesNative. We also know that the native function being called is actually MHI_init_DMH

This function is predominantly performing checks. It does not actually do anything very much to the MethodHandle instance it is passed. The function that it finishes by invoking

    MethodHandles::init_DirectMethodHandle

however, does.

The calls of interest are

    ...
	
    java_dyn_MethodHandle::init_vmslots(mh());

    ...

    sun_dyn_DirectMethodHandle::set_vmtarget(mh(), vmtarget);
    sun_dyn_DirectMethodHandle::set_vmindex(mh(),  vmindex);
	
    ...

    java_dyn_MethodHandle::set_vmentry(mh(), me);
   
    ...

Some experimentation reveals that the value in the instance variable type is changed during the call to

    java_dyn_MethodHandle::set_vmentry()

In fact it is changed to the value of me. It looks very much as though the type instance variable is being over-written with the MethodEntry pointer which is not a pointer to a Java object at all.

The Magical vmentry Field ?

The vm_entry instance variable is defined in the class sun.dyn.MethodHandleImpl as follows.

    private byte       vmentry;    // adapter stub or method entry point

Given that a CPU-level pointer is being stored in vmentry its declaration as a byte is rather curious.

Looking at the definition of set_vmentry() we can see that it is simply placing the pointer to the MethodEntry at _vmentry_offset within the instance which does not itself explain how the instance variable is capable of storing a 32-bit pointer let alone 64-bit one.

Maybe there is something magical about this particular instance variable ?

_vmentry_offset is itself set in the method java_dyn_MethodHandle::compute_offsets() along with the type and vmtarget instance variable offsets

    void java_dyn_MethodHandle::compute_offsets() {

        klassOop k = SystemDictionary::MethodHandle_klass();
  
        if (k != NULL && EnableMethodHandles) {
            compute_offset(_type_offset,      k, vmSymbols::type_name(),      vmSymbols::java_dyn_MethodType_signature(), true);
            compute_offset(_vmtarget_offset,  k, vmSymbols::vmtarget_name(),  vmSymbols::object_signature(), true);
            compute_offset(_vmentry_offset,   k, vmSymbols::vmentry_name(),   vmSymbols::machine_word_signature(), true);

            // Note:  MH.vmslots (if it is present) is a hoisted copy of MH.type.form.vmslots.
            // It is optional pending experiments to keep or toss.
            compute_optional_offset(_vmslots_offset, k, vmSymbols::vmslots_name(), vmSymbols::int_signature(), true);
        }
   }

Looking at the values of these offsets at runtime shows them to be as follows.

    _type_offset     == 24
    _vmtarget_offset == 12
    _vmentry_offset  == 24
    _vmslots_offset  == 0

The type and vmentry offsets are the same so set_vmentry() is indeed clobbering the value in the type instance variable. Something
has gone wrong with the computation of the instance variable offsets, or the instance variable vmentry is insufficently magical in the 64-bit case.

The compute_offset() function simply looks up the field in the class and returns the offset as held by the field descriptor.

The offsets of the fields in an instance of a Java class are computed in the method

    ClassFileParser::parseClassFile()

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/classfile/classFileParser.cpp

This a long and complicated method, its almost a thousand lines long, but approximately half-way down is this

    ...
	
    // adjust the vmentry field declaration in java.dyn.MethodHandle
    if (EnableMethodHandles && class_name() == vmSymbols::sun_dyn_MethodHandleImpl() && class_loader.is_null()) {
      java_dyn_MethodHandle_fix_pre(cp, &fields, &fac, CHECK_(nullHandle));
    }
	
    ...

The Magical vmentry Field Explained

The ClassFileParser::java_dyn_MethodHandle_fix_pre() method is what makes the instance variable vmentry magic. It redefines
its type as either a Java int or long depending upon whether it is a 32-bit or a 64-bit VM.

As an aside despite the name the method is actually operating on the class sun.dyn.MethodHandleImpl not on the class java.dyn.MethodHandle.

It begins by looking for a particular field in the constant pool of the sun.dyn.MethodHandleImpl class.

  ...
  
  int       word_sig_index = 0;
  const int cp_size        = cp->length();
  
  for (int index = 1; index < cp_size; index++) {
      if (cp->tag_at(index).is_utf8() &&
          cp->symbol_at(index) == vmSymbols::machine_word_signature()) {
         word_sig_index = index;
         break;
      }
  }

  ...

The method vmSymbols::machine_word_signature() is defined in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/classfile/vmSymbols.hpp

and its definition is dependent upon whether it is a 32 or a 64 bit field. In the 32 bit case it returns the signature

    I

and in the 64 bit case

    J

If a field with the right signature cannot be found a VirtualMachineError is thrown.

If an appropriate field is found the method then looks for a non-static field of type byte with the name vmentry. If it finds the field, it then re-defines
its type.

    ...

    const int n = (*fields_ptr)()->length();
  
    for (int i = 0; i < n; i += instanceKlass::next_offset) {
        int       name_index = (*fields_ptr)->ushort_at(i + instanceKlass::name_index_offset);
        int       sig_index  = (*fields_ptr)->ushort_at(i + instanceKlass::signature_index_offset);
        int       acc_flags  = (*fields_ptr)->ushort_at(i + instanceKlass::access_flags_offset);
        symbolOop f_name     = cp->symbol_at(name_index);
        symbolOop f_sig      = cp->symbol_at(sig_index);
		
        if (f_sig == vmSymbols::byte_signature() &&
            f_name == vmSymbols::vmentry_name() &&
            (acc_flags & JVM_ACC_STATIC) == 0) {
            // Adjust the field type from byte to an unmanaged pointer.
            assert(fac_ptr->nonstatic_byte_count > 0, "");
			
            fac_ptr->nonstatic_byte_count -= 1;
            (*fields_ptr)->ushort_at_put(
                               i + instanceKlass::signature_index_offset,
                               word_sig_index);
            if (wordSize == jintSize) {
                fac_ptr->nonstatic_word_count += 1;
            } else {
                fac_ptr->nonstatic_double_count += 1;
            }

            FieldAllocationType atype = (FieldAllocationType) (*fields_ptr)->ushort_at(i+4);
			
            assert(atype == NONSTATIC_BYTE, "");
			
            FieldAllocationType new_atype = NONSTATIC_WORD;
      
            if (wordSize > jintSize) {
                if (Universe::field_type_should_be_aligned(T_LONG)) {
                    atype = NONSTATIC_ALIGNED_DOUBLE;
                } else {
                    atype = NONSTATIC_DOUBLE;
                }
            }
            (*fields_ptr)->ushort_at_put(i+4, new_atype);

            found_vmentry = true;
            break;
        }
    }
	
	...

except that in this version of the code there is a bug in the 64-bit case. I didn’t spot it straight away but I did get it eventually, albeit after a lengthy excursion into the mechanics of laying out instance variables in Hotspot.

Found it yet ?

It’s here.

            ...
			
            FieldAllocationType new_atype = NONSTATIC_WORD;
      
            if (wordSize > jintSize) {
                if (Universe::field_type_should_be_aligned(T_LONG)) {
                    atype = NONSTATIC_ALIGNED_DOUBLE;
                } else {
                    atype = NONSTATIC_DOUBLE;
                }
            }
            (*fields_ptr)->ushort_at_put(i+4, new_atype);
			
            ...

In the 64-bit case the field type is not correctly updated, which causes the subsequent field offset problem.

With this fixed the problem goes away.

    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void

Now to work out how to actually invoke the method via the MethodHandle, assuming that it is even possible with this version of the code.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

Bootstrapping OpenJDK 7 On Snow Leopard

If you want to experiment with OpenJDK 7 on MacOS X you currently have no choice but to build it from scratch. This requires JDK 6 which is OK if you are running Snow Leopard.

The source is available as a zip or via Mercurial from here.

.

[Updated: A clarification. To build on Snow Leopard you need the BSD port which is here

       http://hg.openjdk.java.net/bsd-port/bsd-port

]

The build process requires a few variables to be set, so, for convenience, I bundled them up in a script like this.

    
    make \
        ARCH_DATA_MODEL=64                                                          \
        LANG=C                                                                      \
        ALT_BOOTDIR=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home \
        ALT_FREETYPE_HEADERS_PATH=/usr/X11/include                                  \
        ALT_FREETYPE_LIB_PATH=/usr/X11/lib                                          \
        NO_DOCS=true
	

The most important is

    ALT_BOOTDIR

which identifies the home directory of JDK 6 that the build process is going to use to boot with.

This points at the version of JDK 6 which ships with Snow Leopard.

Running the script fairly quickly gets you this

    
    [javac] [...]/ClassReader.java:860: reference to Version is ambiguous, both  ... and  ... match
    [javac]         AttributeReader(Name name, Version version, Set<AttributeKind> kinds) {
    [javac]                                    ^
    [javac] [...]/ClassReader.java:873: reference to Version is ambiguous, both ... and ... match
    [javac]         final Version version;
    [javac]               ^
   

I’ve elided the full path names and the classes for clarity.

The file being compiled is

    $(JDK7_ROOT)/langtools/src/share/classes/com/sun/tools/javac/jvm/ClassReader.java

The two matching classes are

    com.sun.tools.javac.jvm.ClassFile.Version

and

    com.sun.tools.javac.util.Version

The class com.sun.tools.javac.util.Version is being picked up by a wildcard import.

    import com.sun.tools.javac.util.*;

and com.sun.tools.javac.jvm.ClassFile.Version appears in a static import

    import static com.sun.tools.javac.jvm.ClassFile.Version.*;

There is no source for com.sun.tools.javac.util.Version so it looks as though javac has found its own version. Possibly this is a function of the version of javac that ships with Snow Leopard ? Rather than get bogged down at this point I simply edited the source to disambiguate the two uses.

Restarting the build results in a successful make in the langtools directory, but then it falls over again, this time while attempting to build a native library somewhere in the corba directory.

The actual error message is

    ld: library not found for -ljvm

Looking at the offending link command shows this

    -L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/jre/lib/amd64 -ljava

and

    -L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/jre/lib/amd64/server -ljvm

twice for some reason. Be that as it may, the problem is that neither directory exists. In the Snow Leopard version of JDK 6 all the libraries are all kept somewhere else, in

   /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries

in fact.

At this point you can choose to tangle with the makefiles down in the corba directory, mess around with the structure of Apple’s JDK 6 release, or do what I did and set up a new JDK 6 home just for building OpenJDK 7.

Initially the top level directory looks like this.

Macintosh:jdk6home simon$ ls -la
total 8
drwxr-xr-x  4 simon  simon  136 16 Nov 09:33 .
drwxr-xr-x  7 simon  simon  238 16 Nov 09:32 ..
lrwxr-xr-x  1 simon  simon   67 16 Nov 09:33 bin -> /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin
drwxr-xr-x  3 simon  simon  102 16 Nov 09:33 jre

jre/lib/amd64 like this

Macintosh:amd64 simon$ ls -la
total 8
drwxr-xr-x  4 simon  simon  136 16 Nov 15:26 .
drwxr-xr-x  4 simon  simon  136 16 Nov 09:38 ..
lrwxr-xr-x  1 simon  simon   83 16 Nov 09:38 libjava.dylib -> /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries/libjava.jnilib
drwxr-xr-x  3 simon  simon  102 16 Nov 09:38 server

and jre/lib/amd64/server like this

Macintosh:server simon$ ls -la
total 8
drwxr-xr-x  3 simon  simon  102 16 Nov 09:38 .
drwxr-xr-x  4 simon  simon  136 16 Nov 15:26 ..
lrwxr-xr-x  1 simon  simon   81 16 Nov 09:33 libjvm.dylib -> /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries/libjvm.dylib

Changing ALT_BOOTDIR to point at the new ersatz JDK 6 home and restarting the build results in the successful completion of the make in the

• corba,
• jaxp, and
• jaxws

directories, followed, eventually, by this failure, while building in the hotspot directory.

    Missing [...]/jdk6home/lib/tools.jar file. Use 1.6.0 or later version of JDK

Apple JDK 6 doesn’t have a tools.jar but it does have a classes.jar in

    /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes

which seems to contain all of the com.sun.tools packages.

Creating a lib directory and adding a symbolic link to classes.jar as tools.jar seems to do the trick and the build gets a bit further before hitting another problem.

    [...]hotspot/agent/src/os/bsd/StubDebuggerLocal.c:26:17: error: jni.h: No such file or directory

As before symbolic linking is our friend. This time to

	/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Headers/

Having fixed this the hotspot build eventually, it takes a while, completes and then tries to run a sanity test

    All done.
    cd bsd_amd64_compiler2/product && ./test_gamma
    java full version "1.6.0_15-b03-219"
    There was an error trying to initialize the HPI library.
    Could not create the Java virtual machine.

It turns out that this problem, unlike the previous ones, cannot be solved by a sleight of directory so to speak.

     test_gamma

is a shell script generated during the build process that compiles a Java file and then runs something called

     gamma

on it.

gamma is built from launcher.c which simply includes the files

    java.c

and

    java_md.c

These are OS specific files. In this case they are the BSD versions and can be found in

    $(JDK7_ROOT)/hotspot/src/os/bsd/launcher

What gamma does is establish the correct environment for the JVM and then, as you can deduce from the error message, run it. Part of the JVM environment as implemented by Sun is the HPI library alluded to in the error message.

This is dynamically loaded by the method

    void hpi::initialize_get_interface(vm_calls_t *callbacks)

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/os/bsd/vm/hpi_bsd.cpp

except that it is not loaded because it does not exist. Hence the error message.

HPI stands for Host Porting Interface. It abstracts out a number of low-level OS specific operations such as DLL, file, and socket access. It is effectively part of the canonical model of how the Sun JVM is implemented.

The problem is that the Apple version of JDK 6 does not have a library called

    libhpi.dylib

nor anything that looks like as though it is standalone replacement for it.

At this point the JVM, or at least a version of it has been built. Now an attempt is being made to test it by running it in conjunction with the rest of the VM runtime from the JDK identified by ALT_BOOTDIR. It uses the boot version because the rest of the correct runtime has not been built yet, because some of it is dependent on the JVM.

So what happens if we ignore the test by commenting out the contents of

    $(JDK7_ROOT)/build/bsd-amd64/hotspot/outputdir/bsd_amd64_compiler2/product/test_gamma

and carry on regardless ?

Eventually, and it really is quite a long eventually, we get this

...
>>>Making sec-files-win @ Tue Nov 17 08:04:08 GMT 2009 ...
>>>Making jgss-files @ Tue Nov 17 08:04:08 GMT 2009 ...
>>>Finished making images @ Tue Nov 17 08:04:08 GMT 2009 ...
########################################################################
##### Leaving  jdk for target  sanity all  images                  #####
########################################################################
########################################################################

Control bsd amd64 1.7.0-internal build_product_image build finished: 
Control bsd amd64 1.7.0-internal all_product_build build finished: 
Control bsd amd64 1.7.0-internal all build finished: 

Does the result of all this work ? Well, I don’t have access to the TCK or anything, so I cannot say with any certainty that it is a bona fide JDK 7, but a couple of basic sanity tests seem to show that javac and java are doing the right thing, and it can for example run the Equinox OSGI framework.

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

An Extended NewAnnotation Wizard For Eclipse: Defining The NewWizard Extension

To make Eclipse aware that the fragment contains a wizard we have to define an extension corresponding to the Eclipse defined extension point

    org.eclipse.ui.newWizards

As it says in the documentation for this extension point, which can be found here Platform Plug-in Developer Guide:Reference:Extension Points Reference:org.eclipse.ui.newWizards

This extension point is used to register resource creation wizard extensions. Creation wizards appear as choices within the “New Dialog”, and are typically used to create folders and files.

In the “New Dialog”, wizards are organized into categories which usually reflect a particular problem domain. For instance, a Java oriented plugin may define a category called “Java” which is appropriate for “Class” or “Package” creation wizards. The categories defined by one plug-in can be referenced by other plug-ins using the category attribute. Uncategorized wizards, as well as wizards with invalid category paths, will end up in an “Other” category.

An extension can be defined by simply adding a piece of XML conforming to the schema for the corresponding extension point to the fragment.xml file.

There are two ways to do this. The first is to use the ‘Extensions’ tab of the ‘Fragment’ view which you can bring-ip by double-clicking the MANIFEST.MF file in the ‘Package Explorer’ view.

Click Add to bring up the ‘New Extension’ dialog

Enter org.eclipse.ui.newWizards in the ‘Extension Pointer filter’ text field. It will appear as the sole element in the list below it. Select it and hit ‘Finish’.

Control-click or right-click the org.eclipse.ui.newWizards element then select New > wizard from the pop-menu.

Then fill in the fields of the resulting editor.

Alternatively you can create a file called fragment.xml in the root directory of the project and put something like this in it.

    <?xml version="1.0" encoding="UTF-8"?>
    <?eclipse version="3.4"?>
    <fragment>
        <extension
            point = "org.eclipse.ui.newWizards">
            <wizard
                name     = "%NewAnnotationType.label"
                icon     = "$nl$/icons/full/etool16/newannotation_wiz.gif"
                category = "org.eclipse.jdt.ui.java"
                id       = "xper.eclipse.plugin.jdt.ui.wizards.NewAnnotationCreationWizard">
                <class 
                    class = "xper.eclipse.plugin.jdt.ui.annotation.NewAnnotationWizard">
                    <parameter 
                        name  = "javatype" 
                        value = "true"/>
                </class>
            </wizard>
        </extension>
    </fragment>

In this case, the name is the same as that of the Eclipse supplied wizard and is in fact a property. The existing value is accessible because the fragment is hosted by the bundle defining. Using the same value however results in two wizards with the same name appearing in menus, etc., which is a bit confusing so I defined a new value in the fragment.properties file.

The fragment.properties file should be in the root directory of the project. In this case it simply contains a single entry.

    NewAnnotationType.label = Annotation [extended]

The icon is that of the Eclipse suppiied wizard and is accessible for the same reason as the name.

The category is the same as that for the Eclipse supplied Java type creation wizards which ensures that the new wizard appears in all the same places that they do.

Note that If you do create the fragment.xml file by hand, and/or a fragment.properties file you will need to add them to the list of files to be included in the binary build, which is configurable from the ‘Build ‘ tab of the ‘Fragment’ view.

Running the fragment using the Eclipse Application configuration results in the new wizard being available. For example, from the drop-down menu of the Java creation button

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved

An Extended NewAnnotation Wizard For Eclipse: Creating The Annotation

By default, for a class inheriting from NewTypeWizardPage, the creation of the Java type is performed by the inherited method

    public void createType(IProgressMonitor monitor)

This method in turn calls

    protected void createTypeMembers(
                       IType            newType,
                       ImportsManager   imports,
                       IProgressMonitor monitor)
                   throws 
                       CoreException

which is documented as a hook method, and it is this method which the example in the documentation overrides.

To create the annotated definition we need to add the referenced Annotation types and Enums to the list of imports and annotate the created Annotation. Both the created type, in this case the Annotation, and the ImportsManager are being passed to the hook method, so it looks as though it should be possible to do what is necessary by overriding this method.

Adding the imports to the ImportsManager is easily done.

        ...

        if (documented)
        {
            imports.addImport(DOCUMENTED_ANNOTATION_TYPE);
        }
        if (inherited)
        {
            imports.addImport(INHERITED_ANNOTATION_TYPE);
        }
        switch (retention)
        {
            case CLASS:
				
                break;
				
                default:
				
                    imports.addImport(RETENTION_ANNOTATION_TYPE);
                    imports.addImport(RETENTION_POLICY_ENUM_TYPE);
        }
        if (targets.size() != 0)
        {
            imports.addImport(TARGET_ANNOTATION_TYPE);
            imports.addImport(ELEMENT_TYPE_ENUM_TYPE);
        }

        ...

with the constants being defined as follows

        private static final String DOCUMENTED_ANNOTATION_TYPE	= "java.lang.annotation.Documented";
	
        private static final String INHERITED_ANNOTATION_TYPE	= "java.lang.annotation.Inherited";
	
        private static final String RETENTION_ANNOTATION_TYPE	= "java.lang.annotation.Retention";
	
        private static final String TARGET_ANNOTATION_TYPE      = "java.lang.annotation.Target";
	
        private static final String ELEMENT_TYPE_ENUM_TYPE      = "java.lang.annotation.ElementType";
	
        private static final String RETENTION_POLICY_ENUM_TYPE	= "java.lang.annotation.RetentionPolicy";

For the Documented and Inherited meta-annotations only the Annotation types themselves need to be imported. In the Retention and Target cases the Enum value types are also imported.

Adding the Annotations themselves to the newly created type turns out to be more difficult. The IType interface extends the IAnnotable amongst others, but none of the methods on IAnnotable or any of the super-interfaces, or on IType itself apparently enable the addition of Annotations.

It is entirely possible that somewhere in the maze of twisty little interfaces there is a way to do so, but I have yet to stumble upon it.

In the end, following a good deal of random experimentation I discovered the

    protected String constructCUContent(
                         ICompilationUnit cu,
                         String           typeContent,
                         String           lineDelimiter)
                     throws 
                         CoreException
 

method on NewTypeWizardPage.

The typeContent argument contains the source of the annotation as created.

Overriding this method modifying the typeContent argument and then invoking the overridden method turns out to have the required effect.

        @Override
        protected String constructCUContent(
                             ICompilationUnit 	theCompilationUnit,
                             String             theTypeContent, 
                             String             theLineDelimiter) 
                         throws 
                             CoreException 
        {	
            StringBuilder builder = new StringBuilder();
		
            if (documented)
            {
                builder.append("@Documented ");
            }
            if (inherited)
            {
                builder.append("@Inherited ");
            }
            switch (retention)
            {
                case CLASS:
			
                    break;
				
                default:
				
                    builder.append("@Retention(RetentionPolicy.");
                    builder.append(retention);
                    builder.append(") ");
            }
		
            int nTargets = targets.size();
		
            if (nTargets != 0)
            {
                builder.append("@Target(");
			
                if (nTargets != 1)
                {
                    boolean first = true;
				
                    builder.append("{");
                    for (ElementType type : targets)
                    {
                        if (!first)
                        {
                            builder.append(", ");
                        }
                        else
                        {
                            first = false;
                        }
                        builder.append("ElementType.");
                        builder.append(type);
                    }
                    builder.append("}");
                }
                else
                {
                    for (ElementType type : targets)
                    {
                        builder.append("ElementType.");
                        builder.append(type);
                    }
                }
                builder.append(") ");
            }
            builder.append(theTypeContent);
            return super.constructCUContent(theCompilationUnit,  builder.toString(), theLineDelimiter);
        }

In CLOS this would have been an :around method. Sigh …

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved

An Extended NewAnnotation Wizard For Eclipse: Implementing The Wizard Page UI

The UI for a wizard page is created by its implementation of the

    public void createControl(Composite parent);

method defined in the

    org.eclipse.jface.dialogs.IWizardPage

interface.

The NewAnnotationWizardPage implementation of it is as follows

        @Override
        public void createControl(Composite theParent) 
        {
            initializeDialogUnits(theParent);
		
            Composite composite= new Composite(theParent, SWT.NONE);

            composite.setFont(theParent.getFont());
		
            int nColumns = 4;

            GridLayout layout= new GridLayout();
			
            layout.numColumns= nColumns;
            composite.setLayout(layout);

            createContainerControls(composite, nColumns);
            createPackageControls(composite, nColumns);
            createEnclosingTypeControls(composite, nColumns);

            createSeparator(composite, nColumns);

            createTypeNameControls(composite, nColumns);
            createModifierControls(composite, nColumns);
		
            createSeparator(composite, nColumns);
		
            createAnnotationsControls(composite);
		
            createSeparator(composite, nColumns);

            createCommentControls(composite, nColumns);
            enableCommentControl(true);

            setControl(composite);

            Dialog.applyDialogFont(composite);
        }

This is essentially a clone of the same method from the existing Eclipse supplied class with the addition of these three lines.

        ...

        createSeparator(composite, nColumns);
		
        createAnnotationsControls(composite);
		
        createSeparator(composite, nColumns);
		
        ...

The createAnnotationsControls(Composite) method is responsible for the creation of the new UI elements within the page. These are shown below.

The text at the top is a Label and is added directly to the parent composite passed in to the call to createControl(Composite), with a horizontal span set to ensure that it is in a row on its own.

        Label label = new Label(theParent, SWT.NONE);
		
        label.setText(Text.WHICH_ANNOTATIONS_TEXT);
		
        GridData labelData = new GridData();
		
        labelData.horizontalSpan = nColumns;
		
        label.setLayoutData(labelData);

There are controls to specify the addition of the four basic meta-annotations

• Documented
• Inherited
• Retention
• Target

These controls are grouped together into a Composite which is laid out in two columns using a GridLayout

        Composite  group  = new Composite(theParent, SWT.NONE);
        GridLayout layout = new GridLayout();
		
        layout.numColumns = 2;
        group.setLayout(layout);
		
        GridData groupData = new GridData(GridData.FILL_HORIZONTAL);
		
        groupData.horizontalAlignment = SWT.CENTER;
        groupData.horizontalSpan = nColumns - 1;
        group.setLayoutData(groupData);

Both Documented and Inherited are marker annotations: they are either present or not. In each case a checkbox is used to indicate whether they should be present or not.

        Button button = null;
		
        // Documented
	
        label = new Label(group, SWT.NONE);
        label.setText(Text.DOCUMENTED_LABEL);
		
        button = new Button(group, SWT.CHECK);
        button.addSelectionListener(
                   new SelectionAdapter()
                   {
                       public void widgetSelected(SelectionEvent theEvent)
                       {
                           documented = ((Button)(theEvent.widget)).getSelection();
                       }
                   });
		
        // Inherited
		
        label = new Label(group, SWT.NONE);
        label.setText(Text.INHERITED_LABEL);
		
        button = new Button(group, SWT.CHECK);
        button.addSelectionListener(
                   new SelectionAdapter()
                   {
                       public void widgetSelected(SelectionEvent theEvent)
                       {
                           inherited = ((Button)(theEvent.widget)).getSelection();
                       }
                   });

The Retention annotation has value of type RetentionPolicy which is an Enum with a choice of three values. These are represented by a group of radio buttons. The currently selected value is represented by the corresponding value of RetentionPolicy which is stored in the retention instance variable.

        // Retention
		
        label = new Label(group, SWT.NONE);
        label.setText(Text.RETENTION_LABEL);
		
        Composite  retentionGroup        = new Composite(group, SWT.NONE);
        GridLayout retentionGroupLayout  = new GridLayout();
		
        retentionGroupLayout.numColumns   = N_RETENTION_GROUP_COLUMNS;
        retentionGroup.setLayout(retentionGroupLayout);
		
        Listener listener = new Listener()
                            {
                                @Override
                                public void handleEvent(Event theEvent) 
                                {
                                    Button button = (Button)(theEvent.widget);
									
                                    if (button.getSelection())
                                    {
                                        retention = (RetentionPolicy)button.getData();
                                    }
                                }
                            };
		
        for (String name : sortedNames(RetentionPolicy.class))
        {
            label = new Label(retentionGroup, SWT.NONE);
            label.setText(name);
		
            button = new Button(retentionGroup, SWT.RADIO);
            button.addListener(SWT.Selection, listener);
            button.setData(Enum.valueOf(RetentionPolicy.class, name));
            if (RetentionPolicy.CLASS.name().equals(name))
            {
                button.setSelection(true);
            }
        }

The sortedNames method is a utility method which returns the names of an Enum’s values in alphabetical order. It is possible to obtain a set containing all the values of a given Enum but these are in the order they appear in the Enum definition, which is not necessarily alphabetical.

        private <E extends Enum<E>> Iterable<String> sortedNames(Class<E> theEnumClass)
        {
            List<String> names = new ArrayList<String>();
		
            for (Enum<E> e : EnumSet.allOf(theEnumClass))
            {
                names.add(e.name());
            }
            Collections.sort(names);
            return names;
        }

The Target annotation can have multiple values of the Enum type ElementType. These are represented by a group of checkboxes. The currently selected values are held in the instance variable targets which is of type EnumSet<ElementType>.

        // Targets
		
        label = new Label(group, SWT.NONE);
        label.setText(Text.TARGETS_LABEL);
		
        GridData targetsData = new GridData();
		
        targetsData.verticalAlignment = SWT.TOP;
        label.setLayoutData(targetsData);
		
        Composite targetsGroup       = new Composite(group, SWT.NONE);
        RowLayout targetsGroupLayout = new RowLayout(SWT.VERTICAL);
		
        targetsGroupLayout.marginLeft = 0;
        targetsGroupLayout.marginTop  = 0;
        targetsGroup.setLayout(targetsGroupLayout);
		
        listener = new Listener()
        {
            @Override
            public void handleEvent(Event theEvent) 
            {
                Button      button = (Button)(theEvent.widget);
                ElementType type   = (ElementType)button.getData();
				
                if (button.getSelection())
                {
                    targets.add(type);
                }
                else
                {
                    targets.remove(type);
                }
            }
        };
		
        for (String name : sortedNames(ElementType.class))
        {
            button = new Button(targetsGroup, SWT.CHECK);
            button.addListener(SWT.Selection, listener);
            button.setData(Enum.valueOf(ElementType.class, name));
            button.setText(name);
        }

---

Copyright (c) 2009 By Simon Lewis. All Rights Reserved