Just An Application

September 7, 2014

The Mystery Of The Unsigned JAR: Part Three — The Little JAR That Hid

This is the bytecode for a method called load with everything removed except the calls to methods in the Java APIs.

As you can see, it is possible to deduce what the method is doing solely on the basis of the Java API method calls.


    public void load();
      Code:
         0: aload_0
         1: dup
         2: invokevirtual #6                  // Method java/lang/Object.getClass:()Ljava/lang/Class;
         
            ...
            
        11: invokevirtual #7                  // Method java/lang/Class.getResourceAsStream:(Ljava/lang/String;)Ljava/io/InputStream;
        
            ...
        
        47: new           #12                 // class java/util/jar/JarInputStream
        50: dup
        51: new           #13                 // class java/io/ByteArrayInputStream
        54: dup
        55: aload_1
        56: invokespecial #14                 // Method java/io/ByteArrayInputStream."<init>":([B)V
        59: invokespecial #15                 // Method java/util/jar/JarInputStream."<init>":(Ljava/io/InputStream;)V
        62: astore_2
        
            ...
            
        75: aload_2
        76: invokevirtual #16                 // Method java/util/jar/JarInputStream.getNextJarEntry:()Ljava/util/jar/JarEntry;
        79: dup
        80: astore        4
        82: ifnull        235
        85: aload         4
        87: invokevirtual #17                 // Method java/util/jar/JarEntry.getName:()Ljava/lang/String;
        90: dup
        91: astore        5
        
            ...
            
        99: invokevirtual #18                 // Method java/lang/String.endsWith:(Ljava/lang/String;)Z
       102: ifeq          172
       
            ...
            
       117: iconst_0
       118: iconst_1
       119: dup
       120: pop2
       121: aload         5
       123: dup_x1
       124: invokevirtual #19                 // Method java/lang/String.length:()I
       127: bipush        6
       129: iconst_1
       130: dup
       131: pop2
       132: isub
       133: invokevirtual #20                 // Method java/lang/String.substring:(II)Ljava/lang/String;
       136: bipush        47
       138: iconst_1
       139: dup
       140: pop2
       141: bipush        46
       143: iconst_1
       144: dup
       145: pop2
       146: invokevirtual #21                 // Method java/lang/String.replace:(CC)Ljava/lang/String;
       149: astore        5
       
            ...
            
       163: aload_3
       164: invokevirtual #22                 // Method java/util/HashMap.put:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
       167: pop
       168: goto          75
       171: pop
       172: aload         4
       174: invokevirtual #23                 // Method java/util/jar/JarEntry.isDirectory:()Z
       177: ifeq          185
       180: aload_2
       181: goto          76
      
            ...

The calls to the JarInputStream and JarEntry methods make it clear that the method is iterating over the contents of a JAR

The 6 in

    127: bipush        6

just happens to be the length of the suffix

    .class

The 47 in

    136: bipush        47

is the solidus character (‘/‘) and the 46 in

    141: bipush        46

is the dot character (‘.‘).

The sequences following the iconst_0 instruction and each bipush instruction

    117: iconst_0
    118: iconst_1
    119: dup
    120: pop2
    
    ...

    127: bipush        6
    129: iconst_1
    130: dup
    131: pop2
    
    ...
    
    136: bipush        47
    138: iconst_1
    139: dup
    140: pop2
    
    ...

    141: bipush        46
    143: iconst_1
    144: dup
    145: pop2

are all either pretty feeble attempts at obfuscation or evidence of a severely confused compiler as they have no effect at all.

From time immemorial Java class loaders have been converting the JarEntry names of class files into class names and that is what this method is doing as well.

If you have a class, for example,

    xper.mcm.CrashPow

then the name of the JarEntry for the class file in a JAR will be

    xper/mcm/CrashPow.class

Chop off the .class suffix

    117: iconst_0

      0
      
    118: iconst_1
    
      0, 1
      
    119: dup
    
      0, 1, 1

    120: pop2
    
      0
      
    121: aload         5
    
      0, entry_name
      
    123: dup_x1
    
      entry_name, 0, entry_name
      
    124: invokevirtual #19                 // Method java/lang/String.length:()I
    
      entry_name, 0, length(entry_name)
      
    127: bipush        6
    
      entry_name, 0, length(entry_name), 6
      
    129: iconst_1
    
      entry_name, 0, length(entry_name), 6, 1
      
    130: dup
    
      entry_name, 0, length(entry_name), 6, 1, 1
    
    131: pop2
    
      entry_name, 0, length(entry_name), 6
    
    132: isub
    
      entry_name, 0, length(entry_name)_minus_6
      
    133: invokevirtual #20                 // Method java/lang/String.substring:(II)Ljava/lang/String;

      substring(entry_name, 0, length(entry_name)_minus_6))

and replace the occurrences of ‘/‘ with ‘.

    136: bipush        47
    
      substring(...), 47
      
    138: iconst_1
    
      substring(...), 47, 1
      
    139: dup
    
      substring(...), 47, 1, 1
      
    140: pop2
    
      substring(...), 47
      
    141: bipush        46
    
       substring(...), 47, 46
       
    143: iconst_1
    
      substring(...), 47, 46, 1
      
    144: dup
    
      substring(...), 47, 46, 1, 1

    145: pop2
    
      substring(...), 47, 46
       
    146: invokevirtual #21                 // Method java/lang/String.replace:(CC)Ljava/lang/String;
    
      substring(...).replace(47, 46)
        
    149: astore        5

and you get the canonical Java class name back again

The method is storing the contents of the class files in the JAR it is iterating over in a HashMap with the class names as the keys.

Now where’s the JAR ?


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 6, 2014

The Mystery Of The Unsigned JAR: Part Two — The Sekrit Method

The ‘sekrit method’ takes a single String argument and returns a String and it could not be more obvious if it tried.

Unlike the other methods in the class it has a non-meaningful name. Instead its name features a large subset of the lower case letters of the alphabet in order. I don’t think there is any coding standard that requires that, ‘tho I suppose there could be.

It is always passed a String literal and the String literal is always gibberish, yet the return values are often passed to factory methods in the Java API which take specific well-formed names.

The inference is obvious. The method takes obfuscated Strings and turns them into unobfuscated Strings.

The nice thing about Java VM bytecode is that you can decompile it using nothing more than a pencil and a piece of paper, oh, and a knowledge of the Java VM opcodes !

Being stack based it is pretty easy to work out what is going on by tracing the state of the stack and the local variables after each instruction.

Of course if you don’t have a piece of paper and a pencil handy you can always write some code to do the same thing.

The following shows the stack/local variable state after each instruction for a large part of the sekrit method.

The top of the stack is on the right.

    ...

    
    34: invokevirtual #232                // Method java/lang/StringBuffer.toString:()Ljava/lang/String;
    
        S
    
    37: dup
    
        S, S
    
    38: invokevirtual #19                 // Method java/lang/String.length:()I
    
        S, length(S)
    
    41: iconst_1
    
        S, length(S), 0x01
    
    42: isub
    
        S, length(S)_minus_one
    
    43: iconst_2
    
        S, length(S)_minus_one, 0x02
    
    44: iconst_3
    
        S, length(S)_minus_one, 0x02, 0x03
    
    45: ishl
    
        S, length(S)_minus_one, 0x10
    
    46: iconst_3
    
        S, length(S)_minus_one, 0x10, 0x03
    
    47: iconst_5
    
        S, length(S)_minus_one, 0x10, 0x03, 0x05
    
    48: ixor
    
        S, length(S)_minus_one, 0x10, 0x06
    
    49: ixor
    
        S, length(S)_minus_one, 0x16
    
    50: iconst_2
    
        S, length(S)_minus_one, 0x16, 0x02
    
    51: iconst_5
    
        S, length(S)_minus_one, 0x16, 0x02, 0x05
    
    52: ixor
    
        S, length(S)_minus_one, 0x16, 0x07
    
    53: iconst_3
    
        S, length(S)_minus_one, 0x16, 0x07, 0x03
    
    54: ishl
    
        S, length(S)_minus_one, 0x16, 0x38
    
    55: iconst_2
    
        S, length(S)_minus_one, 0x16, 0x38, 0x02
    
    56: ixor
    
        S, length(S)_minus_one, 0x16, 0x3A
    
    57: iconst_2
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x02
    
    58: iconst_5
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x02, 0x05
    
    59: ixor
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x07
    
    60: iconst_4
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x07, 0x04
    
    61: ishl
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70
    
    62: iconst_1
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70, 0x01
    
    63: dup
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70, 0x01, 0x01
    
    64: ishl
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70, 0x02
    
    65: ixor
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72
    
    66: aload_0
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, avar0
    
    67: invokevirtual #19                 // Method java/lang/String.length:()I
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0)
    
    70: dup
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), length(avar0)
    
    71: newarray       char
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)]
    
    73: iconst_1
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)], 0x01
    
    74: dup
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)], 0x01, 0x01
    
    75: pop2
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)]
    
    76: swap
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, char[length(avar0)], length(avar0)
    
    77: iconst_1
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, char[length(avar0)], length(avar0), 0x01
    
    78: isub
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, char[length(avar0)], length(avar0)_minus_1
    
    79: dup_x2
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1, 0x72, char[length(avar0)], length(avar0)_minus_1
    
    80: istore_1
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1, 0x72, char[length(avar0)]
    
        {
            ivar1 == length(avar0)_minus_1
        }
    
    81: astore_3
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1, 0x72
    
        {
            ivar1 == length(avar0)_minus_1
            avar3 == char[length(avar0)]
        }
    
    
    82: istore        7
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1
    
        {
            ivar1 == length(avar0)_minus_1
            avar3 == char[length(avar0)]
            ivar7 == 0x72
        }
    
    84: dup_x2
    
        S, length(S)_minus_one, length(avar0)_minus_1, 0x16, 0x3A, length(avar0)_minus_1
    
    85: pop
    
        S, length(S)_minus_one, length(avar0)_minus_1, 0x16, 0x3A
    
    86: istore        4
    
        S, length(S)_minus_one, length(avar0)_minus_1, 0x16
    
        {
            ivar1 == length(avar0)_minus_1
            avar3 == char[length(avar0)]
            ivar4 == 0X3A
            ivar7 == 0x72
        }
    
    88: pop
    
        S, length(S)_minus_one, length(avar0)_minus_1
    
    89: swap
    
        S, length(avar0)_minus_1, length(S)_minus_one
    
    90: dup
    
        S, length(avar0)_minus_1, length(S)_minus_one, length(S)_minus_one
    
    91: istore_2
    
        S, length(avar0)_minus_1, length(S)_minus_one
    
        {
            ivar1 == length(avar0)_minus_1
            ivar2 == length(S)_minus_one
            avar3 == char[length(avar0)]
            ivar4 == 0X3A
            ivar7 == 0x72
        }
    
    92: istore        5
    
        S, length(avar0)_minus_1
    
        {
            ivar1 == length(avar0)_minus_1
            ivar2 == length(S)_minus_one
            avar3 == char[length(avar0)]
            ivar4 == 0X3A
            ivar5 == length(S)_minus_one
            ivar7 == 0x72
    }
    
    94: swap
    
        length(avar0)_minus_1, S
    
    95: astore        6
    
        length(avar0)_minus_1
    
        {
            ivar1 == length(avar0)_minus_1
            ivar2 == length(S)_minus_one
            avar3 == char[length(avar0)]
            ivar4 == 0x3A
            ivar5 == length(S)_minus_one
            avar6 == S
            ivar7 == 0x72
        }
    
    97: goto          163
    
    
    100: aload_3
    
        char[length(avar0)]
    
    101: iload         4
    
        char[length(avar0)], 0x3A
    
    103: aload_0
    
        char[length(avar0)], 0x3A, avar0
    
    104: iload_1
    
        char[length(avar0)], 0x3A, avar0, ivar1
    
    105: iinc          1, -1
    
        char[length(avar0)], 0x3A, avar0, ivar1
    
    108: dup_x2
    
        char[length(avar0)], ivar1, 0x3A, avar0, ivar1
    
    109: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        char[length(avar0)], ivar1, 0x3A, c
    
    112: aload         6
    
        char[length(avar0)], ivar1, 0x3A, c, avar6
    
    114: iload_2
    
        char[length(avar0)], ivar1, 0x3A, c, avar6, ivar2
    
    115: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        char[length(avar0)], ivar1, 0x3A, c, d
    
    118: ixor
    
        char[length(avar0)], ivar1, 0x3A, c_xor_d
    
    119: ixor
    
        char[length(avar0)], ivar1, c_xor_d_xor_0x3A
    
    120: i2c
    
        char[length(avar0)], ivar1, c_xor_d_xor_0x3A
    
    121: castore
    
    
    122: iload_1
    
        ivar1
    
    123: ifge          130
    126: aload_3
    127: goto          167
    
    
    130: aload_3
    
        avar3
    
    131: iload         7
    
        avar3, 0x72
    
    133: aload_0
    
        avar3, 0x72, avar0
    
    134: iload_1
    
        avar3, 0x72, avar0, ivar1
    
    135: dup_x2
    
        avar3, ivar1, 0x72, avar0, ivar1
    
    136: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        avar3, ivar1, 0x72, c
    
    139: aload         6
    
        avar3, ivar1, 0x72, c, avar6
    
    141: iload_2
    
        avar3, ivar1, 0x72, c, avar6, ivar2
    
    142: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        avar3, ivar1, 0x72, c, d
        
    145: ixor
    
        avar3, ivar1, 0x72, c_xor_d
    
    146: ixor
    
        avar3, ivar1, c_xor_d_xor_0x72
    
    147: i2c
    
        avar3, ivar1, c_xor_d_xor_0x72
    
    148: iinc          1, -1
    151: iinc          2, -1
    154: castore
    
    
    155: iload_2
    
        ivar2
    
    156: ifge          162
    
    159: iload         5
    
        ivar5
        
    161: istore_2
    
    
    162: iload_1
    
        ivar1
        
    163: ifge          100
    
    
    166: aload_3
    
        avar3
        
    167: invokespecial #239                // Method java/lang/String."<init>":([C)V
        
        
    170: areturn

The method is basically is a loop which starts at

    100: aload_3

and ends at

    163: ifge          100

Everything before the loop is initialization code, some of which seems to have been partially obfuscated by the use of various bitwise operators acting on small integers.

Following the initialization we can deduce the following.

The local variable avar0 holds the obfuscated String passed as the argument.

The local variable avar3 holds an array of chars which will contain the unobfuscated characters.

The local variable ivar1 holds the index used to access the characters in avar0 and avar3. Note that it being decremented

    105: iinc          1, -1

and, compared against 0

    123: ifge          130

so both avar0 and avar3 are being iterated over backwards.

The local variable ivar2 is used in the same way to iterate backwards over the String in avar6.

Each time round the loop a character is added to avar3. The character is the result of XORing a character from avar0 and a character from avar6 and the constant 0x3A. [Instructions 100 to 121]

If there are still characters to be added a second character is added which is the result of XORing a character from avar0 and a character from avar6 and the constant 0x72. [Instructions 130 to 154]

Note that ivar1 is being decremented twice each time round the loop but ivar2 only once.

The String in avar6 is constructed using the name of the calling method so the obfuscation is context sensitive.

If you convert all of the above back into Java you will probably end up with something like this

        ...
    
        int     lengthA = a.length();
        int     lengthS = s.length();
        char[]  chars   = new char[lengthA];
        int     i       = lengthA - 1;
        int     j       = lengthS - 1;
    
        while (true)
        {
            char c = a.charAt(i);
            char d = s.charAt(j);
    
            chars[i] = (char)(c ^ d ^ 0x3a);
    
            --i;
            if (i < 0)
            {
                break;
            }
    
            c = a.charAt(i);
            d = s.charAt(j);
    
            chars[i] = (char)(c ^ d ^ 0x72);
    
            --i;
            --j;
            if (j < 0)
            {
                j = lengthS -1;
            }
            if (i < 0)
            {
                break;
            }
        }
        return new String(chars);

where a is the obfuscated String being passed as the argument and s is the context sensitive ‘mask’.

I doubt whether it compiles into exactly the same bytecode but it definitely deobfuscates the String literals in the class files correctly.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

February 8, 2011

“Alien Dalvik” ?

Filed under: Alien Dalvik, Android, Dalvik, Java, Java VM, Mobile Java — Tags: , , , , — Simon Lewis @ 5:51 pm

It is going to be interesting to find out exactly how this works.

Despite the name it cannot simply be a standalone Dalvik port. After all, all Dalvik can do is run Dexes. In fact the following would seem to suggest that it might not involve a Dalvik VM at all

Alien Dalvik enables the majority of Android applications to run unmodified, allowing application store owners to quickly kick start Android application store services by simply repackaging Android Package (APK) files.

Its not clear why you would want to mess with the APKs unless you wanted to do something else at the same time, especially if they are signed, but who knows ?

Presumably unmodified in this context refers to the source code ?

To run the majority of Android applications unmodified it must implement all the standard Java APIs and the Android APIs used by those applications some how. If the source code does not need to be modified then all those classes and methods that are referenced need to be in the runtime.

Implementing a large part of what is approximately Java 5 plus a large part of some version of the Android APIs (which version is of course another problem) is not exactly trivial given that many of the Android API methods are actually native methods, or call native methods almost immediately, and they often use Android platform specific features, for example, Skia and Surfaceflinger to name but two. In fact it is quite difficult to see how it can run the majority of Android applications unmodified unless it actually contains what amounts to a largish chunk of, not to put to fine a point on it, Android.

Which leads on to the question of where this chunk might live ? Presumably it is not per downloaded application, so the chunk must (?) be pre-installed on the device, and it is this that the device manufacturer will be licensing (?).

There is also the question of memory footprint. Android relies quite heavily on code sharing via the Zygote mechanism to keep its memory footprint down. It is possible to achieve the same effect using the standard fork()/exec() mechanism if your VM has been written with that in mind, although this is one of the techniques that Oracle argues they have a patent on. This is not necessarily a problem if the underlying VM technology is licensed from Oracle, which it might be in this case.

Then there is the question of native code. Presumably Android applications that use native code need not apply ?

And then there is security. Android applications are sand-boxed on the basis of UIDs and permissions are enforced on the basis of GIDs. If the host OS does not have these or their equivalents then there is a problem. Equally if host OS native applications are not themselves sand-boxed, even if Android applications cannot get out of their sand-boxes, everything else on the platform may be able to get into them

And if there are host OS native applications as well then how do they co-exist without some form of integration at the window manager (or equivalent) level ?

As I say it will be interesting to find out how it actually works


Copyright (c) 2011 By Simon Lewis. All Rights Reserved.

November 24, 2009

InvokeDynamic And MethodHandles On Snow Leopard: Part Four

The Even More Magical java.dyn.InvokeDynamic

java.dyn.InvokeDynamic is a final class with a private constructor and no other no declared methods, but you can do this with it


    package test;

    import java.dyn.InvokeDynamic;

    public final class Bar
    {
        void foo()
        {
            System.out.println(InvokeDynamic.bar("bar invoked dynamically"));
        }
    }

and the compiler will compile it to this


    Macintosh:tmp simon$ build/bsd-i586/bin/javap -c test.Bar
    Compiled from "Bar.java"
    public final class test.Bar extends java.lang.Object {
      public test.Bar();
        Code:
          0: aload_0       
          1: invokespecial #1                  // Method java/lang/Object."":()V
          4: return        

      void foo();
        Code:
          0: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
          3: ldc           #3                  // String bar invoked dynamically
          5: invokedynamic #4,  0              // NameAndType bar:(Ljava/lang/String;)Ljava/lang/String;
         10: invokevirtual #5                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
         13: return        
    }

java.dyn.InvokeDynamic is a Java language level marker for the JVM instruction formerly known as

    xxxunusedxxx

and now known as

    invokedynamic

The invokedynamic Instruction

The instruction is five bytes long. The one byte opcode, 186 (0xBA), is followed by two bytes which give the index into the constant pool of a name and type, in this case,

    bar

and

    (Ljava/lang/String;)Ljava/lang/String

followed by two zero bytes.

And that is it. Unlike the original invoke instructions

  • invokevirtual
  • invokespecial
  • invokestatic
  • invokeinterface

there is not enough information in the instruction to enable the VM to determine what it should be invoking or how.

Hence, if you run this,


    package test;

    public final class InvokeDynamicTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                Bar bar = new Bar();
			
                bar.foo();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

this happens


    Macintosh:tmp simon$ build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.InvokeDynamicTest
    java.dyn.InvokeDynamicBootstrapError: class has no bootstrap method: class test.Bar
	    at sun.dyn.CallSiteImpl.makeSite(CallSiteImpl.java:63)
	    at test.Bar.foo(Bar.java:10)
	    at test.InvokeDynamicTest.main(InvokeDynamicTest.java:13)

The Bootstrap Method

The first time the VM encounters an invokedynamic instruction it looks for the bootstrap method for the class of the caller.

The bootstrap method is a static method of the form

    static CallSite bootstrapMethod(Class theCallerClass, String theMethodName, MethodType theMethodType)

The bootstrap method for a class is registered either by name or as a MethodHandle using a static method on the class

    java.dyn.Linkage

For example

    ...
	
    static void init()
    {
        Linkage.registerBootstrapMethod(
                    Bar.class, 
                    MethodHandles.
                        lookup().
                            findStatic(
                                Bootstrap.class,
                                "bootstrap",
                                MethodType.make(
                                               CallSite.class, 
                                               Class.class, 
                                               String.class, 
                                               MethodType.class)));
    }
	
    ...

registers the static method

     private static CallSite bootstrap(Class theClass, String theName, MethodType theType)

of the class Bootstrap as the bootstrap method for the class Bar.

The CallSite

The bootstrap method is passed the class of the caller, and the name and type of the method specified by the instance of the invokedynamic instruction, and is responsible for returning an instance of the class

    java.dyn.CallSite

The returned CallSite instance contains a MethodHandle. The JVM uses the method referenced by the MethodHandle as the method to invoke.

Using this as the bootstrap method.


    private static CallSite bootstrap(Class<?> theClass, String theName, MethodType theType)
    {
        MethodHandle mh = MethodHandles.lookup().findStatic(Methods.class, "generic", theType);
        CallSite     cs = new CallSite(theClass, theName, mh.type());
		
        cs.setTarget(mh);
        return cs;
    }

this as the definition of the class Methods


    package test;

    final class Methods
    {
        static String generic(String theArgument)
        {
            return theArgument + ": brought to you by Methods.generic()";
        }
    }

and invoking Bar.foo() like this


    package test;


    public final class InvokeDynamicTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                Bootstrap.init();
			
                Bar bar = new Bar();
			
                bar.foo();
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

gets you this.


    Macintosh:tmp simon$ build/bsd-i586/bin/java -classpath . -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic test.InvokeDynamicTest
    bar invoked dynamically: brought to you by Methods.generic()

While this is not of itself a particularly exciting example there are few things worth noting

  • most importantly, what happens when the method Bar.foo() is invoked is determined entirely at runtime, i.e., dynamically

  • the whole thing was done in Java. Systems which generate bytecode themselves can obviously generate invokedynamic instructions directly as well as generating bytecode on the basis of the behaviour which results

  • in the example the bootstrap method produces a single CallSite. In practice bootstrap methods would produce different instances of CallSite depending upon their arguments, and would also use different MethodHandles depending on their arguments.

  • the CallSite class can be sub-classed, which enables further specialization of certain aspects of its behavior

  • in the example the method actually invoked does not do anything particularly interesting. In practice the behaviour of invoked methods can be dependent on their argument(s), and in the case of systems generating bytecode dynamically, invoked methods can feedback information to the code generation stage.


Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

November 22, 2009

InvokeDynamic And MethodHandles On Snow Leopard

The most interesting feature in JDK 7 for me, at least until the recent reappearance of closures, and probably even then, is the JSR 292 invokedynamic support. It is basically the reason why I started my attempt to build OpenJDK 7 on Snow Leopard.

Ostensibly to use the upcoming JSR 292 features it is necessary to apply a number of patches to a standard OpenJDK 7 release. See here for details. However the current revision of the BSD port seems to contain a lot of the necessary code already so I decided to see how far I could get using that rather than a patched version.

JSR 292 is very much a work-in-progress. There was an early-draft release of the specification almost eighteen months ago, but nothing since. You can see what is going on by looking at the code as it is released, but it is still changing so some or all of the following may change.

Basically JSR 292 defines a new JVM opcode

    invokedynamic

which is intended to provide VM level support for method invocation in languages other than Java which have different invocation semantics. That is a drastic over-simplification but it will do for the moment.

There are also some features visible in Java itself, one of which is method handles, which can be thought of as pointers to individual methods. To obtain a handle to a given method you need an instance of a MethodType, which defines the return type of the method and the parameters it takes. The classes

    MethodHandle

and

    MethodType

are defined in the package

    java.dyn

To determine how they currently work and what else you can do you need to look at the source code and the accompanying comments.

Making A MethodType

The following code is based on my understanding of the currently available revision of the JSR 292 code in the BSD port. Hopefully makes an instance of MethodType specifying a method which returns void and takes a single argument of type String.


    import java.dyn.MethodType;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }

Running this code results in the following. Note that although the JSR292 code is present in the current revision of the BSD port it is still deemed to be experimental and has to be unlocked using the command line incantation shown.


    Macintosh:tmp simon$ build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    OpenJDK 64-Bit Server VM warning: JSR 292 method handle code is mismatched to this JVM.  Disabling support.
    type == (java.lang.String)void

An instance of MethodType appears to have been made, but the warning message is not particularly auspicious.

The message originates in the function

    JVM_RegisterMethodHandleMethods(JNIEnv, jclass)

which is in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp

It looks like this (lightly re-formatted by me)


    JVM_ENTRY(void, JVM_RegisterMethodHandleMethods(JNIEnv *env, jclass MHN_class)) {
        assert(MethodHandles::spot_check_entry_names(), "entry enum is OK");

        // note: this explicit warning-producing stuff will be replaced by auto-detection of the JSR 292 classes

        if (!EnableMethodHandles) {
            warning("JSR 292 method handles are disabled in this JVM.  Use -XX:+UnlockExperimentalVMOptions -XX:+EnableMethodHandles to enable.");
            return;  // bind nothing
        }

        bool enable_MH = true;

        {
            ThreadToNativeFromVM ttnfv(thread);

            int status = env->RegisterNatives(MHN_class, methods, sizeof(methods)/sizeof(JNINativeMethod));

            if (env->ExceptionOccurred()) {
                MethodHandles::set_enabled(false);
                warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                enable_MH = false;
                env->ExceptionClear();
            }
        }

        if (enable_MH) {
	
            KlassHandle MHI_klass = SystemDictionaryHandles::MethodHandleImpl_klass();
		
            if (MHI_klass.not_null()) {
      
                symbolHandle raiseException_name   = oopFactory::new_symbol_handle("raiseException", CHECK);
                symbolHandle raiseException_sig    = oopFactory::new_symbol_handle("(ILjava/lang/Object;Ljava/lang/Object;)V", CHECK);
                methodOop    raiseException_method = instanceKlass::cast(MHI_klass->as_klassOop())
                                                         ->find_method(raiseException_name(), raiseException_sig());
													 
                if (raiseException_method != NULL && raiseException_method->is_static()) {
                    MethodHandles::set_raise_exception_method(raiseException_method);
                } else {
                    warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                    enable_MH = false;
                }
            }
        }

        if (enable_MH) {
            MethodHandles::set_enabled(true);
        }

        if (!EnableInvokeDynamic) {
            warning("JSR 292 invokedynamic is disabled in this JVM.  Use -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic to enable.");
            return;  // bind nothing
        }

        {
            ThreadToNativeFromVM ttnfv(thread);

            int status = env->RegisterNatives(MHN_class, methods2, sizeof(methods2)/sizeof(JNINativeMethod));

            if (env->ExceptionOccurred()) {
                MethodHandles::set_enabled(false);
                env->ExceptionDescribe();
                warning("JSR 292 method handle code is mismatched to this JVM.  Disabling support.");
                env->ExceptionClear();
            } else {
                MethodHandles::set_enabled(true);
            }
        }
    }
    JVM_END

As can be seen it generates the warning message in three cases. Cases one and three are if the initialization of native methods fail. Case two is if the class

	sun.dyn.MethodHandleImpl

does not have a method of the form

    ???? static void raiseException(int arg1, Object arg2, Object arg3)

This is easy enough to check. The class is defined in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleImpl.java

In fact it does not.

Although it is not possible to know exactly what the method should do, the only use made of it is a bit opaque,
it is certainly possible to add something like this just in case it actually does get invoked.


    public static void raiseException(int arg1, Object arg2, Object arg3)
    {
        System.out.println("MethodHandleImpl.raiseException(" + arg1 + ", " + arg2 + ", " + arg3 + ")");
        throw new RuntimeException("MethodHandleImpl.raiseException(...)");
    }

Trying again following the addition


    Macintosh:tmp simon$ !!
    type == (java.lang.String)void

Getting A MethodHandle

So far so good, so let’s try getting a MethodHandle for a method of that type. There is a suitable one in java.io.PrintStream.


    import java.dyn.MethodHandle;
    import java.dyn.MethodHandles;
    import java.dyn.MethodType;

    import java.io.PrintStream;

    public final class DynTest
    {
        public static void main(String[] theArgs)
        {
            try 
            {
                MethodType type = MethodType.make(Void.TYPE, String.class);
			
                System.out.print("type == ");
                System.out.println(type);
				
                MethodHandle handle = MethodHandles.lookup().findVirtual(PrintStream.class, "print", type);
			
                System.out.print("handle == ");
                System.out.println(handle);
            }
            catch (Throwable t) 
            {
                t.printStackTrace();
            }
        }
    }
}

Now what happens ?


    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    java.lang.InternalError
        at sun.dyn.MethodHandleNatives.init(Native Method)
        at sun.dyn.DirectMethodHandle.(DirectMethodHandle.java:48)
        at sun.dyn.MethodHandleImpl.findMethod(MethodHandleImpl.java:163)
        at java.dyn.MethodHandles$Lookup.findVirtual(MethodHandles.java:233)
        at DynTest.main(DynTest.java:19)

Not so good.

The class

    sun.dyn.MethodHandleNatives

can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/MethodHandleNatives.java

The class has four init native methods.


    /** Initialize the method handle to adapt the call. */
    static native void init(AdapterMethodHandle self, MethodHandle target, int argnum);
    /** Initialize the method handle to call the correct method, directly. */
    static native void init(BoundMethodHandle self, Object target, int argnum);
    /** Initialize the method handle to call as if by an invoke* instruction. */
    static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);

    /** Initialize a method type, once per form. */
    static native void init(MethodType self);

The method is being called from the constructor of the class

    sun.dyn.DirectMethodHandle

which can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/DirectMethodHandle.java

The method is being passed the instance of sun.dyn.DirectMethodHandle being constructed, so it is evidently this variant that is being invoked.


     /** Initialize the method handle to call as if by an invoke* instruction. */
     static native void init(DirectMethodHandle self, Object ref, boolean doDispatch, Class caller);

The CPU neutral native method handle code can be found in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.cpp

The native method registration information is defined in the following data structure


static JNINativeMethod methods[] = {
  // void init(MemberName self, AccessibleObject ref)
  {CC"init",                    CC"("AMH""MH"I)V",              FN_PTR(MHI_init_AMH)},
  {CC"init",                    CC"("BMH""OBJ"I)V",             FN_PTR(MHI_init_BMH)},
  {CC"init",                    CC"("DMH""OBJ"Z"CLS")V",        FN_PTR(MHI_init_DMH)},
  {CC"init",                    CC"("MT")V",                    FN_PTR(MHI_init_MT)},
  {CC"init",                    CC"("MEM""OBJ")V",              FN_PTR(MHI_init_Mem)},
  {CC"expand",                  CC"("MEM")V",                   FN_PTR(MHI_expand_Mem)},
  {CC"resolve",                 CC"("MEM""CLS")V",              FN_PTR(MHI_resolve_Mem)},
  {CC"getTarget",               CC"("MH"I)"OBJ,                 FN_PTR(MHI_getTarget)},
  {CC"getConstant",             CC"(I)I",                       FN_PTR(MHI_getConstant)},
  //  static native int getNamedCon(int which, Object[] name)
  {CC"getNamedCon",             CC"(I["OBJ")I",                 FN_PTR(MHI_getNamedCon)},
  //  static native int getMembers(Class defc, String matchName, String matchSig,
  //          int matchFlags, Class caller, int skip, MemberName[] results);
  {CC"getMembers",              CC"("CLS""STRG""STRG"I"CLS"I["MEM")I",  FN_PTR(MHI_getMembers)}
};

from which we can deduce, after looking at the macro definitions for CC, DMH, etc., that the function we are interested in is

    MHI_init_DMH

This function throws an InternalError exception without an error message in two different places, and calls a function which also throws an InternalError exception without an error message in two different places, so it is not possible to statically determine what has caused the InternalError.

Some experimentation results in the discovery that the InternalError is being thrown from the function invoked by MHI_init_DMH, namely

    MethodHandles::init_DirectMethodHandle

at this point

    if (me == NULL) { THROW(vmSymbols::java_lang_InternalError()); }

The variable me can be set in four different places in this function


    ...

    me = MethodHandles::entry(MethodHandles::_invokeinterface_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokespecial_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokestatic_mh);

    ...

    me = MethodHandles::entry(MethodHandles::_invokevirtual_mh);

    ...

Whichever call to MethodHandles::entry(EntryKind) is being made at runtime is clearly returning NULL.

The method MethodHandles::entry(EntryKind) is defined inline in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/prims/methodHandles.hpp

It is doing a straightforward array access indexed by the EntryKind argument


    static MethodHandleEntry* entry(EntryKind ek) { assert(ek_valid(ek), "initialized");
                                                    return _entries[ek]; }

so it is the element in the _entries array corresponding to the EntryKind argument which is NULL.

The _entries array is static and is not directly accessible, it is declared private, and there is only one method which can be used to set its elements


    static void init_entry(EntryKind ek, MethodHandleEntry* me) {
        assert(ek_valid(ek), "oob");
        assert(_entries[ek] == NULL, "no double initialization");
    
        _entries[ek] = me;
    }

This method is only called from one place, the method

    MethodHandles::generate_method_handle_stub

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/methodHandles_x86.cpp

This method in turn is only called from one place, the method

    StubGenerator::generate_all()

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_32.cpp

The code in question looks like this


    ...
	
    if (EnableMethodHandles && SystemDictionary::MethodHandle_klass() != NULL) {
        for (MethodHandles::EntryKind ek = MethodHandles::_EK_FIRST;
             ek < MethodHandles::_EK_LIMIT;
             ek = MethodHandles::EntryKind(1 + (int)ek)) {
            StubCodeMark mark(this, "MethodHandle", MethodHandles::entry_name(ek));
            MethodHandles::generate_method_handle_stub(_masm, ek);
            }
    }

    ...

It is not called from the corresponding version of the method in the file

    $(JDK7_ROOT)/hotspot/src/cpu/x86/vm/stubGenerator_x86_64.cpp

which is the version used in the 64-bit version of the JVM that is running the code. Hence the problem. The array has not been initialized and all the entries are NULL.

The code in the 32-bit version does not appear to be 32-bit specific, and the method MethodHandles::generate_method_handle_stub() is defined in a generic x86 file rather than an x86_32 one so what happens if we graft the code segment above into the 64-bit version of StubGenerator::generate_all() ?

Crash, Bang, Wallop …


    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    testtype == (java.lang.String)void
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  SIGSEGV (0xb) at pc=0x000000010281b7d2, pid=13801, tid=4298117120
    #
    # JRE version: 7.0
    # Java VM: OpenJDK 64-Bit Server VM (17.0-b05 mixed mode bsd-amd64 compressed oops)
    # Problematic frame:
    # j  sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
    #
    ...

Possibly not such a good idea after all then.

Having said that, at first glance it is not obvious that this problem is a direct result of problems with the stub generation, especially since it seems to be crashing inside a Java method.

The method in question can be found in the file

    $(JDK7_ROOT)/jdk/src/share/classes/sun/dyn/AdapterMethodHandle.java

and it looks like this.


    public static boolean canPairwiseConvert(MethodType newType, MethodType oldType) {
        // same number of args, of course
        int len = newType.parameterCount();
        if (len != oldType.parameterCount())
            return false;

        // Check return type.  (Not much can be done with it.)
        Class exp = newType.returnType();
        Class ret = oldType.returnType();
        if (!VerifyType.isNullConversion(ret, exp))
            return false;

        // Check args pairwise.
        for (int i = 0; i < len; i++) {
            Class src = newType.parameterType(i); // source type
            Class dst = oldType.parameterType(i); // destination type
            if (!canConvertArgument(src, dst))
                return false;
        }

        return true;
    }

It doesn’t look as though it is doing anything particularly foolhardy.

The crash site is identified in terms of the bytecode rather than the source code, so we need actually need that


    public static boolean canPairwiseConvert(java.dyn.MethodType, java.dyn.MethodType);
    Code:
        0:	aload_0
        1:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I
        4:	istore_2
        5:	iload_2
        6:	aload_1
        7:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I
        10:	if_icmpeq	15
        13:	iconst_0
        14:	ireturn
   
        ...

The pc is seven when the crash occurs which is this instruction

    7:	invokevirtual	#9; //Method java/dyn/MethodType.parameterCount:()I

which corresponds to the call

    oldType.parameterCount()

so it looks as though there is something drastically wrong with the value of oldType at this point.

Using the stack trace from the error log


    j  sun.dyn.AdapterMethodHandle.canPairwiseConvert(Ljava/dyn/MethodType;Ljava/dyn/MethodType;)Z+7
    j  sun.dyn.AdapterMethodHandle.makePairwiseConvert(Lsun/dyn/Access;Ljava/dyn/MethodType;Ljava/dyn/MethodHandle;)Ljava/dyn/MethodHandle;+18
    j  sun.dyn.MethodHandleImpl.findMethod(Lsun/dyn/Access;Lsun/dyn/MemberName;ZLjava/lang/Class;)Ljava/dyn/MethodHandle;+107
    j  java.dyn.MethodHandles$Lookup.findVirtual(Ljava/lang/Class;Ljava/lang/String;Ljava/dyn/MethodType;)Ljava/dyn/MethodHandle;+43
    j  DynTest.main([Ljava/lang/String;)V+34

we can determine that the MethodType value passed as the oldType argument originates from a call to the type() method on an instance of MethodHandle which has been passed to the calling method makePairwiseConvert()

The MethodHandle instance being passed to makePairwiseConvert is a newly created instance of DirectMethodHandle. The instance of MethodType it contains and which is returned by the call to the type() method, is in theory, the instance of MethodType created by the test program. This has already been printed successfully which implies that at that point it was sufficiently valid, or not sufficiently invalid to cause problems. This assumes of cource that it is the object actually being returned from the call to the type() method.

Given that it is extremely difficult to do anything from inside Java to a Java object which can subsequently cause the VM to crash, the problem is most likely to originate somewhere in native code.

As we already know the DirectMethodHandle constructor is invoking the static native method init() in the class MethodHandlesNative. We also know that the native function being called is actually MHI_init_DMH

This function is predominantly performing checks. It does not actually do anything very much to the MethodHandle instance it is passed. The function that it finishes by invoking

    MethodHandles::init_DirectMethodHandle

however, does.

The calls of interest are


    ...
	
    java_dyn_MethodHandle::init_vmslots(mh());

    ...

    sun_dyn_DirectMethodHandle::set_vmtarget(mh(), vmtarget);
    sun_dyn_DirectMethodHandle::set_vmindex(mh(),  vmindex);
	
    ...

    java_dyn_MethodHandle::set_vmentry(mh(), me);
   
    ...

Some experimentation reveals that the value in the instance variable type is changed during the call to

    java_dyn_MethodHandle::set_vmentry()

In fact it is changed to the value of me. It looks very much as though the type instance variable is being over-written with the MethodEntry pointer which is not a pointer to a Java object at all.

The Magical vmentry Field ?

The vm_entry instance variable is defined in the class sun.dyn.MethodHandleImpl as follows.

    private byte       vmentry;    // adapter stub or method entry point

Given that a CPU-level pointer is being stored in vmentry its declaration as a byte is rather curious.

Looking at the definition of set_vmentry() we can see that it is simply placing the pointer to the MethodEntry at _vmentry_offset within the instance which does not itself explain how the instance variable is capable of storing a 32-bit pointer let alone 64-bit one.

Maybe there is something magical about this particular instance variable ?

_vmentry_offset is itself set in the method java_dyn_MethodHandle::compute_offsets() along with the type and vmtarget instance variable offsets


    void java_dyn_MethodHandle::compute_offsets() {

        klassOop k = SystemDictionary::MethodHandle_klass();
  
        if (k != NULL && EnableMethodHandles) {
            compute_offset(_type_offset,      k, vmSymbols::type_name(),      vmSymbols::java_dyn_MethodType_signature(), true);
            compute_offset(_vmtarget_offset,  k, vmSymbols::vmtarget_name(),  vmSymbols::object_signature(), true);
            compute_offset(_vmentry_offset,   k, vmSymbols::vmentry_name(),   vmSymbols::machine_word_signature(), true);

            // Note:  MH.vmslots (if it is present) is a hoisted copy of MH.type.form.vmslots.
            // It is optional pending experiments to keep or toss.
            compute_optional_offset(_vmslots_offset, k, vmSymbols::vmslots_name(), vmSymbols::int_signature(), true);
        }
   }

Looking at the values of these offsets at runtime shows them to be as follows.


    _type_offset     == 24
    _vmtarget_offset == 12
    _vmentry_offset  == 24
    _vmslots_offset  == 0

The type and vmentry offsets are the same so set_vmentry() is indeed clobbering the value in the type instance variable. Something
has gone wrong with the computation of the instance variable offsets, or the instance variable vmentry is insufficently magical in the 64-bit case.

The compute_offset() function simply looks up the field in the class and returns the offset as held by the field descriptor.

The offsets of the fields in an instance of a Java class are computed in the method

    ClassFileParser::parseClassFile()

which is defined in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/classfile/classFileParser.cpp

This a long and complicated method, its almost a thousand lines long, but approximately half-way down is this


    ...
	
    // adjust the vmentry field declaration in java.dyn.MethodHandle
    if (EnableMethodHandles && class_name() == vmSymbols::sun_dyn_MethodHandleImpl() && class_loader.is_null()) {
      java_dyn_MethodHandle_fix_pre(cp, &fields, &fac, CHECK_(nullHandle));
    }
	
    ...

The Magical vmentry Field Explained

The ClassFileParser::java_dyn_MethodHandle_fix_pre() method is what makes the instance variable vmentry magic. It redefines
its type as either a Java int or long depending upon whether it is a 32-bit or a 64-bit VM.

As an aside despite the name the method is actually operating on the class sun.dyn.MethodHandleImpl not on the class java.dyn.MethodHandle.

It begins by looking for a particular field in the constant pool of the sun.dyn.MethodHandleImpl class.


  ...
  
  int       word_sig_index = 0;
  const int cp_size        = cp->length();
  
  for (int index = 1; index < cp_size; index++) {
      if (cp->tag_at(index).is_utf8() &&
          cp->symbol_at(index) == vmSymbols::machine_word_signature()) {
         word_sig_index = index;
         break;
      }
  }

  ...


The method vmSymbols::machine_word_signature() is defined in the file

    $(JDK7_ROOT)/hotspot/src/share/vm/classfile/vmSymbols.hpp

and its definition is dependent upon whether it is a 32 or a 64 bit field. In the 32 bit case it returns the signature

    I

and in the 64 bit case

    J

If a field with the right signature cannot be found a VirtualMachineError is thrown.

If an appropriate field is found the method then looks for a non-static field of type byte with the name vmentry. If it finds the field, it then re-defines
its type.



    ...

    const int n = (*fields_ptr)()->length();
  
    for (int i = 0; i < n; i += instanceKlass::next_offset) {
        int       name_index = (*fields_ptr)->ushort_at(i + instanceKlass::name_index_offset);
        int       sig_index  = (*fields_ptr)->ushort_at(i + instanceKlass::signature_index_offset);
        int       acc_flags  = (*fields_ptr)->ushort_at(i + instanceKlass::access_flags_offset);
        symbolOop f_name     = cp->symbol_at(name_index);
        symbolOop f_sig      = cp->symbol_at(sig_index);
		
        if (f_sig == vmSymbols::byte_signature() &&
            f_name == vmSymbols::vmentry_name() &&
            (acc_flags & JVM_ACC_STATIC) == 0) {
            // Adjust the field type from byte to an unmanaged pointer.
            assert(fac_ptr->nonstatic_byte_count > 0, "");
			
            fac_ptr->nonstatic_byte_count -= 1;
            (*fields_ptr)->ushort_at_put(
                               i + instanceKlass::signature_index_offset,
                               word_sig_index);
            if (wordSize == jintSize) {
                fac_ptr->nonstatic_word_count += 1;
            } else {
                fac_ptr->nonstatic_double_count += 1;
            }

            FieldAllocationType atype = (FieldAllocationType) (*fields_ptr)->ushort_at(i+4);
			
            assert(atype == NONSTATIC_BYTE, "");
			
            FieldAllocationType new_atype = NONSTATIC_WORD;
      
            if (wordSize > jintSize) {
                if (Universe::field_type_should_be_aligned(T_LONG)) {
                    atype = NONSTATIC_ALIGNED_DOUBLE;
                } else {
                    atype = NONSTATIC_DOUBLE;
                }
            }
            (*fields_ptr)->ushort_at_put(i+4, new_atype);

            found_vmentry = true;
            break;
        }
    }
	
	...

except that in this version of the code there is a bug in the 64-bit case. I didn’t spot it straight away but I did get it eventually, albeit after a lengthy excursion into the mechanics of laying out instance variables in Hotspot.

Found it yet ?

It’s here.


            ...
			
            FieldAllocationType new_atype = NONSTATIC_WORD;
      
            if (wordSize > jintSize) {
                if (Universe::field_type_should_be_aligned(T_LONG)) {
                    atype = NONSTATIC_ALIGNED_DOUBLE;
                } else {
                    atype = NONSTATIC_DOUBLE;
                }
            }
            (*fields_ptr)->ushort_at_put(i+4, new_atype);
			
            ...

In the 64-bit case the field type is not correctly updated, which causes the subsequent field offset problem.

With this fixed the problem goes away.


    Macintosh:tmp simon$ !!
    build/bsd-amd64/j2sdk-image/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableInvokeDynamic DynTest
    type == (java.lang.String)void
    handle == print(java.io.PrintStream,java.lang.String)void

Now to work out how to actually invoke the method via the MethodHandle, assuming that it is even possible with this version of the code.


Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

August 19, 2009

A Standalone Android Runtime: Getting Started

So what do you need to run an Android application ? At a minimum it looks like you need the System Server which is implemented by the class

    com.android.server.SystemServer

which is in

    $(ANDROID_SRC)/frameworks/base/services/java/com/android/server/SystemServer.java

Usefully it has a main(String[]) method, which doesn’t look very complicated at all, so what happens if we call it ?

It falls over.

The first thing it does is invoke the

     dalvik.system.VMRuntime.setTargetHeapUtilization(float)

method. This is in turn calls the

    dalvik.system.VMRuntime.nativeSetTargetHeapUtilization(float)

which unsuprisingly turns out to be a native method, which we don’t have.

We don’t want any Dalvik dependencies, and we don’t want any native code, but for the moment we’ll just replace the native call with a print statement so we don’t forget we have to do something about it later.

The main(String[]) method then wants to load the android_servers native library. We don’t have one of those, so we’ll comment that out.

Finally it calls the init1(String[]) static method. This is a native method so we could comment it out as well, but then we wouldn’t have a whole lot of functionality.

The init1(String[]) method is defined in

    $(ANDROID_SRC)/frameworks/base/services/jni/com_android_server_SystemServer.cpp

and it turns out to be a call to the

    system_init()

function and nothing more.

The system_init() function is defined in

    $(ANDROID_SRC)/frameworks/base/cmds/system_server/library/system_init.cpp

This does some interesting things we don’t quite undestand yet, and then calls back into Java, invoking the

    com.android.server.SystemServer.init2()

static method.

The comment for the main(String[]) method does actually say that this is what happens but comments have been known to get out of sync with the code so it is worth checking, and we also need to see what else is being done, just in case it turns out that anything else important gets done natively at this point.

Possibly it does but at the moment its not clear what, so we will short-cut the long and roundabout method of calling the init2() method and call it directly from main. It might work.

The init2() method simply forks a thread implemented by the class

    com.android.server.ServerThread

and then the fun really starts


Copyright (c) 2009 By Simon Lewis. All Rights Reserved.

November 9, 2008

Workaround

Filed under: Android, Dalvik, Java, Java VM, Maemo, N800, Nokia — Tags: , , , , , , — Simon Lewis @ 12:32 am

It looks as though the problem with mmap() is that the filesystem doesn’t support making mmap()ed files writable which prevents Dalvik as currently implemented from optimizing its Dex file, which prevents it from starting. A workaround for this is to provide Dalvik with an optimized Dex file we prepared earlier. If you do then you get this

screenshot00

November 8, 2008

Dalvik Running Standalone (but not quite working) on an N800

Filed under: Android, Dalvik, Java, Java VM, Maemo, N800, Nokia — Tags: , , , , , , — Simon Lewis @ 2:52 pm

A screenshot of Dalvik running in an xterm on an N800

screenshot05

The error (which has scrolled off the top) seems to be a failed call to mmap()

November 7, 2008

Dalvik Running Standalone On MacOSX

Filed under: Android, Dalvik, Java, Java VM, MacOSX — Tags: , , , , — Simon Lewis @ 5:35 pm

When the source for Google’s Dalvik Java VM became available as part of the Android open source release I thought I would see whether it was possible to build and run it standalone, i.e not as part of Android.

The answer is yes, and here’s a shot of it running in the debugger under Xcode on MacOSX to prove it

dalvik

Unfortunately the canonical HelloWorld application on its own doesn’t make for particularly interesting viewing

With verbose logging on the output is certainly a lot more voluminous.

Here’s the debugger window scrolled to different positions to show all of it

dalvik_verbose_a3

dalvik_verbose_b

dalvik_verbose_c2

Blog at WordPress.com.

%d bloggers like this: