Just An Application

November 17, 2016

Java To Swift: Much Ado About Null

Filed under: Java, Java2Swift, Swift — Tags: , , — Simon Lewis @ 12:09 pm

The Java literal null is the only value of the null type.

The null type is anonymous, that is to say, there is no way to refer to it in a Java program so it is not possible to declare anything to be of type null.

But not to worry, although nothing can be declared to be of type null, any field, local variable or parameter which is a reference, that is to say, whose type is declared to be an array, class or interface, can have the value null.

The following code shows the things that can be done with a reference in Java

    package xper.nullexample;

    public class Cod 
    {
	public void swim()
	{
	    // TODO
	}
	
	public int numberOfFins;
    }

    public class XperNullExampleMain 
    {
	public static void main(String[] args)
	{
	    test((Cod)null);
	}
	
	private static void test(Cod fish)
	{
	    if (fish != null)
	    {
	        System.out.println("fish is not null");
	    }
	    else
	    {
		System.out.println("fish is null");
	    }
	    if (fish == null)
	    {
		System.out.println("fish is still null");
	    }
	    else
	    {
		System.out.println("fish is not null !");
	    }
	    if (fish instanceof Cod)
	    {
		System.out.println("fish is an instanceof Cod");
	    }
	    else
	    {
		System.out.println("fish is not an instanceof Cod");
	    }
		
	    System.out.println(fish + " concatenated with string");

	    try
	    {
		System.out.format("fish has %d fins\n", fish.numberOfFins);
	    }
	    catch (NullPointerException npe)
	    {
		System.out.println("Caught NPE when accessing numberOfFins field");
	    }
	    try
	    {
		fish.swim();
		System.out.println("fish is swimming");
	    }
	    catch (NullPointerException npe)
	    {
		System.out.println("Caught NPE when invoking swim method");
	    }
        }
    }	

They are

  • field access
  • method invocation
  • string concatenation
  • casting
  • comparison using == and !=
  • type checking using instanceof

As the example also shows, if you run it, all but two of these work even when the reference is null

The two that do not work are

  • field access

and

  • method invocation

Attempting to use a reference with the value null to access a field or invoke a method or will result in a NullPointerException (NPE).

There is no null in Swift.

The nearest thing is nil but in some respects nil is the anti-null.

In Java the field declaration

    private java.lang.Character c;

means that c can be a reference to an object of type java.lang.Character or null.

In Swift the declaration

    private c : java_lang_Character

means that c cannot be nil.

So whereas in Java the default is that c can be null, in Swift the default is that c cannot be nil.

In Swift an ‘equivalent’ to the Java declaration would be

    private c : java_lang_Character?

The property c must be explicitly declared optional.

To invoke a method on c in Java we can simply do

    c.compareTo(d)

and if c has the value null we get an NPE.

There are two ways of doing the same thing in Swift.

Either

    c?.compareTo(d)

or

    c!.compareTo(d)

The first is optional-chaining, the second is forced unwrapping.

Optional chaining results in the value nil if something in the chain is nil or an optional value of the same type as the last thing in the chain.

Optional chaining works by stopping the first time it encounters nil.

If c is nil in the first example then nothing happens. No attempt is made to invoke the compareTo method.

Forced unwrapping takes a more robust approach.

If c is not nil then the result is a value of type java_lang_Character. If c is nil then the program goes bang.

To preserve the runtime behaviour of Java in Swift when using nil as null is obviously going to require some additional effort.

This is the definition of the compareTo method for java.lang.Character

    public int compareTo(Character anotherCharacter) {
        return compare(this.value, anotherCharacter.value);
    }

The value field is defined like this

    private final char value;

and the compare method is defined like this

    public static int compare(char x, char y) {
        return x - y;
    }

Since anotherCharacter can legitimately be null in Java, the equivalent in Swift would have to be something like

    public func compareTo(anotherCharacter:java_lang_Character?)
    {
        return java_lang_Character.compare(self.value, anotherCharacter?.value)
    }

Assuming the compare method is defined something like this in Swift

    public func compare(x:JavaChar, y:JavaChar) -> Int
    {
        return x - y
    }

and the field value like this

 private let value : JavaChar

then compareTo won’t compile.

The result of the optional chaining on anotherChar is going to be an optional JavaChar

    JavaChar?

To make it compile it is necessary to unwrap anotherCharacter.

One way to do this is like so

    public func compareTo(anotherCharacter:java_lang_Character?)
    {
        return java_lang_Character.compare(self.value, anotherCharacter!.value)
    }

This now compiles but if anotherCharacter is nil then the program stops abruptly.

To preserve the Java runtime semantics it should throw a NullPointerException.

One way to achieve this by defining a ‘magic’ function which checks whether is something is nil, throws an NPE if it is, or returns the unwrapped value if it is not.

The signature needs to be something like this.

    func JRTEnsureNotNull(reference:R?) throws -> R

Assuming such a thing existed it would be possible to define the compareTo ‘method’ like this

    func compareTo(anotherCharacter:java_lang_Character?) throws
    {
        return java_lang_Character.compare(self.value, try JRTEnsureNotNull(anotherCharacter).value)
    }

Because the function JRTEnsureNotNull is declared to throw the call to it must be prefaced by try and since there is no attempt to catch whatever is thrown the ‘method’ itself must be declared to throw as well.

Clearly there is an overhead associated with doing this but the same overhead must exist in some form in the Java runtime since it can detect when a reference is null and throw a NullPointerException if necessary.

In addition, it should be possible in some circumstances to reduce this overhead during the conversion from Java to Swift by identifying when a reference has successfully been dereferenced and using it directly on subsequent occasions rather than checking it each time.

Advertisements

November 16, 2016

Java To Swift: Packages, Imports And Access Control

Filed under: Java, Java2Swift, Swift — Tags: , , — Simon Lewis @ 11:30 am

Packages

Java has packages and is in the process of acquiring modules.

Swift has modules.

It is tempting to map Java packages on to Swift modules but, amongst other things, that would leave the vexed of question of what to map Java modules on to when they eventually arrive.

Instead I’m going to go with a flat namespace for now.

Everything gets named explicitly using the package prefix, for example,

   java.lang.Boolean

becomes

   java_lang_Boolean

Imports

Swift has an import declaration which is pretty much to Swift modules as the Java import declaration is to Java packages but since I’m not using Swift modules to represent Java packages I don’t really have a use for it.

Rather more useful in this context is the typealias declaration, for example, declaring

   public typealias Object = java_lang_Object

means that I can write

    public final java_lang_Boolean: Object ... 

Not brilliant but its a start.

Given its uniquity being able to declare

   public typealias String = java_lang_String

would be handy but it collides with the Swift’s String so it can’t be done.

Access Control

Java has

  • public
  • protected
  • private

and the access control level with no name, aka package

Now the dust has settled Swift has

  • open
  • public
  • internal
  • fileprivate
  • private

Of these, the first three are module-relative so strictly speaking they are not relevant but public is at least familiar so I’ll use that.

The fileprivate level doesn’t correspond to anything in Java but may be useful for transformation generated artifacts.

Swift’s private which does correspond to Java’s private so I’ll use that as well.

That leaves Java’s protected and package without an equivalent so they will just have to be Swift internal for now.

November 13, 2016

Java To Swift: Thirty Three And A Third Things You Might Like To Know

Filed under: Java, Java2Swift, Swift — Tags: , , — Simon Lewis @ 8:26 pm

What ?

Its not my fault. Its an epidemic. Everybody’s doing it.

Consider yourselves lucky I didn’t go with

Java To Swift: You’ll Be Astonished By What Happens Next !

Anyway Java has classes and Swift has classes, so transforming Java source into Swift source should be trivial.

Probably an afternoon’s work, if that.

Well, maybe, but probably not.

Here is the result of running my first attempt at a tool for turning Java into Swift on Boolean.java.

import Foundation


public final class java_lang_Boolean: java_lang_Object
{
    public init(_ value:JavaBoolean)
    {
        self.value = value
        
    }
    
    public convenience init(_ s:java_lang_String?)
    {
        self.init(java_lang_Boolean.parseBoolean(s))
        
    }
    
    public static func parseBoolean(s:java_lang_String?) -> JavaBoolean
    {
        return s != nil && s!.equalsIgnoreCase(javaStringConstant("true"))
    }
    
    public func booleanValue() -> JavaBoolean
    {
        return value
    }
    
    public static func valueOf(b:JavaBoolean) -> java_lang_Boolean
    {
        return b ? TRUE : FALSE
    }
    
    public static func valueOf(s:java_lang_String?) -> java_lang_Boolean
    {
        return parseBoolean(s) ? TRUE : FALSE
    }
    
    public static func toString(b:JavaBoolean) -> java_lang_String
    {
        return b ? javaStringConstant("true") : javaStringConstant("false")
    }
    
    public func toString() -> java_lang_String
    {
        return value ? javaStringConstant("true") : javaStringConstant("false")
    }
    
    public func hashCode() -> JavaInt
    {
        return java_lang_Boolean.hashCode(value)
    }
    
    public static func hashCode(value:JavaBoolean) -> JavaInt
    {
        return value ? JavaInt(1231) : JavaInt(1237)
    }
    
    public func equals(obj:java_lang_Object?) -> JavaBoolean
    {
        if obj is java_lang_Boolean
        {
            return value == (obj as! java_lang_Boolean).booleanValue()
        }
        return false
    }
    
    public static func getBoolean(name:java_lang_String?) -> JavaBoolean
    {
        var result : JavaBoolean = false
        do
        {
            result = parseBoolean(java_lang_System.getProperty(name))
        }
        catch let e where e is java_lang_IllegalArgumentException || e is java_lang_NullPointerException
        {
        }
        return result
    }
    
    public func compareTo(b:java_lang_Boolean?) -> JavaInt
    {
        return java_lang_Boolean.compare(self.value, b!.value)
    }
    
    public static func compare(x:JavaBoolean, _ y:JavaBoolean) -> JavaInt
    {
        return x == y ? JavaInt(0) : x ? JavaInt(1) : JavaInt(-1)
    }
    
    public static func logicalAnd(a:JavaBoolean, _ b:JavaBoolean) -> JavaBoolean
    {
        return a && b
    }
    
    public static func logicalOr(a:JavaBoolean, _ b:JavaBoolean) -> JavaBoolean
    {
        return a || b
    }
    
    public static func logicalXor(a:JavaBoolean, _ b:JavaBoolean) -> JavaBoolean
    {
        return a ^ b
    }
    
    public static let TRUE : java_lang_Boolean = java_lang_Boolean( true)
    
    public static let FALSE : java_lang_Boolean = java_lang_Boolean(false)
    
    private let value : JavaBoolean
    
    private static let serialVersionUID : JavaLong = JavaLong(-3665804199014368530)
    
}

Pretty convincing don’t you think ?

It even compiles with a bit of judicious pushing and shoving.

But looks can be deceptive.

The class java.lang.Boolean isn’t a particularly Java’ry class as Java classes go so it is not really a very good test.

There are a number of ways in which Java and Swift are not at all alike, so it looks as though it is probably going to be more like two afternoon’s work.

September 7, 2014

The Mystery Of The Unsigned JAR: Part Three — The Little JAR That Hid

This is the bytecode for a method called load with everything removed except the calls to methods in the Java APIs.

As you can see, it is possible to deduce what the method is doing solely on the basis of the Java API method calls.


    public void load();
      Code:
         0: aload_0
         1: dup
         2: invokevirtual #6                  // Method java/lang/Object.getClass:()Ljava/lang/Class;
         
            ...
            
        11: invokevirtual #7                  // Method java/lang/Class.getResourceAsStream:(Ljava/lang/String;)Ljava/io/InputStream;
        
            ...
        
        47: new           #12                 // class java/util/jar/JarInputStream
        50: dup
        51: new           #13                 // class java/io/ByteArrayInputStream
        54: dup
        55: aload_1
        56: invokespecial #14                 // Method java/io/ByteArrayInputStream."<init>":([B)V
        59: invokespecial #15                 // Method java/util/jar/JarInputStream."<init>":(Ljava/io/InputStream;)V
        62: astore_2
        
            ...
            
        75: aload_2
        76: invokevirtual #16                 // Method java/util/jar/JarInputStream.getNextJarEntry:()Ljava/util/jar/JarEntry;
        79: dup
        80: astore        4
        82: ifnull        235
        85: aload         4
        87: invokevirtual #17                 // Method java/util/jar/JarEntry.getName:()Ljava/lang/String;
        90: dup
        91: astore        5
        
            ...
            
        99: invokevirtual #18                 // Method java/lang/String.endsWith:(Ljava/lang/String;)Z
       102: ifeq          172
       
            ...
            
       117: iconst_0
       118: iconst_1
       119: dup
       120: pop2
       121: aload         5
       123: dup_x1
       124: invokevirtual #19                 // Method java/lang/String.length:()I
       127: bipush        6
       129: iconst_1
       130: dup
       131: pop2
       132: isub
       133: invokevirtual #20                 // Method java/lang/String.substring:(II)Ljava/lang/String;
       136: bipush        47
       138: iconst_1
       139: dup
       140: pop2
       141: bipush        46
       143: iconst_1
       144: dup
       145: pop2
       146: invokevirtual #21                 // Method java/lang/String.replace:(CC)Ljava/lang/String;
       149: astore        5
       
            ...
            
       163: aload_3
       164: invokevirtual #22                 // Method java/util/HashMap.put:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
       167: pop
       168: goto          75
       171: pop
       172: aload         4
       174: invokevirtual #23                 // Method java/util/jar/JarEntry.isDirectory:()Z
       177: ifeq          185
       180: aload_2
       181: goto          76
      
            ...

The calls to the JarInputStream and JarEntry methods make it clear that the method is iterating over the contents of a JAR

The 6 in

    127: bipush        6

just happens to be the length of the suffix

    .class

The 47 in

    136: bipush        47

is the solidus character (‘/‘) and the 46 in

    141: bipush        46

is the dot character (‘.‘).

The sequences following the iconst_0 instruction and each bipush instruction

    117: iconst_0
    118: iconst_1
    119: dup
    120: pop2
    
    ...

    127: bipush        6
    129: iconst_1
    130: dup
    131: pop2
    
    ...
    
    136: bipush        47
    138: iconst_1
    139: dup
    140: pop2
    
    ...

    141: bipush        46
    143: iconst_1
    144: dup
    145: pop2

are all either pretty feeble attempts at obfuscation or evidence of a severely confused compiler as they have no effect at all.

From time immemorial Java class loaders have been converting the JarEntry names of class files into class names and that is what this method is doing as well.

If you have a class, for example,

    xper.mcm.CrashPow

then the name of the JarEntry for the class file in a JAR will be

    xper/mcm/CrashPow.class

Chop off the .class suffix

    117: iconst_0

      0
      
    118: iconst_1
    
      0, 1
      
    119: dup
    
      0, 1, 1

    120: pop2
    
      0
      
    121: aload         5
    
      0, entry_name
      
    123: dup_x1
    
      entry_name, 0, entry_name
      
    124: invokevirtual #19                 // Method java/lang/String.length:()I
    
      entry_name, 0, length(entry_name)
      
    127: bipush        6
    
      entry_name, 0, length(entry_name), 6
      
    129: iconst_1
    
      entry_name, 0, length(entry_name), 6, 1
      
    130: dup
    
      entry_name, 0, length(entry_name), 6, 1, 1
    
    131: pop2
    
      entry_name, 0, length(entry_name), 6
    
    132: isub
    
      entry_name, 0, length(entry_name)_minus_6
      
    133: invokevirtual #20                 // Method java/lang/String.substring:(II)Ljava/lang/String;

      substring(entry_name, 0, length(entry_name)_minus_6))

and replace the occurrences of ‘/‘ with ‘.

    136: bipush        47
    
      substring(...), 47
      
    138: iconst_1
    
      substring(...), 47, 1
      
    139: dup
    
      substring(...), 47, 1, 1
      
    140: pop2
    
      substring(...), 47
      
    141: bipush        46
    
       substring(...), 47, 46
       
    143: iconst_1
    
      substring(...), 47, 46, 1
      
    144: dup
    
      substring(...), 47, 46, 1, 1

    145: pop2
    
      substring(...), 47, 46
       
    146: invokevirtual #21                 // Method java/lang/String.replace:(CC)Ljava/lang/String;
    
      substring(...).replace(47, 46)
        
    149: astore        5

and you get the canonical Java class name back again

The method is storing the contents of the class files in the JAR it is iterating over in a HashMap with the class names as the keys.

Now where’s the JAR ?


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 6, 2014

The Mystery Of The Unsigned JAR: Part Two — The Sekrit Method

The ‘sekrit method’ takes a single String argument and returns a String and it could not be more obvious if it tried.

Unlike the other methods in the class it has a non-meaningful name. Instead its name features a large subset of the lower case letters of the alphabet in order. I don’t think there is any coding standard that requires that, ‘tho I suppose there could be.

It is always passed a String literal and the String literal is always gibberish, yet the return values are often passed to factory methods in the Java API which take specific well-formed names.

The inference is obvious. The method takes obfuscated Strings and turns them into unobfuscated Strings.

The nice thing about Java VM bytecode is that you can decompile it using nothing more than a pencil and a piece of paper, oh, and a knowledge of the Java VM opcodes !

Being stack based it is pretty easy to work out what is going on by tracing the state of the stack and the local variables after each instruction.

Of course if you don’t have a piece of paper and a pencil handy you can always write some code to do the same thing.

The following shows the stack/local variable state after each instruction for a large part of the sekrit method.

The top of the stack is on the right.

    ...

    
    34: invokevirtual #232                // Method java/lang/StringBuffer.toString:()Ljava/lang/String;
    
        S
    
    37: dup
    
        S, S
    
    38: invokevirtual #19                 // Method java/lang/String.length:()I
    
        S, length(S)
    
    41: iconst_1
    
        S, length(S), 0x01
    
    42: isub
    
        S, length(S)_minus_one
    
    43: iconst_2
    
        S, length(S)_minus_one, 0x02
    
    44: iconst_3
    
        S, length(S)_minus_one, 0x02, 0x03
    
    45: ishl
    
        S, length(S)_minus_one, 0x10
    
    46: iconst_3
    
        S, length(S)_minus_one, 0x10, 0x03
    
    47: iconst_5
    
        S, length(S)_minus_one, 0x10, 0x03, 0x05
    
    48: ixor
    
        S, length(S)_minus_one, 0x10, 0x06
    
    49: ixor
    
        S, length(S)_minus_one, 0x16
    
    50: iconst_2
    
        S, length(S)_minus_one, 0x16, 0x02
    
    51: iconst_5
    
        S, length(S)_minus_one, 0x16, 0x02, 0x05
    
    52: ixor
    
        S, length(S)_minus_one, 0x16, 0x07
    
    53: iconst_3
    
        S, length(S)_minus_one, 0x16, 0x07, 0x03
    
    54: ishl
    
        S, length(S)_minus_one, 0x16, 0x38
    
    55: iconst_2
    
        S, length(S)_minus_one, 0x16, 0x38, 0x02
    
    56: ixor
    
        S, length(S)_minus_one, 0x16, 0x3A
    
    57: iconst_2
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x02
    
    58: iconst_5
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x02, 0x05
    
    59: ixor
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x07
    
    60: iconst_4
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x07, 0x04
    
    61: ishl
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70
    
    62: iconst_1
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70, 0x01
    
    63: dup
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70, 0x01, 0x01
    
    64: ishl
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x70, 0x02
    
    65: ixor
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72
    
    66: aload_0
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, avar0
    
    67: invokevirtual #19                 // Method java/lang/String.length:()I
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0)
    
    70: dup
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), length(avar0)
    
    71: newarray       char
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)]
    
    73: iconst_1
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)], 0x01
    
    74: dup
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)], 0x01, 0x01
    
    75: pop2
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, length(avar0), char[length(avar0)]
    
    76: swap
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, char[length(avar0)], length(avar0)
    
    77: iconst_1
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, char[length(avar0)], length(avar0), 0x01
    
    78: isub
    
        S, length(S)_minus_one, 0x16, 0x3A, 0x72, char[length(avar0)], length(avar0)_minus_1
    
    79: dup_x2
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1, 0x72, char[length(avar0)], length(avar0)_minus_1
    
    80: istore_1
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1, 0x72, char[length(avar0)]
    
        {
            ivar1 == length(avar0)_minus_1
        }
    
    81: astore_3
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1, 0x72
    
        {
            ivar1 == length(avar0)_minus_1
            avar3 == char[length(avar0)]
        }
    
    
    82: istore        7
    
        S, length(S)_minus_one, 0x16, 0x3A, length(avar0)_minus_1
    
        {
            ivar1 == length(avar0)_minus_1
            avar3 == char[length(avar0)]
            ivar7 == 0x72
        }
    
    84: dup_x2
    
        S, length(S)_minus_one, length(avar0)_minus_1, 0x16, 0x3A, length(avar0)_minus_1
    
    85: pop
    
        S, length(S)_minus_one, length(avar0)_minus_1, 0x16, 0x3A
    
    86: istore        4
    
        S, length(S)_minus_one, length(avar0)_minus_1, 0x16
    
        {
            ivar1 == length(avar0)_minus_1
            avar3 == char[length(avar0)]
            ivar4 == 0X3A
            ivar7 == 0x72
        }
    
    88: pop
    
        S, length(S)_minus_one, length(avar0)_minus_1
    
    89: swap
    
        S, length(avar0)_minus_1, length(S)_minus_one
    
    90: dup
    
        S, length(avar0)_minus_1, length(S)_minus_one, length(S)_minus_one
    
    91: istore_2
    
        S, length(avar0)_minus_1, length(S)_minus_one
    
        {
            ivar1 == length(avar0)_minus_1
            ivar2 == length(S)_minus_one
            avar3 == char[length(avar0)]
            ivar4 == 0X3A
            ivar7 == 0x72
        }
    
    92: istore        5
    
        S, length(avar0)_minus_1
    
        {
            ivar1 == length(avar0)_minus_1
            ivar2 == length(S)_minus_one
            avar3 == char[length(avar0)]
            ivar4 == 0X3A
            ivar5 == length(S)_minus_one
            ivar7 == 0x72
    }
    
    94: swap
    
        length(avar0)_minus_1, S
    
    95: astore        6
    
        length(avar0)_minus_1
    
        {
            ivar1 == length(avar0)_minus_1
            ivar2 == length(S)_minus_one
            avar3 == char[length(avar0)]
            ivar4 == 0x3A
            ivar5 == length(S)_minus_one
            avar6 == S
            ivar7 == 0x72
        }
    
    97: goto          163
    
    
    100: aload_3
    
        char[length(avar0)]
    
    101: iload         4
    
        char[length(avar0)], 0x3A
    
    103: aload_0
    
        char[length(avar0)], 0x3A, avar0
    
    104: iload_1
    
        char[length(avar0)], 0x3A, avar0, ivar1
    
    105: iinc          1, -1
    
        char[length(avar0)], 0x3A, avar0, ivar1
    
    108: dup_x2
    
        char[length(avar0)], ivar1, 0x3A, avar0, ivar1
    
    109: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        char[length(avar0)], ivar1, 0x3A, c
    
    112: aload         6
    
        char[length(avar0)], ivar1, 0x3A, c, avar6
    
    114: iload_2
    
        char[length(avar0)], ivar1, 0x3A, c, avar6, ivar2
    
    115: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        char[length(avar0)], ivar1, 0x3A, c, d
    
    118: ixor
    
        char[length(avar0)], ivar1, 0x3A, c_xor_d
    
    119: ixor
    
        char[length(avar0)], ivar1, c_xor_d_xor_0x3A
    
    120: i2c
    
        char[length(avar0)], ivar1, c_xor_d_xor_0x3A
    
    121: castore
    
    
    122: iload_1
    
        ivar1
    
    123: ifge          130
    126: aload_3
    127: goto          167
    
    
    130: aload_3
    
        avar3
    
    131: iload         7
    
        avar3, 0x72
    
    133: aload_0
    
        avar3, 0x72, avar0
    
    134: iload_1
    
        avar3, 0x72, avar0, ivar1
    
    135: dup_x2
    
        avar3, ivar1, 0x72, avar0, ivar1
    
    136: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        avar3, ivar1, 0x72, c
    
    139: aload         6
    
        avar3, ivar1, 0x72, c, avar6
    
    141: iload_2
    
        avar3, ivar1, 0x72, c, avar6, ivar2
    
    142: invokevirtual #236                // Method java/lang/String.charAt:(I)C
    
        avar3, ivar1, 0x72, c, d
        
    145: ixor
    
        avar3, ivar1, 0x72, c_xor_d
    
    146: ixor
    
        avar3, ivar1, c_xor_d_xor_0x72
    
    147: i2c
    
        avar3, ivar1, c_xor_d_xor_0x72
    
    148: iinc          1, -1
    151: iinc          2, -1
    154: castore
    
    
    155: iload_2
    
        ivar2
    
    156: ifge          162
    
    159: iload         5
    
        ivar5
        
    161: istore_2
    
    
    162: iload_1
    
        ivar1
        
    163: ifge          100
    
    
    166: aload_3
    
        avar3
        
    167: invokespecial #239                // Method java/lang/String."<init>":([C)V
        
        
    170: areturn

The method is basically is a loop which starts at

    100: aload_3

and ends at

    163: ifge          100

Everything before the loop is initialization code, some of which seems to have been partially obfuscated by the use of various bitwise operators acting on small integers.

Following the initialization we can deduce the following.

The local variable avar0 holds the obfuscated String passed as the argument.

The local variable avar3 holds an array of chars which will contain the unobfuscated characters.

The local variable ivar1 holds the index used to access the characters in avar0 and avar3. Note that it being decremented

    105: iinc          1, -1

and, compared against 0

    123: ifge          130

so both avar0 and avar3 are being iterated over backwards.

The local variable ivar2 is used in the same way to iterate backwards over the String in avar6.

Each time round the loop a character is added to avar3. The character is the result of XORing a character from avar0 and a character from avar6 and the constant 0x3A. [Instructions 100 to 121]

If there are still characters to be added a second character is added which is the result of XORing a character from avar0 and a character from avar6 and the constant 0x72. [Instructions 130 to 154]

Note that ivar1 is being decremented twice each time round the loop but ivar2 only once.

The String in avar6 is constructed using the name of the calling method so the obfuscation is context sensitive.

If you convert all of the above back into Java you will probably end up with something like this

        ...
    
        int     lengthA = a.length();
        int     lengthS = s.length();
        char[]  chars   = new char[lengthA];
        int     i       = lengthA - 1;
        int     j       = lengthS - 1;
    
        while (true)
        {
            char c = a.charAt(i);
            char d = s.charAt(j);
    
            chars[i] = (char)(c ^ d ^ 0x3a);
    
            --i;
            if (i < 0)
            {
                break;
            }
    
            c = a.charAt(i);
            d = s.charAt(j);
    
            chars[i] = (char)(c ^ d ^ 0x72);
    
            --i;
            --j;
            if (j < 0)
            {
                j = lengthS -1;
            }
            if (i < 0)
            {
                break;
            }
        }
        return new String(chars);

where a is the obfuscated String being passed as the argument and s is the context sensitive ‘mask’.

I doubt whether it compiles into exactly the same bytecode but it definitely deobfuscates the String literals in the class files correctly.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

July 28, 2013

The Great Android Security Hole Of ’08 ? – Appendix: JAR Signing The Easy Way (TM)

1.0 How To Sign A JAR The Easy Way (TM)

  1. You create a digital signature for the entire JAR file as specified by one of the more reputable digital signature standards, PKCS#1 for example.

  2. That’s it.

2.0 How To Verify A JAR That Has Been Signed The Easy Way (TM)

  1. You compute the digest of the entire JAR file and verify it using the digital signature.

  2. That’s it.

3.0 Packaging A JAR That Has Been Signed The Easy Way (TM)

Given a JAR and its signature and the certificate chain of the signer you can either package them together or separately.

3.1 Packaging Them Together

To package them together you can either invent your own mechanism, not recommended, or you can use an existing one such as CMS/PKCS#7 with the signed data i.e. the JAR file, inline.

3.2 Packaging Them Separately

3.2.1 Use CMS/PKCS#7

You can also use CMS/PKCS#7 to package the signature separately from the JAR file, i.e. with the signed data, the JAR file, out-of-line.

If you are going to all the trouble of using CMS/PKCS#7 its unclear why you would want to do this, but you can if you want.

3.2.2 Define A New File Format

The alternative is to define a new file format.

At a minimum the file needs to include the encrypted signature and the certificate chain for the signer but it could also include other information about the Application, like how big the JAR is and where to download it from and what permissions it needs and things like that.

Including additional information could potentially enable optimizations such as deciding that you do not trust the signer before you have even downloaded the JAR.

You could even come up with a fancy name for your new file format. Something like JAR Descriptor, or even Java Application Descriptor, or JAD for short.

4.0 The Advantages Of JAR Signing The Easy Way (TM)

The overwhelming advantage of JAR signing the Easy Way (TM) is simplicity, particularly the simplicity of verification.

The opportunity for getting things wrong is orders of magnitude less than when verifying a signed JAR and at the end of it you also know that what you have got is exactly what was signed.


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content

The Great Android Security Hole Of ’08 ? – Part Nine: The Root Cause ? — Signed JARs Considered Harmful

I would argue that the root cause of the Great Android Security Hole Of ’08 was the decision, either explicit or implicit, to use signed JARs as the mechanism for supporting the authentication and verification of Android Applications.

1.0 What Are Signed JARs For ?

If you want to ensure the integrity of a JAR as a self-contained entity such as an Application then the ability to sign individual files is not a requirement.

In fact it is difficult to see in what circumstances the ability to sign individual files and only individual files could be a requirement.

Because it is only possible to sign individual files, a signed JAR is really nothing more than a collection of files which may or may not be signed and the verification of a signed JAR is a very convoluted way of determining into which category each file belongs.

All of which leads to question of what signed JARs are actually for ?

The ability to package files in this way was presumably considered useful when the specification was produced but it is clear that it is a decidedly sub-optimal way of attempting to ensure the integrity of an Application made up of a number of files which have been packaged as a ZIP file.

2.0 Complexity

While signed JARs undoubtedly constitute a flexible mechanism for doing something, its just not clear what, they do so at a cost.

As we have seen the cost is the complexity of the verification process and the inconclusiveness of the result.

The process of verification is ridiculously complicated and consequently dangerously error-prone which is not what you want from something which is a key part of ensuring the security of your platform.

3.0 Conclusion

If you need some way to both authenticate and ensure the integrity of self-contained Applications delivered in ZIP files don’t use signed JARs.

If for some reason you really feel you must

  1. do an exhaustive analysis of your requirements

  2. identify every single assumption you are making at every stage of the verification and authentication process, and then

  3. generate the enormous amount of test collateral you will require to ensure that your assumptions remain true in the face of every single thing that can be done legitimately and illegitimately to signed JARs and ZIP files in isolation and in combination

And then when you have done all that, then think about using something else instead.


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

July 27, 2013

The Great Android Security Hole Of ’08 ? – Part Eight: Signed JAR Verification Revisited

1.0 Basic Verification Using The java.util.jar.JarFile API

1.1 JARVerifierV1

A class that does basic verification of a signed JAR using the java.util.jar.JarFile API.

    final class JARVerifierV1 
                implements
                    JARVerifier
    {
        public boolean verify(File theFile)
        {
            Log.d(TAG, "verify: verifying " + theFile);
            try
            {
                JarFile jf = new JarFile(theFile, true);
			
                try
                {
                    verify(jf);
                    Log.d(TAG, "verify: " + theFile + " verified");
                    return true;
                }
                finally
                {
                    jf.close();
                }
            }
            catch (Exception e)
            {
                Log.d(TAG, "verify: verification failed");
                e.printStackTrace();
                return false;
            }
        }
	
        //
	
        JARVerifierV1()
        {
            buffer = new byte[8192];
        }
	
        //

        private void verify(JarFile theJARFile) 
                     throws
                         Exception
        {
            Enumeration<JarEntry> entries = theJARFile.entries();
		
            while (entries.hasMoreElements())
            {
                verify(theJARFile, entries.nextElement());
            }
		
        }

        private void verify(JarFile theJARFile, JarEntry theEntry) 
                     throws
                         Exception
        {
            Log.d(TAG, "verify: " + theEntry.getName());
		
            InputStream is = theJARFile.getInputStream(theEntry);
		
            while (is.read(buffer) != -1)
            {
			
            }
        }
	
        //
	
        private byte[] buffer;
	
        //
	
        private static final String TAG = "JARVerifierV1";
    }

1.1 Verification Of A Vanilla APK

If we create a vanilla APK which looks like this

    Archive:  GoodApp.apk
      Length     Date   Time    Name
     --------    ----   ----    ----
          624  07-10-13 21:11   res/layout/activity_main.xml
          464  07-10-13 21:11   res/menu/activity_main.xml
         1588  07-10-13 21:11   AndroidManifest.xml
         2504  07-10-13 21:11   resources.arsc
          409  07-10-13 21:11   res/drawable-hdpi/ic_action_search.png
         4129  07-10-13 21:11   res/drawable-hdpi/ic_launcher.png
         1756  07-10-13 21:11   res/drawable-ldpi/ic_launcher.png
          311  07-10-13 21:11   res/drawable-mdpi/ic_action_search.png
         2654  07-10-13 21:11   res/drawable-mdpi/ic_launcher.png
          491  07-10-13 21:11   res/drawable-xhdpi/ic_action_search.png
         5456  07-10-13 21:11   res/drawable-xhdpi/ic_launcher.png
       366112  07-10-13 21:11   classes.dex
         1034  07-10-13 21:11   META-INF/MANIFEST.MF
         1087  07-10-13 21:11   META-INF/CERT.SF
          776  07-10-13 21:11   META-INF/CERT.RSA
     --------                   -------
       389395                   15 files

and we attempt to verify it using JARVerifierV1 we succeed.

    ...

    D/JARVerifierV1(  281): verify: verifying /data/data/xper.jv/files/GoodApp.apk
    D/JARVerifierV1(  281): verify: res/layout/activity_main.xml
    D/JARVerifierV1(  281): verify: res/menu/activity_main.xml
    D/JARVerifierV1(  281): verify: AndroidManifest.xml
    D/JARVerifierV1(  281): verify: resources.arsc
    D/JARVerifierV1(  281): verify: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV1(  281): verify: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV1(  281): verify: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV1(  281): verify: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV1(  281): verify: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV1(  281): verify: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV1(  281): verify: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV1(  281): verify: classes.dex
    D/JARVerifierV1(  281): verify: META-INF/MANIFEST.MF
    D/JARVerifierV1(  281): verify: META-INF/CERT.SF
    D/JARVerifierV1(  281): verify: META-INF/CERT.RSA
    D/JARVerifierV1(  281): verify: /data/data/xper.jv/files/GoodApp.apk verified

    ...

This is what we would expect but it is not much of a test.

We need to know what happens when we modify a signed JAR in some way.

1.2 Verification After File Modification

If we modify our vanilla APK by replacing the original version of AndroidManifest.xml with an empty one

    Archive:  GoodApp_modified.apk
      Length     Date   Time    Name
     --------    ----   ----    ----
          624  07-10-13 21:11   res/layout/activity_main.xml
          464  07-10-13 21:11   res/menu/activity_main.xml
            0  07-21-13 11:11   AndroidManifest.xml
         2504  07-10-13 21:11   resources.arsc
          409  07-10-13 21:11   res/drawable-hdpi/ic_action_search.png
         4129  07-10-13 21:11   res/drawable-hdpi/ic_launcher.png
         1756  07-10-13 21:11   res/drawable-ldpi/ic_launcher.png
          311  07-10-13 21:11   res/drawable-mdpi/ic_action_search.png
         2654  07-10-13 21:11   res/drawable-mdpi/ic_launcher.png
          491  07-10-13 21:11   res/drawable-xhdpi/ic_action_search.png
         5456  07-10-13 21:11   res/drawable-xhdpi/ic_launcher.png
       366112  07-10-13 21:11   classes.dex
         1034  07-10-13 21:11   META-INF/MANIFEST.MF
         1087  07-10-13 21:11   META-INF/CERT.SF
          776  07-10-13 21:11   META-INF/CERT.RSA
     --------                   -------
       387807                   15 files

and then attempt to verify it we get a SecurityException which is what we would expect.

    ...

    D/JARVerifierV1(  280): verify: verifying /data/data/xper.jv/files/GoodApp_modified.apk
    D/JARVerifierV1(  280): verify: res/layout/activity_main.xml
    D/JARVerifierV1(  280): verify: res/menu/activity_main.xml
    D/JARVerifierV1(  280): verify: AndroidManifest.xml
    D/JARVerifierV1(  280): verify: verification failed
    W/System.err(  280): java.lang.SecurityException: META-INF/MANIFEST.MF has \
         invalid digest for AndroidManifest.xml in /data/data/xper.jv/files/GoodApp_modified.apk
    W/System.err(  280): 	at java.util.jar.JarVerifier$VerifierEntry.verify(JarVerifier.java:129)
    W/System.err(  280): 	at java.util.jar.JarFile$JarFileInputStream.read(JarFile.java:127)
    W/System.err(  280): 	at java.io.FilterInputStream.read(FilterInputStream.java:130)
    W/System.err(  280): 	at xper.jv.JARVerifierV1.verify(JARVerifierV1.java:71)
    W/System.err(  280): 	at xper.jv.JARVerifierV1.verify(JARVerifierV1.java:58)
    W/System.err(  280): 	at xper.jv.JARVerifierV1.verify(JARVerifierV1.java:24)
    W/System.err(  280): 	at xper.jv.XperJVActivity.verify(XperJVActivity.java:127)
    W/System.err(  280): 	at xper.jv.XperJVActivity.access$0(XperJVActivity.java:117)
    W/System.err(  280): 	at xper.jv.XperJVActivity$1.run(XperJVActivity.java:111)
    W/System.err(  280): 	at java.lang.Thread.run(Thread.java:1096)

    ...

1.3 Verification After File Addition

If we add the file assets/new_asset to our vanilla APK

    Archive:  GoodApp_added.apk
      Length     Date   Time    Name
     --------    ----   ----    ----
          624  07-10-13 21:11   res/layout/activity_main.xml
          464  07-10-13 21:11   res/menu/activity_main.xml
         1588  07-10-13 21:11   AndroidManifest.xml
         2504  07-10-13 21:11   resources.arsc
          409  07-10-13 21:11   res/drawable-hdpi/ic_action_search.png
         4129  07-10-13 21:11   res/drawable-hdpi/ic_launcher.png
         1756  07-10-13 21:11   res/drawable-ldpi/ic_launcher.png
          311  07-10-13 21:11   res/drawable-mdpi/ic_action_search.png
         2654  07-10-13 21:11   res/drawable-mdpi/ic_launcher.png
          491  07-10-13 21:11   res/drawable-xhdpi/ic_action_search.png
         5456  07-10-13 21:11   res/drawable-xhdpi/ic_launcher.png
       366112  07-10-13 21:11   classes.dex
         1034  07-10-13 21:11   META-INF/MANIFEST.MF
         1087  07-10-13 21:11   META-INF/CERT.SF
          776  07-10-13 21:11   META-INF/CERT.RSA
           48  07-24-13 21:56   assets/new_asset
     --------                   -------
       389443                   16 files

and then we attempt to verify it we succeed which is a bit disconcerting.

    ...

    D/JARVerifierV1(  285): verify: verifying /data/data/xper.jv/files/GoodApp_added.apk
    D/JARVerifierV1(  285): verify: res/layout/activity_main.xml
    D/JARVerifierV1(  285): verify: res/menu/activity_main.xml
    D/JARVerifierV1(  285): verify: AndroidManifest.xml
    D/JARVerifierV1(  285): verify: resources.arsc
    D/JARVerifierV1(  285): verify: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV1(  285): verify: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV1(  285): verify: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV1(  285): verify: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV1(  285): verify: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV1(  285): verify: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV1(  285): verify: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV1(  285): verify: classes.dex
    D/JARVerifierV1(  285): verify: META-INF/MANIFEST.MF
    D/JARVerifierV1(  285): verify: META-INF/CERT.SF
    D/JARVerifierV1(  285): verify: META-INF/CERT.RSA
    D/JARVerifierV1(  285): verify: assets/new_asset
    D/JARVerifierV1(  285): verify: /data/data/xper.jv/files/GoodApp_added.apk verified

    ...

In fact the JarFile implementation does not distinguish between a file whose computed digest matches the digest specified in the manifest, and a file which does not have a digest specified in the manifest.

This is a feature just not necessarily a good one.

1.4 Verification After File Removal

If we were to remove a file from our vanilla APK then we know a priori that verification would still succeed because all we are doing is iterating over the files that are in the JAR and verifying them.

To determine whether files had been removed from the JAR would require additional code to look for digests and for the digests of digests.

2.0 Fun With Digests

We can modify the code so that it checks for digest entries in the manifest.

2.1 JarVerifierV2

    final class JARVerifierV2 
                implements
                    JARVerifier
    {
        public boolean verify(File theFile)
        {
            Log.d(TAG, "verify: verifying " + theFile);
            try
            {
                JarFile jf = new JarFile(theFile, true);
			
                try
                {
                    verify(jf);
                    Log.d(TAG, "verify: " + theFile + " verified");
                    return true;
                }
                finally
                {
                    jf.close();
                }
            }
            catch (Exception e)
            {
                Log.d(TAG, "verify: verification failed");
                e.printStackTrace();
                return false;
            }
        }
	
        //
	
        JARVerifierV2()
        {
            buffer = new byte[8192];
        }
	
        //

        private void verify(JarFile theJARFile) 
                     throws
                         Exception
        {
            Enumeration<JarEntry> entries = theJARFile.entries();
		
            while (entries.hasMoreElements())
            {
                verify(theJARFile, entries.nextElement());
            }
		
        }

        private void verify(JarFile theJARFile, JarEntry theEntry) 
                     throws
                         Exception
        {
            Log.d(TAG, "verify: " + theEntry.getName());
		
            ensureDigest(theEntry);
		
            InputStream is = theJARFile.getInputStream(theEntry);
		
            while (is.read(buffer) != -1)
            {
			
            }
        }
	
        private void ensureDigest(JarEntry theEntry)
                     throws
                         Exception
        {
            if (!theEntry.getName().startsWith(META_INF))
            {
                Log.d(TAG, "ensureDigest: " + theEntry.getName());
		
                Attributes attributes = theEntry.getAttributes();
		
                if (attributes != null)
                {
                    for (Object o : attributes.keySet())
                    {
                        if (o.toString().endsWith(DIGEST_SUFFIX))
                        {
                            return;
                        }
                    }
                }
                throw new SecurityException("No digest attribute: " + theEntry.getName());
            }
        }
	
        //
	
        private byte[] buffer;
	
        //
	

        private static final String DIGEST_SUFFIX = "-Digest";
	
        private static final String META_INF      = "META-INF/";
	
        //
	
        private static final String TAG = "JARVerifierV2";
    }


2.2 Verification After File Addition

This time when we attempt we get a SecurityException, which is nice

    ...

    D/JARVerifierV2(  316): verify: verifying /data/data/xper.jv/files/GoodApp_added.apk
    D/JARVerifierV2(  316): verify: res/layout/activity_main.xml
    D/JARVerifierV2(  316): ensureDigest: res/layout/activity_main.xml
    D/JARVerifierV2(  316): verify: res/menu/activity_main.xml
    D/JARVerifierV2(  316): ensureDigest: res/menu/activity_main.xml
    D/JARVerifierV2(  316): verify: AndroidManifest.xml
    D/JARVerifierV2(  316): ensureDigest: AndroidManifest.xml
    D/JARVerifierV2(  316): verify: resources.arsc
    D/JARVerifierV2(  316): ensureDigest: resources.arsc
    D/JARVerifierV2(  316): verify: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV2(  316): verify: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV2(  316): verify: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV2(  316): verify: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV2(  316): verify: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV2(  316): verify: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV2(  316): verify: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV2(  316): ensureDigest: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV2(  316): verify: classes.dex
    D/JARVerifierV2(  316): ensureDigest: classes.dex
    D/JARVerifierV2(  316): verify: META-INF/MANIFEST.MF
    D/JARVerifierV2(  316): verify: META-INF/CERT.SF
    D/JARVerifierV2(  316): verify: META-INF/CERT.RSA
    D/JARVerifierV2(  316): verify: assets/new_asset
    D/JARVerifierV2(  316): ensureDigest: assets/new_asset
    D/JARVerifierV2(  316): verify: verification failed
    W/System.err(  316): java.lang.SecurityException: No digest attribute: assets/new_asset
    W/System.err(  316): 	at xper.jv.JARVerifierV2.ensureDigest(JARVerifierV2.java:100)
    W/System.err(  316): 	at xper.jv.JARVerifierV2.verify(JARVerifierV2.java:70)
    W/System.err(  316): 	at xper.jv.JARVerifierV2.verify(JARVerifierV2.java:59)
    W/System.err(  316): 	at xper.jv.JARVerifierV2.verify(JARVerifierV2.java:25)
    W/System.err(  316): 	at xper.jv.XperJVActivity.verify(XperJVActivity.java:127)
    W/System.err(  316): 	at xper.jv.XperJVActivity.access$0(XperJVActivity.java:117)
    W/System.err(  316): 	at xper.jv.XperJVActivity$1.run(XperJVActivity.java:111)
    W/System.err(  316): 	at java.lang.Thread.run(Thread.java:1096)

    ...

2.3 Manipulating The Manifest

There is a slight problem with this approach, namely that it is possible to add entries to the manifest of a signed JAR.

We can add an entry for our new file to the manifest like so

    Manifest-Version: 1.0
    Created-By: 1.0 (Android)

    Name: res/drawable-xhdpi/ic_launcher.png
    SHA1-Digest: TJE1tg63y88xLdIALKGpuLmgh0s=

    Name: res/drawable-mdpi/ic_action_search.png
    SHA1-Digest: XjltJdEB3tvakTn9CN7KwdaNT68=

    Name: AndroidManifest.xml
    SHA1-Digest: iXwTXVvLX3gY+ByGhybEJvIxP08=

    Name: res/drawable-mdpi/ic_launcher.png
    SHA1-Digest: aM73uFWfPOOvq5Kk9Ffd3cWm0OQ=

    Name: res/drawable-hdpi/ic_launcher.png
    SHA1-Digest: ioH2V9g4FYKkqpDHk7jPryMKtcE=

    Name: res/layout/activity_main.xml
    SHA1-Digest: zd1eNopMfZ7CmMk8GjyPGPpYloc=

    Name: resources.arsc
    SHA1-Digest: 7nYq90PW0OZVg8ytyjBEcwQhYiA=

    Name: classes.dex
    SHA1-Digest: b80KFg5gdqlORweY5LrUS87tbVU=

    Name: res/drawable-hdpi/ic_action_search.png
    SHA1-Digest: rOWnNxWCWKwkicf+FpYGMg1fX2Y=

    Name: res/drawable-xhdpi/ic_action_search.png
    SHA1-Digest: WzSVDDIHZpn0cFeuxgKRJete4TU=

    Name: res/menu/activity_main.xml
    SHA1-Digest: TfxVPQP5IRVlJun7A7PdOqdkp+I=

    Name: res/drawable-ldpi/ic_launcher.png
    SHA1-Digest: r6Mdl54h2qEvVnqgfsgxU4CysiI=

    Name: assets/new_asset
    SHA1-Digest: PW9oQUbA9MQOVYfbtHutF0urHCg=

and then create a new APK containing the added file and the modified manifest

   Archive:  GoodApp_added_modified_manifest.apk
      Length     Date   Time    Name
     --------    ----   ----    ----
          624  07-10-13 21:11   res/layout/activity_main.xml
          464  07-10-13 21:11   res/menu/activity_main.xml
         1588  07-10-13 21:11   AndroidManifest.xml
         2504  07-10-13 21:11   resources.arsc
          409  07-10-13 21:11   res/drawable-hdpi/ic_action_search.png
         4129  07-10-13 21:11   res/drawable-hdpi/ic_launcher.png
         1756  07-10-13 21:11   res/drawable-ldpi/ic_launcher.png
          311  07-10-13 21:11   res/drawable-mdpi/ic_action_search.png
         2654  07-10-13 21:11   res/drawable-mdpi/ic_launcher.png
          491  07-10-13 21:11   res/drawable-xhdpi/ic_action_search.png
         5456  07-10-13 21:11   res/drawable-xhdpi/ic_launcher.png
       366112  07-10-13 21:11   classes.dex
         1103  07-26-13 14:51   META-INF/MANIFEST.MF
         1087  07-10-13 21:11   META-INF/CERT.SF
          776  07-10-13 21:11   META-INF/CERT.RSA
           48  07-24-13 21:56   assets/new_asset
     --------                   -------
       389512                   16 files

If we attempt to verify it we succeed once more

    ...

    D/JARVerifierV2(  280): verify: verifying /data/data/xper.jv/files/GoodApp_added_modified_manifest.apk
    D/JARVerifierV2(  280): verify: res/layout/activity_main.xml
    D/JARVerifierV2(  280): ensureDigest: res/layout/activity_main.xml
    D/JARVerifierV2(  280): verify: res/menu/activity_main.xml
    D/JARVerifierV2(  280): ensureDigest: res/menu/activity_main.xml
    D/JARVerifierV2(  280): verify: AndroidManifest.xml
    D/JARVerifierV2(  280): ensureDigest: AndroidManifest.xml
    D/JARVerifierV2(  280): verify: resources.arsc
    D/JARVerifierV2(  280): ensureDigest: resources.arsc
    D/JARVerifierV2(  280): verify: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV2(  280): verify: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV2(  280): verify: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV2(  280): verify: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV2(  280): verify: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV2(  280): verify: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV2(  280): verify: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV2(  280): ensureDigest: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV2(  280): verify: classes.dex
    D/JARVerifierV2(  280): ensureDigest: classes.dex
    D/JARVerifierV2(  280): verify: META-INF/MANIFEST.MF
    D/JARVerifierV2(  280): verify: META-INF/CERT.SF
    D/JARVerifierV2(  280): verify: META-INF/CERT.RSA
    D/JARVerifierV2(  280): verify: assets/new_asset
    D/JARVerifierV2(  280): ensureDigest: assets/new_asset
    D/JARVerifierV2(  280): verify: /data/data/xper.jv/files/GoodApp_added_modified_manifest.apk verified

    ...

Back to square one.

2.4 But What About The Digest Manifest ?

Given that the signature file contains a digest for the manifest and that has not been changed why is it possible to modify the manifest ?

The answer is because the digest manifest is considered to be an ‘optimization’ not a way of preventing the manifest from being changed.

If the computed digest of the manifest matches the one in the signature file then it is known to be unchanged and there is no need to check the individual digest entries in the manifest.

This is the ‘optimization’

If the computed digest of the manifest does not match then the individual digests in the signature file are used to check the corresponding digest entries in the manifest.

3.0 It Will Have To Be The Certificates Then

3.1 The JarEntry getCertificates Method

The Javadoc for the JerEntry getCertificates method for the implementation we are using says

Returns an array of Certificate Objects associated with this entry or null if none exists. Make sure that the everything is read from the input stream before calling this method, or else the method returns null.

The Javadoc for a.n.other implementation says

Returns the Certificate objects for this entry, or null if none. This method can only be called once the JarEntry has been completely verified by reading from the entry input stream until the end of the stream has been reached. Otherwise, this method will return null.

The returned certificate array comprises all the signer certificates that were used to verify this entry. Each signer certificate is followed by its supporting certificate chain (which may be empty). Each signer certificate and its supporting certificate chain are ordered bottom-to-top (i.e., with the signer certificate first and the (root) certificate authority last).

which is a bit more useful.

If the getCertificates method returns an array of Certificates we know that there is a digest entry in the manifest and a digest of the digest entry in a signature file which has been ‘verified’

3.2 JARVerifierV3

    final class JARVerifierV3 
                implements
                    JARVerifier
    {
        public boolean verify(File theFile)
        {
            Log.d(TAG, "verify: verifying " + theFile);
		    try
            {
                JarFile jf = new JarFile(theFile, true);
			
                try
                {
                    verify(jf);
                    Log.d(TAG, "verify: " + theFile + " verified");
                    return true;
                }
                finally
                {
                    jf.close();
                }
            }
            catch (Exception e)
            {
                Log.d(TAG, "verify: verification failed");
                e.printStackTrace();
                return false;
            }
        }
	
        //
	
        JARVerifierV3()
        {
            buffer = new byte[8192];
        }
	
        //

        private void verify(JarFile theJARFile) 
                     throws
                         Exception
        {
            Enumeration<JarEntry> entries = theJARFile.entries();
		
            while (entries.hasMoreElements())
            {
                verify(theJARFile, entries.nextElement());
            }
		
        }

        private void verify(JarFile theJARFile, JarEntry theEntry) 
                     throws
                         Exception
        {
            Log.d(TAG, "verify: " + theEntry.getName());
		
		
		
            InputStream is = theJARFile.getInputStream(theEntry);
		
            while (is.read(buffer) != -1)
            {
			
            }
		
            ensureCertificates(theEntry);
        }
	
        private void ensureCertificates(JarEntry theEntry)
                     throws
                         Exception
        {
            if (!theEntry.getName().startsWith(META_INF))
            {
                Log.d(TAG, "ensureCertificates: " + theEntry.getName());
		
                Certificate[] certificates = theEntry.getCertificates();
			
                if (certificates == null)
                {
                    throw new SecurityException("No certificates: " + theEntry.getName());
                }
            }
        }
	
        //
	
        private byte[] buffer;
	
        //
		
        private static final String META_INF      = "META-INF/";
	
        //
	
        private static final String TAG = "JARVerifierV3";
    }

3.3 Verification After File Addition

If we attempt to verify our APK with added file and modified manifest using JARVerifierV3 we return to the status quo ante.

    ...

    D/JARVerifierV3(  280): verify: verifying /data/data/xper.jv/files/GoodApp_added_modified_manifest.apk
    D/JARVerifierV3(  280): verify: res/layout/activity_main.xml
    D/JARVerifierV3(  280): ensureCertificates: res/layout/activity_main.xml
    D/JARVerifierV3(  280): verify: res/menu/activity_main.xml
    D/JARVerifierV3(  280): ensureCertificates: res/menu/activity_main.xml
    D/JARVerifierV3(  280): verify: AndroidManifest.xml
    D/JARVerifierV3(  280): ensureCertificates: AndroidManifest.xml
    D/JARVerifierV3(  280): verify: resources.arsc
    D/JARVerifierV3(  280): ensureCertificates: resources.arsc
    D/JARVerifierV3(  280): verify: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV3(  280): verify: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV3(  280): verify: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV3(  280): verify: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV3(  280): verify: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV3(  280): verify: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV3(  280): verify: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV3(  280): ensureCertificates: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV3(  280): verify: classes.dex
    D/JARVerifierV3(  280): ensureCertificates: classes.dex
    D/JARVerifierV3(  280): verify: META-INF/MANIFEST.MF
    D/JARVerifierV3(  280): verify: META-INF/CERT.SF
    D/JARVerifierV3(  280): verify: META-INF/CERT.RSA
    D/JARVerifierV3(  280): verify: assets/new_asset
    D/JARVerifierV3(  280): ensureCertificates: assets/new_asset
    D/JARVerifierV3(  280): verify: verification failed
    W/System.err(  280): java.lang.SecurityException: No certificates: assets/new_asset
    W/System.err(  280): 	at xper.jv.JARVerifierV3.ensureCertificates(JARVerifierV3.java:95)
    W/System.err(  280): 	at xper.jv.JARVerifierV3.verify(JARVerifierV3.java:80)
    W/System.err(  280): 	at xper.jv.JARVerifierV3.verify(JARVerifierV3.java:60)
    W/System.err(  280): 	at xper.jv.JARVerifierV3.verify(JARVerifierV3.java:26)
    W/System.err(  280): 	at xper.jv.XperJVActivity.verify(XperJVActivity.java:128)
    W/System.err(  280): 	at xper.jv.XperJVActivity.access$0(XperJVActivity.java:118)
    W/System.err(  280): 	at xper.jv.XperJVActivity$1.run(XperJVActivity.java:112)
    W/System.err(  280): 	at java.lang.Thread.run(Thread.java:1096)

    ...

3.4 Signing Additional Manifest Entries

Needless to say there is a catch. Not only is it possible to add digest entries to the manifest of a signed JAR it is possible to sign them as well.

We can create a second signature file BERT.SF which contains the digest of the added manifest entry

    Signature-Version: 1.0

    Name: assets/new_asset
    SHA1-Digest: 6G+NGDseZOsVO904ZG1VLkFAr30=

and a signed signature file BERT.DSA and then create a new APK

    Archive:  GoodApp_added_modified_manifest_ds.apk
      Length     Date   Time    Name
     --------    ----   ----    ----
          624  07-10-13 21:11   res/layout/activity_main.xml
          464  07-10-13 21:11   res/menu/activity_main.xml
         1588  07-10-13 21:11   AndroidManifest.xml
         2504  07-10-13 21:11   resources.arsc
          409  07-10-13 21:11   res/drawable-hdpi/ic_action_search.png
         4129  07-10-13 21:11   res/drawable-hdpi/ic_launcher.png
         1756  07-10-13 21:11   res/drawable-ldpi/ic_launcher.png
          311  07-10-13 21:11   res/drawable-mdpi/ic_action_search.png
         2654  07-10-13 21:11   res/drawable-mdpi/ic_launcher.png
          491  07-10-13 21:11   res/drawable-xhdpi/ic_action_search.png
         5456  07-10-13 21:11   res/drawable-xhdpi/ic_launcher.png
       366112  07-10-13 21:11   classes.dex
         1103  07-26-13 14:51   META-INF/MANIFEST.MF
         1087  07-10-13 21:11   META-INF/CERT.SF
          776  07-10-13 21:11   META-INF/CERT.RSA
           48  07-24-13 21:56   assets/new_asset
         2365  07-26-13 22:10   META-INF/BERT.DSA
           95  07-26-13 22:10   META-INF/BERT.SF
     --------                   -------
       391972                   18 files

Now our new file assets/new_asset has certificates associated with it and verification succeeds one more

    ...

    D/JARVerifierV3(  281): verify: verifying /data/data/xper.jv/files/GoodApp_added_modified_manifest_ds.apk
    D/JARVerifierV3(  281): verify: res/layout/activity_main.xml
    D/dalvikvm(  281): GC_FOR_MALLOC freed 4701 objects / 294968 bytes in 59ms
    D/JARVerifierV3(  281): ensureCertificates: res/layout/activity_main.xml
    D/JARVerifierV3(  281): verify: res/menu/activity_main.xml
    D/JARVerifierV3(  281): ensureCertificates: res/menu/activity_main.xml
    D/JARVerifierV3(  281): verify: AndroidManifest.xml
    D/JARVerifierV3(  281): ensureCertificates: AndroidManifest.xml
    D/JARVerifierV3(  281): verify: resources.arsc
    D/JARVerifierV3(  281): ensureCertificates: resources.arsc
    D/JARVerifierV3(  281): verify: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-hdpi/ic_action_search.png
    D/JARVerifierV3(  281): verify: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-hdpi/ic_launcher.png
    D/JARVerifierV3(  281): verify: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-ldpi/ic_launcher.png
    D/JARVerifierV3(  281): verify: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-mdpi/ic_action_search.png
    D/JARVerifierV3(  281): verify: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-mdpi/ic_launcher.png
    D/JARVerifierV3(  281): verify: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-xhdpi/ic_action_search.png
    D/JARVerifierV3(  281): verify: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV3(  281): ensureCertificates: res/drawable-xhdpi/ic_launcher.png
    D/JARVerifierV3(  281): verify: classes.dex
    D/JARVerifierV3(  281): ensureCertificates: classes.dex
    D/JARVerifierV3(  281): verify: META-INF/MANIFEST.MF
    D/JARVerifierV3(  281): verify: META-INF/CERT.SF
    D/JARVerifierV3(  281): verify: META-INF/CERT.RSA
    D/JARVerifierV3(  281): verify: assets/new_asset
    D/JARVerifierV3(  281): ensureCertificates: assets/new_asset
    D/JARVerifierV3(  281): verify: META-INF/BERT.DSA
    D/JARVerifierV3(  281): verify: META-INF/BERT.SF
    D/JARVerifierV3(  281): verify: /data/data/xper.jv/files/GoodApp_added_modified_manifest_ds.apk verified

    ...

Back to square one again

4.0 PackageParser collectCertificates Again

This is why the PackageParser collectCertificates method is doing this

                ...

                Enumeration<JarEntry> entries = jarFile.entries();
                final Manifest manifest = jarFile.getManifest();
                while (entries.hasMoreElements()) {
                    final JarEntry je = entries.nextElement();
                    if (je.isDirectory()) continue;

                    final String name = je.getName();

                    if (name.startsWith("META-INF/"))
                        continue;

                    if (ANDROID_MANIFEST_FILENAME.equals(name)) {
                        final Attributes attributes = manifest.getAttributes(name);
                        pkg.manifestDigest = ManifestDigest.fromAttributes(attributes);
                    }

                    final Certificate[] localCerts = loadCertificates(jarFile, je, readBuffer);
                    if (DEBUG_JAR) {
                        Slog.i(TAG, "File " + mArchiveSourcePath + " entry " + je.getName()
                                + ": certs=" + certs + " ("
                                + (certs != null ? certs.length : 0) + ")");
                    }

                    if (localCerts == null) {
                        Slog.e(TAG, "Package " + pkg.packageName
                                + " has no certificates at entry "
                                + je.getName() + "; ignoring!");
                        jarFile.close();
                        mParseError = PackageManager.INSTALL_PARSE_FAILED_NO_CERTIFICATES;
                        return false;
                    } else if (certs == null) {
                        certs = localCerts;
                    } else {
                        // Ensure all certificates match.
                        for (int i=0; i<certs.length; i++) {
                            boolean found = false;
                            for (int j=0; j<localCerts.length; j++) {
                                if (certs[i] != null &&
                                    certs[i].equals(localCerts[j])) {
                                    found = true;
                                    break;
                                }
                            }
                            if (!found || certs.length != localCerts.length) {
                                Slog.e(TAG, "Package " + pkg.packageName
                                        + " has mismatched certificates at entry "
                                        + je.getName() + "; ignoring!");
                                jarFile.close();
                                mParseError = PackageManager.INSTALL_PARSE_FAILED_INCONSISTENT_CERTIFICATES;
                                return false;
                            }
                        }
                    }
                }
                
                ...

not only checking for the existence of certificates associated with each file

                    ...

                    if (localCerts == null) {
                        Slog.e(TAG, "Package " + pkg.packageName
                                + " has no certificates at entry "
                                + je.getName() + "; ignoring!");
                        jarFile.close();
                        mParseError = PackageManager.INSTALL_PARSE_FAILED_NO_CERTIFICATES;
                        return false;
                    } 
                    
                    ...

but also checking, albeit in a somewhat convoluted manner, that all the files have been signed with the same set of certificates .

                        ...

                        // Ensure all certificates match.
                        for (int i=0; i<certs.length; i++) {
                            boolean found = false;
                            for (int j=0; j<localCerts.length; j++) {
                                if (certs[i] != null &&
                                    certs[i].equals(localCerts[j])) {
                                    found = true;
                                    break;
                                }
                            }
                            if (!found || certs.length != localCerts.length) {
                                Slog.e(TAG, "Package " + pkg.packageName
                                        + " has mismatched certificates at entry "
                                        + je.getName() + "; ignoring!");
                                jarFile.close();
                                mParseError = PackageManager.INSTALL_PARSE_FAILED_INCONSISTENT_CERTIFICATES;
                                return false;
                            }
                        }

                        ...

This is what verification of a signed JAR amounts to.

Establishing in the most complicated way possible whether all of the files present in the JAR have been signed with the ‘same’set of certificates.

Note that this does not address the question of whether files have been removed.

Note also that this relies on establishing the ‘equality’ of certficates which is an ‘interesting’ problem in its own right.


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

July 15, 2013

The Great Android Security Hole Of ’08 ? – Part Six: The Gory Details — The APK Verification Considered Ineffective Edition

As we have seen, verification of the files within an APK occurs during the execution of the PackageParser method collectCertificates.

This method constructs a JarFile instance using the JarFile(File) constructor.

It then calls the JarFile method entries to obtain an Enumeration which it uses to iterate over all the files in the APK.

It is the entries method that is the issue, or rather, what it returns.

1.0 The JarFile entries Method

Class: java.util.jar.JarFile

Source: $(ANDROID_SRC)/libcore/luni/src/main/java/java/util/jar/JarFile.java

    @Override
    public Enumeration entries() {
        class JarFileEnumerator implements Enumeration<JarEntry> {
            Enumeration<? extends ZipEntry> ze;

            JarFile jf;

            JarFileEnumerator(Enumeration<? extends ZipEntry> zenum, JarFile jf) {
                ze = zenum;
                this.jf = jf;
            }

            public boolean hasMoreElements() {
                return ze.hasMoreElements();
            }

            public JarEntry nextElement() {
                JarEntry je = new JarEntry(ze.nextElement());
                je.parentJar = jf;
                return je;
            }
        }
        return new JarFileEnumerator(super.entries(), this);
    }

It turns out that this method is effectively just wrapping the entries method of the super class java.util.zip.ZipFile.

2.0 The ZipFile entries Method

Class: java.util.zip.ZipFile

Source: $(ANDROID_SRC)/libcore/luni/src/main/java/java/util/zip/ZipFile.java

    public Enumeration<? extends ZipEntry> entries() {
        checkNotClosed();
        final Iterator<ZipEntry> iterator = entries.values().iterator();

        return new Enumeration<ZipEntry>() {
            public boolean hasMoreElements() {
                checkNotClosed();
                return iterator.hasNext();
            }

            public ZipEntry nextElement() {
                checkNotClosed();
                return iterator.next();
            }
        };

This method is in turn wrapping the values held in the entries instance variable.

3.0 The ZipFile entries Instance Variable

The entries instance variable is declared like this

    private final LinkedHashMap<String, ZipEntry> entries = new LinkedHashMap<String, ZipEntry>();

It is populated by the readCentralDirectory method which is invoked by the ZipFile(File,int) constructor.

4.0 ZipFile readCentralDirectory

The readCentralDirectory method starts, as you might expect, by scanning backwards through the file looking for the End of Central Directory record.

        ...

        // Scan back, looking for the End Of Central Directory field. If the zip file doesn't
        // have an overall comment (unrelated to any per-entry comments), we'll hit the EOCD
        // on the first try.
        // No need to synchronize raf here -- we only do this when we first open the zip file.
        long scanOffset = raf.length() - ENDHDR;
        if (scanOffset < 0) {
            throw new ZipException("File too short to be a zip file: " + raf.length());
        }

        long stopOffset = scanOffset - 65536;
        if (stopOffset < 0) {
            stopOffset = 0;
        }

        final int ENDHEADERMAGIC = 0x06054b50;
        while (true) {
            raf.seek(scanOffset);
            if (Integer.reverseBytes(raf.readInt()) == ENDHEADERMAGIC) {
                break;
            }

            scanOffset--;
            if (scanOffset < stopOffset) {
                throw new ZipException("EOCD not found; not a zip file?");
            }
        }
        
        ...

Once it has found it, it reads the fields it is interested in and does a basic sanity check.

        ...

        // Read the End Of Central Directory. We could use ENDHDR instead of the magic number 18,
        // but we don't actually need all the header.
        byte[] eocd = new byte[18];
        raf.readFully(eocd);

        // Pull out the information we need.
        BufferIterator it = HeapBufferIterator.iterator(eocd, 0, eocd.length, ByteOrder.LITTLE_ENDIAN);
        int diskNumber = it.readShort() & 0xffff;
        int diskWithCentralDir = it.readShort() & 0xffff;
        int numEntries = it.readShort() & 0xffff;
        int totalNumEntries = it.readShort() & 0xffff;
        it.skip(4); // Ignore centralDirSize.
        long centralDirOffset = ((long) it.readInt()) & 0xffffffffL;

        if (numEntries != totalNumEntries || diskNumber != 0 || diskWithCentralDir != 0) {
            throw new ZipException("spanned archives not supported");
        }
        
        ...

It then seeks to the start of the Central Directory and reads the File Headers and adds them to the entries LinkedHashMap.

        ...

        RAFStream rafStream = new RAFStream(raf, centralDirOffset);
        BufferedInputStream bufferedStream = new BufferedInputStream(rafStream, 4096);
        byte[] hdrBuf = new byte[CENHDR]; // Reuse the same buffer for each entry.
        for (int i = 0; i < numEntries; ++i) {
            ZipEntry newEntry = new ZipEntry(hdrBuf, bufferedStream);
            entries.put(newEntry.getName(), newEntry);
        }
        
        ...

It is the line in red that is the problem.

No check is made to see whether the ZIP file contains multiple files with the same name.

As a result, in the unlikely event of a ZIP file containing N files of the same name, then the JarFile entries method will only return the N'th. The preceding N-1 files will silently disappear.

5.0 Implications For The Verification Of An APK

If an APK just happens to have two files of the same name in it, then only the second one which gets verified, which is a bit unfortunate really if those files are called classes.dex, as we shall see.


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog's author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

July 13, 2013

The Great Android Security Hole Of ’08 ? – Part Two: Signed JARs

1.0 Structure

An Android APK is a signed JAR and a signed JAR is a good old-fashioned ZIP file.

If you list the contents of an APK using unzip you will see something like this

    Archive:  GoodAppBadApp.apk
      Length     Date   Time    Name
     --------    ----   ----    ----
          624  07-10-13 21:11   res/layout/activity_main.xml
          464  07-10-13 21:11   res/menu/activity_main.xml
         1588  07-10-13 21:11   AndroidManifest.xml
         2504  07-10-13 21:11   resources.arsc
          409  07-10-13 21:11   res/drawable-hdpi/ic_action_search.png
         4129  07-10-13 21:11   res/drawable-hdpi/ic_launcher.png
         1756  07-10-13 21:11   res/drawable-ldpi/ic_launcher.png
          311  07-10-13 21:11   res/drawable-mdpi/ic_action_search.png
         2654  07-10-13 21:11   res/drawable-mdpi/ic_launcher.png
          491  07-10-13 21:11   res/drawable-xhdpi/ic_action_search.png
         5456  07-10-13 21:11   res/drawable-xhdpi/ic_launcher.png
       366112  07-10-13 21:11   classes.dex
         1034  07-10-13 21:11   META-INF/MANIFEST.MF
         1087  07-10-13 21:11   META-INF/CERT.SF
          776  07-10-13 21:11   META-INF/CERT.RSA
     --------                   -------
       389395                   15 files

It is the presence of the manifest file

    META-INF/MANIFEST.MF

which make this ZIP file a JAR, and it is the presence of the ‘signature’ files

    META-INF/CERT.SF
    META-INF/CERT.RSA

that makes it a signed JAR.

2.0 The Meta Files

2.1 MANIFEST.MF

This is what’s in the MANIFEST.MF file contained in the APK shown in the example above

    Manifest-Version: 1.0
    Created-By: 1.0 (Android)

    Name: res/drawable-xhdpi/ic_launcher.png
    SHA1-Digest: TJE1tg63y88xLdIALKGpuLmgh0s=

    Name: res/drawable-mdpi/ic_action_search.png
    SHA1-Digest: XjltJdEB3tvakTn9CN7KwdaNT68=

    Name: AndroidManifest.xml
    SHA1-Digest: iXwTXVvLX3gY+ByGhybEJvIxP08=

    Name: res/drawable-mdpi/ic_launcher.png
    SHA1-Digest: aM73uFWfPOOvq5Kk9Ffd3cWm0OQ=

    Name: res/drawable-hdpi/ic_launcher.png
    SHA1-Digest: ioH2V9g4FYKkqpDHk7jPryMKtcE=

    Name: res/layout/activity_main.xml
    SHA1-Digest: zd1eNopMfZ7CmMk8GjyPGPpYloc=

    Name: resources.arsc
    SHA1-Digest: 7nYq90PW0OZVg8ytyjBEcwQhYiA=

    Name: classes.dex
    SHA1-Digest: b80KFg5gdqlORweY5LrUS87tbVU=

    Name: res/drawable-hdpi/ic_action_search.png
    SHA1-Digest: rOWnNxWCWKwkicf+FpYGMg1fX2Y=

    Name: res/drawable-xhdpi/ic_action_search.png
    SHA1-Digest: WzSVDDIHZpn0cFeuxgKRJete4TU=

    Name: res/menu/activity_main.xml
    SHA1-Digest: TfxVPQP5IRVlJun7A7PdOqdkp+I=

    Name: res/drawable-ldpi/ic_launcher.png
    SHA1-Digest: r6Mdl54h2qEvVnqgfsgxU4CysiI=

As you can see there is a section for each file in the APK with the exception of those contained in the META-INF directory.

The SHA1-Digest attribute specifies the SHA-1[1] digest of the named file. The value is Base64 encoded.

2.2 CERT.SF

The CERT.SF file is a ‘signature file’. This is odd because technically it does not contain any signatures in the cryptographic sense, but there it is.

Signature files are identified by the suffix .SF

This is what’s in the CERT.SF file contained in the APK shown in the example above

    Signature-Version: 1.0
    Created-By: 1.0 (Android)
    SHA1-Digest-Manifest: uy1/gV4Q91cFNXDPKFWnIkdp7Hs=

    Name: res/drawable-xhdpi/ic_launcher.png
    SHA1-Digest: uQhdAiwpzle4OMU4KiZ8rHZZW0s=

    Name: res/drawable-mdpi/ic_action_search.png
    SHA1-Digest: jAGFiT7RUDOYcmBQqk2FP01Xtb0=

    Name: AndroidManifest.xml
    SHA1-Digest: app7CkR5QClO2DngK2SwqyoYT/4=

    Name: res/drawable-mdpi/ic_launcher.png
    SHA1-Digest: 7AYLMxFKX2g7vudLLZTXlC9DWHI=

    Name: res/drawable-hdpi/ic_launcher.png
    SHA1-Digest: diYsvf3zRlU+JNDApc7+KLVhhk0=

    Name: res/layout/activity_main.xml
    SHA1-Digest: tK4fMdQKbbzD/KoE5awGKM7o++Y=

    Name: resources.arsc
    SHA1-Digest: mMTrtN8w3SByhMHUQ19nJ3OT3wo=

    Name: res/drawable-hdpi/ic_action_search.png
    SHA1-Digest: WvsoZfcfGbz7vtTCwzXLh3VcxFw=

    Name: classes.dex
    SHA1-Digest: /okgNUxVNi1erXMkC7ftetu9S9U=

    Name: res/drawable-xhdpi/ic_action_search.png
    SHA1-Digest: 772ITFXOlprwo44bOLvLMa3Vdeg=

    Name: res/menu/activity_main.xml
    SHA1-Digest: K1PTOAgvzmJK47MHIsJIWgVr/04=

    Name: res/drawable-ldpi/ic_launcher.png
    SHA1-Digest: zC+Z/B5ib1ntZmz9D80Rj3muYqE=

The SHA1-Digest-Manifest in the first section specifies the SHA-1 digest of the manifest file.

In the following sections the SHA1-Digest attribute specifies the SHA-1 digest of the corresponding section in the manifest file.

2.3 CERT.RSA

The CERT.RSA file is the ‘signed signature file’.

The CERT.RSA file contained in the APK shown in the example above is a binary file, so this is a representation of its contents produced using the ubiquitous dumpasn1.


   0  772: SEQUENCE {
   4    9:   OBJECT IDENTIFIER signedData (1 2 840 113549 1 7 2)
  15  757:   [0] {
  19  753:     SEQUENCE {
  23    1:       INTEGER 1
  26   11:       SET {
  28    9:         SEQUENCE {
  30    5:           OBJECT IDENTIFIER sha1 (1 3 14 3 2 26)
  37    0:           NULL
         :           }
         :         }
  39   11:       SEQUENCE {
  41    9:         OBJECT IDENTIFIER data (1 2 840 113549 1 7 1)
         :         }
  52  489:       [0] {
  56  485:         SEQUENCE {
  60  334:           SEQUENCE {
  64    3:             [0] {
  66    1:               INTEGER 2
         :               }
  69    4:             INTEGER 1322302491
  75   13:             SEQUENCE {
  77    9:               OBJECT IDENTIFIER
         :                 sha1withRSAEncryption (1 2 840 113549 1 1 5)
  88    0:               NULL
         :               }
  90   55:             SEQUENCE {
  92   11:               SET {
  94    9:                 SEQUENCE {
  96    3:                   OBJECT IDENTIFIER countryName (2 5 4 6)
 101    2:                   PrintableString 'US'
         :                   }
         :                 }
 105   16:               SET {
 107   14:                 SEQUENCE {
 109    3:                   OBJECT IDENTIFIER organizationName (2 5 4 10)
 114    7:                   PrintableString 'Android'
         :                   }
         :                 }
 123   22:               SET {
 125   20:                 SEQUENCE {
 127    3:                   OBJECT IDENTIFIER commonName (2 5 4 3)
 132   13:                   PrintableString 'Android Debug'
         :                   }
         :                 }
         :               }
 147   30:             SEQUENCE {
 149   13:               UTCTime 26/11/2011 10:14:51 GMT
 164   13:               UTCTime 18/11/2041 10:14:51 GMT
         :               }
 179   55:             SEQUENCE {
 181   11:               SET {
 183    9:                 SEQUENCE {
 185    3:                   OBJECT IDENTIFIER countryName (2 5 4 6)
 190    2:                   PrintableString 'US'
         :                   }
         :                 }
 194   16:               SET {
 196   14:                 SEQUENCE {
 198    3:                   OBJECT IDENTIFIER organizationName (2 5 4 10)
 203    7:                   PrintableString 'Android'
         :                   }
         :                 }
 212   22:               SET {
 214   20:                 SEQUENCE {
 216    3:                   OBJECT IDENTIFIER commonName (2 5 4 3)
 221   13:                   PrintableString 'Android Debug'
         :                   }
         :                 }
         :               }
 236  159:             SEQUENCE {
 239   13:               SEQUENCE {
 241    9:                 OBJECT IDENTIFIER
         :                   rsaEncryption (1 2 840 113549 1 1 1)
 252    0:                 NULL
         :                 }
 254  141:               BIT STRING, encapsulates {
 258  137:                 SEQUENCE {
 261  129:                   INTEGER
         :                   00 8A 0A 1D 53 48 FC 26 6D BF 65 57 04 8B 0B E9
         :                   58 EE E7 11 CE A8 A2 01 72 7B 34 C1 31 60 B4 64
         :                   FC D3 0D 1E 94 BE B3 5B 55 7F 23 97 B9 3D 6F E8
         :                   17 17 FE CA 50 EB F3 EA 4B 7C F6 F9 B6 6E E7 5D
         :                   0D 64 79 00 8F 12 1A 76 09 C1 32 4F 28 E8 27 94
         :                   34 F5 C8 50 C3 0D E9 2E FF A2 97 71 6C 18 7F 7C
         :                   8A 78 EA 37 50 20 8D DA 6C F2 3F 65 85 67 6A 7B
         :                   35 BF 5A 8D 53 DE AD 9D 50 70 30 B4 0D A4 60 D4
         :                           [ Another 1 bytes skipped ]
 393    3:                   INTEGER 65537
         :                   }
         :                 }
         :               }
         :             }
 398   13:           SEQUENCE {
 400    9:             OBJECT IDENTIFIER
         :               sha1withRSAEncryption (1 2 840 113549 1 1 5)
 411    0:             NULL
         :             }
 413  129:           BIT STRING
         :             2E 77 85 3F F4 AF EE 1E 81 9B 8B BB E2 9C AA 95
         :             BE 38 1E 20 EA E3 63 BE 2B A7 F2 AB 8A F4 F5 27
         :             35 F6 7C 67 CF E5 07 82 D0 B5 D8 00 77 CC 9A 8B
         :             86 DD 4C DD 41 88 2E C5 B4 B9 7E E9 C4 B2 B9 FA
         :             31 A0 60 6F DA 8D D1 F9 FA F6 94 08 92 C2 58 D6
         :             BF 93 1E 4D B8 DF DD D3 C1 11 89 14 22 C8 8B 85
         :             98 04 7A F5 00 D5 1D F2 E0 54 42 82 46 2D FE 2D
         :             C3 AB 2B C6 BC E7 07 A2 B8 CD 04 26 01 F1 EB E5
         :           }
         :         }
 545  228:       SET {
 548  225:         SEQUENCE {
 551    1:           INTEGER 1
 554   63:           SEQUENCE {
 556   55:             SEQUENCE {
 558   11:               SET {
 560    9:                 SEQUENCE {
 562    3:                   OBJECT IDENTIFIER countryName (2 5 4 6)
 567    2:                   PrintableString 'US'
         :                   }
         :                 }
 571   16:               SET {
 573   14:                 SEQUENCE {
 575    3:                   OBJECT IDENTIFIER organizationName (2 5 4 10)
 580    7:                   PrintableString 'Android'
         :                   }
         :                 }
 589   22:               SET {
 591   20:                 SEQUENCE {
 593    3:                   OBJECT IDENTIFIER commonName (2 5 4 3)
 598   13:                   PrintableString 'Android Debug'
         :                   }
         :                 }
         :               }
 613    4:             INTEGER 1322302491
         :             }
 619    9:           SEQUENCE {
 621    5:             OBJECT IDENTIFIER sha1 (1 3 14 3 2 26)
 628    0:             NULL
         :             }
 630   13:           SEQUENCE {
 632    9:             OBJECT IDENTIFIER rsaEncryption (1 2 840 113549 1 1 1)
 643    0:             NULL
         :             }
 645  128:           OCTET STRING
         :             22 52 11 23 F5 AA 38 2E 28 FD 96 1E 2C AB 96 FC
         :             E7 85 95 54 A0 F7 14 49 93 F4 19 A7 F0 E5 CC B9
         :             FF 82 B6 8D 84 A2 3A DD F6 CA 6A 4F 3A 38 B4 EA
         :             69 23 BF B1 29 5F C2 49 48 34 63 35 84 59 C6 69
         :             0C E3 F1 D3 90 B5 02 F5 0D 79 CC B4 B0 25 0C BD
         :             38 4F B2 FC D1 D7 D4 11 84 EA A1 66 31 F0 08 DA
         :             0F F6 94 23 96 5C 83 96 C9 DA CA C0 FC 60 84 C2
         :             54 21 DA CA E4 C4 D0 7D FD 98 C4 91 8A 2C 5D E4
         :           }
         :         }
         :       }
         :     }
         :   }

If you are well-versed in the arcana of ASN.1 and the myriad security standards you will recognize this as ‘signed data’ as specified by PKCS#7. If you aren’t, you won’t, but that’s what it is nonetheless.

It is the SHA-1 digest of the CERT.SF file which has been encrypted using the private key corresponding to the public key contained in the accompanying certificate.

There is a bit more to it than that, but that is basically it.

3.0 Signed JAR Verification In Theory

To make life more exciting there can in fact be multiple signature files and signed signature files in a signed JAR but in what follows I am going to assume we are attempting to verify the APK shown in the example above in which there is just the one signature/signed signature file pair.

3.1 Decide Whether You ‘Trust’ The Signer Of The JAR

The signer of the JAR is the entity identified by the certificate which accompanies the signed data in the signed signature file (CERT.RSA).

The decision to ‘trust’ that entity is often made by going up the chain of certificates looking for one which is defined to be a ‘trust root’.

If the certificate that accompanies the signed data in the signed signature file is self-signed or was not signed directly or indirectly by a certificate defined to be a ‘trust root’ then the JAR should be rejected.

3.2 Verify The Signature File

Compute the digest of the signature file (CERT.SF) and use the signature in the signed signature file (CERT.SF) to verify it. If verification fails then reject the JAR.

3.3 Verify The Manifest File

If the expected digest of the manifest file is specified in the signature file compute the digest of the manifest file. If the expected and computed digests do not match then reject the JAR.

Alternatively, for each file in the JAR other than those in the META-INF directory find the corresponding section in the manifest.

If there is no corresponding section reject the JAR.

Otherwise find the expected digest of the section in the signature file.

If there is no expected digest fo the section in the signature file reject the JAR.

Otherwise compute the hash of the section in the manifest. If the computed and expected digests do not match reject the JAR.

3.4 Verify The Remaining Files In The JAR

For each file in the JAR that has not already been verified check that its digest has been specified in the manifest file.

If it has not then reject the JAR.

If the expected digest is specified in the manifest file then compute the actual digest. If the expected and computed digests do not match then reject the JAR.

If all the files in the JAR are successfully verified then accept the JAR.

4.0 Signed JAR Verification In Practice

The Java class java.util.JarFile supports the piecemeal implicit verification of files in signed JARs

4.1 java.util.JarFile

Source: $(ANDROID_SRC)/libcore/luni/src/main/java/java/util/jar/JarFile.java

4.1.1 Constructing A JarFile

4.1.1.1 JarFile(File)

    public JarFile(File file) throws IOException {
        this(file, true);
    }

4.1.1.2 JarFile(File, boolean)

    public JarFile(File file, boolean verify) throws IOException

Passing this constructor a File argument which represents a signed JAR and a verify argument of true will result in a JarFile instance which supports the verification of individual files within the underlying signed JAR.

4.2 The Verification Of Individual Files Within A Signed JAR Using A JarFile Instance

Given a JarFile instance initialized to verify an underlying signed JAR as descibed above there is surprisingly no way to explicitly verify the entire contents of the signed JAR.

Nor is there any way to explicitly verify an individual file within the signed JAR.

The only way to verify an individual file is to read the entire contents of the file. Doing this will implicitly verify the file but only at the point when all the contents have been read.

Since verification of a file involves computing the digest of the contents it is obviously necessary to read the entire contents before being able to verify it.

What is extremely problematic is that there is nothing whatsoever in the API documentation that explains that this is what happens and more particularly that it is not safe to act on anything in the contents of a file in a signed JAR, despite the fact that the contents are being made available via the API, until the entire file has been read.

4.2.1 The ‘Verification’ API

4.2.1.1 getInputStream

     public InputStream getInputStream(ZipEntry ze) throws IOException

The getInputStream method returns an InputStream which can be used to read the contents of the file represented by the JarEntry argument.

If the JarFile instance was constructed with a signed JAR and it is being verified then the method returns a an instance of the inner class JarFileInputStream with an associated JarVerifier.VerifyEntry instance.

4.2.1.2 The JarFileInputStream read Methods

The read methods read from the underlying stream. They then write the bytes read to the associated verifier.

If the call results in all of the file data having been read then verify method of the associated verifier is called. This will throw a SecurityException if the verification fails.

5.0 Signed JAR Verification During The Installation Of An APK By The Android PackageManagerService

When the PackageManagerService finds an APK while scanning a directory it calls the method scanPackageLI.

This triggers the verification of the APK as a signed JAR as follows.

5.1 PackageManagerService

Source: $(ANDROID_SRC)/frameworks/base/services/java/com/android/server/pm/PackageManagerService.java

5.1.1 scanPackageLI

    private PackageParser.Package scanPackageLI(
                                      File       scanFile, 
                                      int        parseFlags, 
                                      int        scanMode, 
                                      long       currentTime, 
                                      UserHandle user)

After determining whether it already knows about the APK, in which case it may return, scanPackageLI calls the method collectCertificatesLI.

5.1.2 collectCertificatesLI

private boolean collectCertificatesLI(
                    PackageParser         pp, 
                    PackageSetting        ps,
                    PackageParser.Package pkg, 
                    File                  srcFile, 
                    int                   parseFlags)

The method calls collectCertificates on the PackageParser argument to do the actual work.

5.2 PackageParser

Class: android.content.pm.PackageParser

Source: $(ANDROID_SRC)/frameworks/base/core/java/android/content/pm/PackageParser.java

5.2.1 collectCertificates

    public boolean collectCertificates(Package pkg, int flags)

The method begins by creating a JarFile using the JarFile(File) constructor.

It then calls the method entries on the JarFile instance.

It then iterates over these entries. For each entry it calls the method loadCertificates.

5.2.2 loadCertificates

    private Certificate[] loadCertificates(JarFile jarFile, JarEntry je, byte[] readBuffer)

The method begins by calling getInputStream on the JarFile argument passing it the JarEntry argument.

It then calls read repeatedly on the InputStream, ignoring what it is actually read, until the entire contents of the file have been read.

As we have seen it is the call to the read method which results in the last of the data being returned which causes the verification of that file.

Notes

  1. The official SHA standard is here (This is a PDF document).


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

Older Posts »

Create a free website or blog at WordPress.com.

%d bloggers like this: