Just An Application

April 25, 2013

A Bad Attack Of Plagiarism

Filed under: Plagiarism — Tags: — Simon Lewis @ 10:07 am

Given the nature of the ‘interwebs’ it was bound to happen sooner or later, indeed it probably already has and I just don’t know about it.

Anyway somebody has decided to simply lift wholesale a particular post of mine, and re-post it as their own, but leaving in the links to the original diagrams and the other associated posts, but mysteriously omitting the copyright notice.

Leaving the links to the other posts means I got pingback comments posted to those posts on my original post which is the only reason I know about it at all.

I know that imitation is the sincerest form of flattery and all that but for obvious reasons I won’t be linking to the post in question.

I have left them a comment asking them to remove the post so we will have to see what happens next.


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

January 22, 2013

Android Internals: Resource Ids And Resource Lookup

1.0 Resource Id Structure

An Android Resource id is a 32-bit integer. It comprises

  • an 8-bit Package id [bits 24-31]
  • an 8-bit Type id [bits 16-23]
  • a 16-bit Entry index [bits 0-15]

The Package id identifies the Package chunk which contains the Resource.

The Type id identifies the type of the Resource and hence the corresponding Typespec chunk and
Type chunk or chunks which contain its value or value(s)

The Entry index identifies the individual Resource within the Typespec chunk and Type chunk(s).

1.1 Example

The four Resouce identifiers generated as part of the Resource Table example here break down as follows

Name Value Package id Type id Entry index
icon 0x7f020000 0x7f 2 0
main 0x7f030000 0x7f 3 0
hello 0x7f040000 0x7f 4 0
app_name 0x7f040001 0x7f 4 1

From this we can see that the icon resource is of type 2, the main resource is of type 3, and the hello and app_name resources are of type 4.

We know that the name of the Resource type with id i can be found at the index i - 1 in the typeStrings StringPool Chunk

Looking at the Package chunk of the Resource Table example here we can see that the names corresponding to the type ids

  • 2
  • 3
  • 4

are

  • drawable
  • layout
  • string

respectively as we would expect.

2.0 Resource Value Lookup

To find the value of a Resource in a Resource Table given its id.

  1. Extract the package id (p) the type id (t) and the entry index (e) from the given Resource id

  2. Find the Package chunk with id p. If there is no corresponding Package chunk then the Resource id is invalid.

  3. Find the Typespec chunk and associated Type chunk(s) with id t within the Package chunk. If there is no corresponding Typespec chunk then the Resource id is invalid.

  4. Check that the entry index e is a valid index into the entries in the Typespec chunk. If it is not then the Resource id is invalid.

At this point there are two possibilities, either there is a single Type chunk with the given id, or there are multiple Type chunks with the given id.

2.1 The Single Type Chunk Case

  1. Lookup the offset o of the entry for the Resource with the given id at index e within the offsets table of the Type chunk

  2. Go to the offset o past the start of the entries in the Type chunk as specified by the field entriesStart

  3. At this offset there will be either a simple or a complex resource entry which will specify the value of the Resource.

2.2 The Multiple Type Chunk Case

For each Type chunk

  1. Lookup the offset o of the entry for the Resource with the given id at index e within the offsets table of the Type chunk

  2. If the value of offset o is 0xFFFFFFFF then there is no value for the Resource in this Type chunk.

  3. Otherwise the value cane be found in the same way as for the single Type chunk case above.

If we are looking for a value for a specifiec configuration then for each Type chunk that contains a value for the Resource we need to record the configuration associated with the value as stored in the config field of the Type chunk.

Once we have collected all the available values for the Resource we then look for the closest match for the configuration we are interested in.

Exactly how you match configurations is left as an exercise for the reader.

2.3 Examples

2.3.1 Single Type Chunk Example

Given the id of the hello Resource, in the Resource table example here, which is 0x7f040000, then

  • p is 127
  • t is 4
  • e is 0

Using the algorithm above we end up with the Typespec chunk with id 4 and a single associated Type chunk which looks like this

    ...

    00000408 01 02       // type [TYPE]
    0000040a 34 00       // header size
    0000040c 5c 00 00 00 // chunk size
    --------------------

    00000410 04          // id
    00000411 00          // 0
    00000412 00 00       // 0
    00000414 02 00 00 00 // entryCount
    00000418 3c 00 00 00 // entriesStart (address 00000444)
    0000041c 20 00 00 00 // config: size
    00000420 00 00 00 00 // config: imsi
    00000424 00 00 00 00 // config: locale
    00000428 00 00 00 00 // config: screenType
    0000042c 00 00 00 00 // config: input
    00000430 00 00 00 00 // config: screenSize
    00000434 00 00 00 00 // config: version
    00000438 00 00 00 00 // config: screenConfig
    ++++++++++++++++++++

    0000043c 00 00 00 00 // entry_index[0]
    00000440 10 00 00 00 // entry_index[1]
    00000444 08 00       // entry[0] size
    00000446 00 00       // entry[0] flags
    00000448 02 00 00 00 // entry[0] key
    0000044c 08 00       // size
    0000044e 00          // 0
    0000044f 03          // dataType
    00000450 04 00 00 00 // data
    00000454 08 00       // entry[1] size
    00000456 00 00       // entry[1] flags
    00000458 03 00 00 00 // entry[1] key
    0000045c 08 00       // size
    0000045e 00          // 0
    0000045f 03          // dataType
    00000460 05 00 00 00 // data
    ==================== [End of TYPE]
    
    ...

The entry corresponding to the index 0 starts at 0x00000444.

It is a simple entry, the flag FLAG_COMPLEX is not set in the flags field, which means that it is immediately followed by an instance of struct Res_value. which specifies the value of the Resource.

The dataType field is 3 which identifies it as as string. The data field is 4, which, in the case of a value of type string, is the index of the string within the String pool chunk

Looking at the example the string at index 4 is

    "Hello World, PendragonActivity!"

as we would expect

2.3.2 Muliple Type Chunk Example

Given the id of the icon Resource () in the Resource table example here, which is 0x7f020000, then

  • p is 127
  • t is 2
  • e is 0

Using the algorithm above we end up with the Typespec chunk with id 2 and three associated Type chunks which look like this

    ...

    000002bc 01 02       // type [TYPE]
    000002be 34 00       // header size
    000002c0 48 00 00 00 // chunk size
    --------------------

    000002c4 02          // id
    000002c5 00          // 0
    000002c6 00 00       // 0
    000002c8 01 00 00 00 // entryCount
    000002cc 38 00 00 00 // entriesStart (address 000002f4)
    000002d0 20 00 00 00 // config: size
    000002d4 00 00 00 00 // config: imsi
    000002d8 00 00 00 00 // config: locale
    000002dc 00 00 78 00 // config: screenType
    000002e0 00 00 00 00 // config: input
    000002e4 00 00 00 00 // config: screenSize
    000002e8 04 00 00 00 // config: version
    000002ec 00 00 00 00 // config: screenConfig
    ++++++++++++++++++++

    000002f0 00 00 00 00 // entry_index[0]
    000002f4 08 00       // entry[0] size
    000002f6 00 00       // entry[0] flags
    000002f8 00 00 00 00 // entry[0] key
    000002fc 08 00       // size
    000002fe 00          // 0
    000002ff 03          // dataType
    00000300 00 00 00 00 // data
    ==================== [End of TYPE]

    00000304 01 02       // type [TYPE]
    00000306 34 00       // header size
    00000308 48 00 00 00 // chunk size
    --------------------

    0000030c 02          // id
    0000030d 00          // 0
    0000030e 00 00       // 0
    00000310 01 00 00 00 // entryCount
    00000314 38 00 00 00 // entriesStart (address 0000033c)
    00000318 20 00 00 00 // config: size
    0000031c 00 00 00 00 // config: imsi
    00000320 00 00 00 00 // config: locale
    00000324 00 00 a0 00 // config: screenType
    00000328 00 00 00 00 // config: input
    0000032c 00 00 00 00 // config: screenSize
    00000330 04 00 00 00 // config: version
    00000334 00 00 00 00 // config: screenConfig
    ++++++++++++++++++++

    00000338 00 00 00 00 // entry_index[0]
    0000033c 08 00       // entry[0] size
    0000033e 00 00       // entry[0] flags
    00000340 00 00 00 00 // entry[0] key
    00000344 08 00       // size
    00000346 00          // 0
    00000347 03          // dataType
    00000348 01 00 00 00 // data
    ==================== [End of TYPE]

    0000034c 01 02       // type [TYPE]
    0000034e 34 00       // header size
    00000350 48 00 00 00 // chunk size
    --------------------

    00000354 02          // id
    00000355 00          // 0
    00000356 00 00       // 0
    00000358 01 00 00 00 // entryCount
    0000035c 38 00 00 00 // entriesStart (address 00000384)
    00000360 20 00 00 00 // config: size
    00000364 00 00 00 00 // config: imsi
    00000368 00 00 00 00 // config: locale
    0000036c 00 00 f0 00 // config: screenType
    00000370 00 00 00 00 // config: input
    00000374 00 00 00 00 // config: screenSize
    00000378 04 00 00 00 // config: version
    0000037c 00 00 00 00 // config: screenConfig
    ++++++++++++++++++++

    00000380 00 00 00 00 // entry_index[0]
    00000384 08 00       // entry[0] size
    00000386 00 00       // entry[0] flags
    00000388 00 00 00 00 // entry[0] key
    0000038c 08 00       // size
    0000038e 00          // 0
    0000038f 03          // dataType
    00000390 02 00 00 00 // data
    ==================== [End of TYPE]
	
    ...

The configurations stored in the Type chunks differ only in the screenType field, the values of which are

  • 0x78
  • 0xa0
  • 0xf0

respectively.

These values correspond to the anonymous enum values

  • DENSITY_LOW

  • DENSITY_MEDIUM

  • DENSITY_HIGH

In each Type chunk there is a single entry at index 0.

The entries start at

  • 0x000002f4

  • 0x0000033c

  • 0x00000384

Each entry is a simple entry.

In each case the dataType field of the value following the entry is 3 indicating that the data is a string.

The values of the data fields of the values are

  • 0
  • 1
  • 2

These are indexes into the String pool chunk.

Looking at the example we can see that the corresponding strings are

  • “res/drawable-ldpi/icon.png”

  • “res/drawable-mdpi/icon.png”

  • “res/drawable-hdpi/icon.png”

which is what we would expect given the configurations specified by the Type chunks.


Copyright (c) 2013 By Simon Lewis. All Rights Reserved.

September 27, 2012

Building The Project Lambda JDK From Source On Mac OS X Snow Leopard

As I mentioned in my previous post I first tried to build the Project Lambda JDK from source on Snow Leopard using the Mac OS X OpenJDK port of JDK 7 and the build failed with a slew of compilation errors.

On this occasion I will spare you the gory details but it looks as though the Mac OS X OpenJDK port version of JDK 7 is out of date, so is there a more up to date version of JDK 7 that can be built from source on Snow Leopard ?

The answer is yes the OpenJDK JDK 7 Update project (JDK7u) version.

1.0 Building JDK7u From Source

1.1 Getting The Source

The JDK7u Mercurial repository can be cloned using the command

    hg clone http://hg.openjdk.java.net/jdk7u/jdk7u jdk7u

Once you have done this you need to clone the nested Mercurial repositories which comprise the JDK itself by changing to the directory containing the JDK7u repository and running the command

    sh ./get_source.sh

1.2 Building

To build the JDK7u version of the JDK you need to run the command

    make \
        LANG=C \
        ALT_BOOTDIR=/Library/Java/JavaVirtualMachines/1.6.0_31-b04-413.jdk/Contents/Home

in the root directory of the JDK7u repository

The ALT_BOOTDIR argument identifies the location of the home directory of the installed version of JDK 6 for Snow Leopard.

Note that although the documentation in README-builds.html requires that you install XCode 4.1 or higher fwiw I built it sucessfully with Xcode 3.2.4 installed.

The built JDK ends up in the directory

    build/macosx-x86_64/j2sdk-image

2.0 Building The Project Lambda JDK

You can build the Project Lambda JDK in exactly the same way as on Lion.

Change to the common/makefiles directory

    cd common/makefiles

Do the configuration

    ../autoconf/configure --with-boot-jdk=<elided>/jdk7u/build/macosx-x86_64/j2sdk-image

The --with-boot-jdk argument identifies the location of the built JDK7u JDK

Then do the build

    make images

As on Lion the resulting JDK will be in the directory

    build/macosx-x64-normal-server-release/images/j2sdk-image

Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

September 25, 2012

Building The Project Lambda JDK From Source On Mac OS X Lion

If you are interested in the work being done by the JSR 335 working group on defining Lambda Expressions for Java and you would like to get a feel for how they are going to work in practice you can either download a binary release of the Project Lambda version of the JDK which includes the reference implementation or, if you are feeling adventerous or possibly lucky, you can build it from source.

As you can tell from the title of this post I went with option two.

1.0 Prerequisites

1.1 Mac OS X Lion

JDK 7 is required to build the Project Lambda version of the JDK and the version of JDK 7 available for download will only install on Lion, and I assume Mountain Lion.

I started by trying to build on Snow Leopard using a locally built version of the Mac OS X OpenJDK port but ran into a few problems during the building of the javax.nio buffer code. The problems do not look as though they are insurmountable, but for the moment I am not able to build the Project Lambda JDK on Snow Leopard.

I have not done so but I would assume that it also possible to build the Project Lambda JDK on Mountain Lion.

1.2 JDK 7

The Project Lambda JDK needs to be built using JDK 7. You can get JDK 7 for Mac OS X Lion here.

1.3 Project Lambda JDK Source Code

You can clone the Project Lambda Mercurial repository using the command.

    hg clone http://hg.openjdk.java.net/lambda/lambda lambda

This should result in something like the following output

    ...

    requesting all changes
    adding changesets
    adding manifests
    adding file changes
    added 502 changesets with 536 changes to 85 files
    updating working directory
    82 files updated, 0 files merged, 0 files removed, 0 files unresolved

Once you have done this you need to clone the nested Mercurial repositories which comprise the JDK itself.

You can do this by changing to the directory into which you cloned the Project Lambda Mercurial repository, if you used the command above that will be the lambda directory, and then running the command.

 sh ./get_source.sh

Once you have got all the source you can do the build.

2.0 Building

There are currently two build systems for the JDK. The existing one and a new one being developed for use when building JDK 8.

Following advice given on the lambda-dev mailing list I went with the new one.

You need to start by changing to the directory common/makefiles

    cd common/makefiles

and then perform a configuration step

    sh ../autoconf/configure --with-boot-jdk=/Library/Java/JavaVirtualMachines/jdk1.7.0_07.jdk/Contents/Home

The --with-boot-jdk argument identifies the location of the JDK 7 binaries, libraries and JARs, etc.. JDK 7 is installed as a Mac OS X package and the given path points inside the package.

Running the command should produce a lot of output ending with something like the following which has been edited slightly for clarity

    ...

    ===================================================
    A new configuration has been successfully created in \
        elided/lambda/build/macosx-x64-normal-server-release
    using configure arguments '--with-boot-jdk=\
       /Library/Java/JavaVirtualMachines/jdk1.7.0_07.jdk/Contents/Home'.

    Configuration summary:
    * Debug level:    release
    * JDK variant:    normal
    * JVM variants:   server
    * OpenJDK target: OS: macosx, CPU architecture: x86, address length: 64
    * Boot JDK:       /Library/Java/JavaVirtualMachines/jdk1.7.0_07.jdk/Contents/Home

    Build performance summary:
    * Cores to use:   2
    * Memory limit:   4096 MB
    * ccache status:  not installed (consider installing)

    Build performance tip: ccache gives a tremendous speedup for C++ recompilations.
    You do not have ccache installed. Try installing it.

Once the configuration has completed you simply run the command

    make images

This should start by producing the output

    Building OpenJDK for target 'images' in configuration 'macosx-x64-normal-server-release'

and complete some time later with something like the following

    ...

    ##### Leaving jdk-images for target(s) images-only                 #####
    ########################################################################
    ##### Build time 00:00:48 jdk-images for target(s) images-only     #####
    ########################################################################

    -- Build times ----------
    Target 
    Start 2012-09-25 00:11:17
    End   2012-09-25 00:22:38
    00:00:28 corba
    00:02:45 hotspot
    00:00:26 jaxp
    00:00:31 jaxws
    00:05:43 jdk
    00:00:48 jdk-images
    00:00:40 langtools
    00:11:21 TOTAL
    -------------------------

If the build is successful then the resulting JDK can be found in the directory

    build/macosx-x64-normal-server-release/images/j2sdk-image

Unlike JDK 7 it is not currently built as a Mac OS X package.

3.0 Testing The Build

To check whether I had a functioning build at least as far as Java Lambda Expressions were concerned I wrote three very simple pieces of code.

All of them use the forthcoming java.util.streams.Stream interface because it defines lots of methods which can be passed Lambda Expressions.

All of them actually work as expected when compiled and run using the Project Lambda JDK that I built.

3.1 Echo

    package xper.lambda;

    import java.util.Arrays;

    public final class Echo
    {
        public static void main(String[] theArgs)
        {       
            new Echo().run(theArgs);
        }
    
        //
    
        private Echo()
        {
        }
    
        //
    
        private void run(String[] theArgs)
        {
            first = true;
            Arrays.
                stream(
                    theArgs).
                forEach(
                    s -> { 
                             if (!first) 
                             { 
                                 System.out.print(' '); 
                             }
                             else
                             {
                                 first = false;
                             }
                             System.out.print(s);
                         });
            System.out.println();
        }
    
        //
    
        private boolean first;
    }

The code creates a Stream from the program’s arguments using the Arrays.stream() utility method and then uses the Stream’s forEach() method to apply a Lambda Expression to each of the arguments.

The forEach() method is declared like this

    void forEach(Block<? super T> block);

and the Block interface is defined in the java.util.functions package like this

    public interface Block {

    /**
     * Performs operations upon the provided object which may modify that object
     * and/or external state.
     *
     * @param t an input object
     */
    void apply(T t);

    /**
     * Returns a Block which performs in sequence the {@code apply} methods of
     * multiple Blocks. This Block's {@code apply} method is performed followed
     * by the {@code apply} method of the specified Block operation.
     *
     * @param other an additional Block which will be chained after this Block
     * @return a Block which performs in sequence the {@code apply} method of
     * this Block and the {@code apply} method of the specified Block operation
     */
    public Block<T> chain(Block<? super T> other) default {
        return (T t) -> { apply(t); other.apply(t); };
    }
}

At first glance it is not entirely obvious why the forEach() method accepts a Lambda Expression at all, but the
Block interface is what JSR 335 defines as a functional interface

A functional interface is an interface that has just one abstract method, and thus represents a single function contract. (In some cases, this “single” method may take the form of multiple abstract methods with override-equivalent signatures inherited from superinterfaces; in this case, the inherited methods logically represent a single method.)

In addition to the usual process of creating an interface instance by declaring and instantiating a class, instances of functional interfaces can be created with lambda expressions, method references, or constructor references.

Note that although the Block interface declares two methods it is still a functional interface because only one of them is abstract. The other is a default method which defines an implementation of the method despite appearing in an interface declaration. This is another new feature defined by JSR 335.

A default method is a method that is declared in an interface and that has a block body prefixed by the keyword default. Its body provides a default implementation for any class that implements the interface without overriding the method. This allows new functionality to be added to existing (and perhaps already widely-distributed) interfaces. More generally, it provides a mechanism for multiple inheritance of behavior.

In this example the Lambda Expression passed to the forEach() can be made to match the definition of the Block
interface by the compiler.

3.2 ToUpper

    package xper.lambda;

    import java.util.Arrays;

    public final class ToUpper
    {
        public static void main(String[] theArgs)
        {       
            new ToUpper().run(theArgs);
        }
    
        //
    
        private ToUpper()
        {
        }
    
        private void run(String[] theArgs)
        {
        first = true;
        Arrays.
            stream(
                theArgs).
            map(
                s -> s.toUpperCase()).
            forEach(
                s -> { 
                         if (!first) 
                         { 
                             System.out.print(' '); 
                         }
                         else
                         {
                             first = false;
                         }
                         System.out.print(s);
                     });
        System.out.println();
        }
    
        //
    
        private boolean first;
    }

This code is a modified version of the Echo code above. The map() method is used to covert each of the program’s argument to upper case before they are printed out.

The map() method is declared like this

    <R> Stream<R> map(Mapper<? super T, ? extends R> mapper);

and the Mapper interface is declared in the java.util.functions package like this

    public interface Mapper<T, R> {

        /**
         * Map the provided input object to an appropriate output object.
         *
         * @param t the input object to be mapped.
         * @return the mapped output object.
         */
        R map(T t);

        /**
         * Combine with another mapper producing a mapper which preforms both
         * mappings.
         *
         * @param  Type of output objects from the combined mapper. May be the
         * same type as {@code <U>}.
         * @param after An additional mapping to be applied to the result of this
         * mapping.
         * @return A mapper which performs both the original mapping followed by
         * a second mapping.
         */
        public <V> Mapper<T, V> compose(Mapper<? super R, ? extends V> after) default {
            return Mappers.chain(this, after);
        }
     }

Like the Block interface above it is a functional interface which is why the Lambda Expression

    s -> s.toUpperCase()

can be passed to the map() method.

3.3 Sum

    package xper.lambda;

    import java.util.Arrays;

    public final class Sum
    {
        public static void main(String[] theArgs)
        {       
            System.out.print("sum == ");
            System.out.println(
                Arrays.
                    stream(
                        theArgs).
                    map(
                        s -> Integer.parseInt(s)).
                    reduce(
                        0, 
                        (i, j) -> i + j));
        }
    }

This code relies heavily on auto-boxing and unboxing and type inference by the compiler and is an example of how gnomic it is possible to be when using Lambda Expressions.

It uses the map() method to produce a Stream of Integers and the reduce() method to sum them.

The reduce() method is declared like this

    T reduce(T base, BinaryOperator<T> op);

and the BinaryOperator interface is declared in the java.util.functions package like this

   public interface BinaryOperator<T> extends Combiner<T,T,T> {

        public T operate(T left, T right);

        @Override
        T combine(T t1, T t2) default {
            return operate(t1, t2);
        }
    }

Like the Block and Mapper interfaces this is a functional interface which is why it is possible to pass the Lambda Expression

    (i, j) -> i + j

to the reduce() method.

The resulting program behaves as expected.

For example, the command

    java xper.lambda.Sum 1 2 3 4 5 6

produces the output

    sum == 21

Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

June 6, 2012

Adventures In Distributed Garbage Collection: Part Eighteen – The Great Service Record Mystery Revisited

Filed under: Android, Android Internals, Java, Mobile Java, Services — Tags: , , , , — Simon Lewis @ 9:31 am

1.0 Is The ServiceRecord Part Of A Binder Reference Cycle ?

As we have seen it is possible for a Java Binder object to be part of a Binder reference cycle which will prevent it from being garbage collected, and we know that a ServiceRecord is a Java Binder object, so does this explain what is happening in this case ?

To be sure we need to find a Binder reference cycle which starts and ends at the ServiceRecord object [0x406f89b8]

1.1 The ServiceRecord

We know that there is a JNI Global reference to the ServiceRecord object [0x406f89b8] which means that there exists at least one BinderProxy somewhere that references it.

1.2 The ServiceRecord BinderProxy

We know

  1. that the ServiceRecord object representing a Service is passed to the process running that Service via a remote procedure call to the scheduleCreateService() method of the ApplicationThread

  2. that during the the unmarshalling of the arguments to this method a BinderProxy object which references the ServiceRecord object is created

  3. that this BinderProxy object is passed to the Service.attach() method as the token argument

  4. that it is then assigned to the mToken instance variable of the Service Component object that implements the Service

  5. that in this case the Service Component object is the FastnetService object [0x0x40517518].

From this and in conjunction with the object graph we can identify the Java object in the Service process which references the ServiceRecord object [0x0x0x406f89b8] in the System process as the BinderProxy object [0x40513580].

Direct inspection of the Java heap dump confirms that the only effective reference to the BinderProxy object [0x40513580] is from the FastnetService object [0x0x40517518].

We also know that the process running the Service is the only one to which the ServiceRecord object representing that Service is sent, so this is the only BinderProxy which references the ServiceRecord object.

1.3 The FastnetService Object

We can see from this object graph that there are two references to the FastnetService object [0x40517518]

  • one from the ContextImpl object [0x40517548], and

  • one from the FastnetServiceBinder object [0x40517fc0].

We can ignore the reference from the ContextImpl object [0x40517548] as it in turn is only referenced by the FastnetService object [0x40517518] and an inner class object which only it in turn references.

The reference that is keeping the FastnetService object [0x40517518] in existence is the one from the FastnetServiceBinder object [0x40517fc0].

1.4 The FastnetServiceBinder Object

We know

  1. that the FastnetServiceBinder [0x40517fc0] object is a Java Binder object

  2. that there is a JNI Global reference to it

  3. that the JNI Global reference is the only effective reference to it, that is, it is the one which is preventing it from being garbage collected

  4. that it was returned from a call to the FastnetService implementation of the onBind() method

We can infer from the Service process runtime Binder information below

    binder proc state:
    proc 301
      thread 301: l 00
      thread 307: l 12
      thread 308: l 11
      node 4335: u0014c9a0 c00094f88 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      node 4358: u001568a8 c00156888 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      ref 4330: desc 0 node 1 s 1 w 1 d (null)
      ref 4333: desc 1 node 147 s 1 w 1 d (null)
      ref 4338: desc 2 node 209 s 1 w 1 d (null)
      ref 4339: desc 3 node 143 s 1 w 1 d (null)
      ref 4340: desc 4 node 196 s 1 w 1 d (null)
      ref 4343: desc 5 node 4342 s 1 w 1 d (null)
      ref 4350: desc 6 node 23 s 1 w 1 d (null)
      ref 4353: desc 7 node 44 s 1 w 1 d (null)

that there can be only one BinderProxy referencing it since there are only two Binders associated with the process and both of them are only referenced from one other process (60) which is in fact the System process.

1.5 The FastnetServiceBinder BinderProxy

We know

  1. that the Java Binder object returned from the call to a Service component’s implementation of the onBind() method is passed to the
    System process via a remote procedure call to the publishService() method of the ActivityManagerService.

  2. that during the unmarshalling of the arguments to this method a BinderProxy object is created which references the original Binder object

  3. that this BinderProxy object is then assigned to the binder instance variable of the IntentBindRecord associated with the ServiceRecord which represents the Service which created the original Binder object

From this and in conjunction with the object graph we can identify the Java object corresponding to the FastnetServiceBinder object [0x40517fc0] in the Service process as being BinderProxy object [0x4069ce20] which is referenced by the IntentBindRecord object [0x40704020].

Direct inspection of the Java heap dump confirms that this is the only reference.

1.6 The IntentBindRecord

We can see from the object graph that there is a single reference to the IntentBindRecord object [0x40704020] from the HashMap object [0x406f8a80].

Direct inspection of the Java heap dump confirms that this the only reference.

We can also see that there is a single reference to the HashMap object [0x406f8a80] from the ServiceRecord object [0x0x406f89b8].

Direct inspection of the Java heap dump confirms this the only reference.

1.7 The ServiceRecord

We are back at the ServiceRecord object [0x0x406f89b8] so there is indeed a Binder reference cycle and it looks like this

2.0 Does It Matter ?

Any Android Service implementation based on the example in the documentation can end up causing a Binder reference cycle.

If it does, then depending upon the exact implementation of the Service and its usage patterns it is possible that the process running the Service will crash when it runs out of memory.This is especially true if the Service component directly or indirectly uses a large amount of memory and a number of Service components are leaked as clients repeatedly bind and unbind.

Given enough different Service implementations all with the same problem then it is possible that the System process will crash as eventually it will run out of JNI Global references.

So it does matter, especially since if a given Service implementation does run into this problem it is not at all obvious what the underlying cause is.

3.0 Bug Fix

One of the problems associated with ServiceRecord reference cycles, namely the Service process runnning out of memory, was reported as an issue two years ago.

There is a fix for it in the Android code base but it is not clear when it actually turned up in shipping products. The submission date was just after the initial release of 2.3 so it probably did not start to appear until later revisions of 2.3 which may mean that there are still many devices on which this is potentially a problem.

The fix is specific to the ServiceRecord reference cycle. It is simply to break the reference cycle in the System process by clearing the HashMap referenced by the bindings instance variable of the ServiceRecord. This is done in the bringDownServiceLocked() method of the ActivityManagerService. Doing this breaks the reference between the ServiceRecord and the associated IntentBindRecord(s).

4.0 A Workaround

It is possble to work around the ServiceRecord reference cycle problem specifically by ensuring that the Java Binder object returned from a call to a Service component’s implementation of the onBind() method, only holds a weak reference to the Service component. This prevents the establishment of a Binder reference cycle.

The Java Binder object should either be an instance of a separate class, as in the Fastnet Service example, or a static inner class of the Service and it should use an instance of java.lang.WeakReference to hold the reference to the Service component.

For example, the Service class

    package xper.service.hebrides;

    import android.app.Service;
    import android.content.Intent;
    import android.os.IBinder;
    import android.util.Log;

    public final class HebridesService 
                       extends 
                           Service 
    {
        public HebridesService()
        {
        }
	
        @Override
        public void onCreate() 
        {
            super.onCreate();
            binder = new HebridesServiceBinder(this);
        }

        @Override
        public IBinder onBind(Intent intent) 
        {
            Log.d(TAG, "onBind(" + intent + ")");
            return binder;
        }
	
        //
	
        @Override
        public boolean onUnbind(Intent intent) 
        {
            Log.d(TAG, "onUnbind(" + intent + ")");
            return super.onUnbind(intent);
        }

        void doNothingInParticular()
        {
		
        }

        //
	
        private HebridesServiceBinder	binder;
	
        //
	
        private static final String TAG = "HebridesService";
    }

and the Binder class

    package xper.service.hebrides;

    import java.lang.ref.WeakReference;

    import android.os.RemoteException;
    import xper.service.hebrides.Hebrides.Stub;

    final class HebridesServiceBinder 
                extends 
                    Stub 
    {
        public void doNothingInParticular() 
                    throws 
                        RemoteException 
        {
            HebridesService hs = service.get();
		
            if (hs != null)
            {
                hs.doNothingInParticular();
            }
        }

        //
	
        HebridesServiceBinder(HebridesService theService)
        {
            service = new WeakReference<HebridesService>(theService);
        }
	
        //
	
        private WeakReference<HebridesService>	service;
    }

5.0 An Alternative Design Pattern

By convention, the documentation, implication, a Service Component is a sub-class of a class called Service, and it is declared in the Android manifest in an element called service, and probably, though not necessarily intention, the Service Component implements the functionality of the Service, that is the methods defined in the AIDL definition of that Service.

If the Service Component implements the functionality of the Service then a Binder created to enable access to that Service necessarily has a reference to the Service Component and it is that reference that results in a Binder reference cycle.

However, as a sub-class of the Service class the only method the Service Component has to implement is onBind(), that is, the only function it is required to perform is that of a Binder Factory.

As the example below shows it is possible to adopt a different design pattern where the Service Component does nothing other than create Binder objects when necessary. Another object, in the example, the Application, implements the Service functionality.

One consequence of this is that the Binder objects do not need to have a reference to the Service Component, but another is that the lifetime of the implementation of the Service is decoupled from that of the Service Component. This can be useful if the setting up and/or tearing down of the implementation of the Service functionality is computationally expensive.

5.1. Example

5.1.1 An Interface Defining The Service

    package xper.service.faeroes;

    public interface FaeroesService 
    {
        public void doNothingInParticular();
    }

5.1.2 The Application Class

    package xper.service.faeroes;

    import android.app.Application;

    public class FaeroesServiceApplication 
                 extends 
                     Application 
                 implements
                     FaeroesService
    {
        public void doNothingInParticular() 
        {		
        }

        //
	
        FaeroesService getService()
        {
            return this;
        }
    }

5.1.3 The Binder Factory Implementation

    package xper.service.faeroes;

    import android.app.Service;
    import android.content.Intent;
    import android.os.IBinder;
    import android.util.Log;

    public class FaeroesServiceBinderFactory 
                 extends 
                     Service 
    {
        @Override
        public void onCreate() 
        {
            super.onCreate();
		
            FaeroesServiceApplication app = (FaeroesServiceApplication)getApplication();
		
            binder = new FaeroesServiceBinder(app.getService());
        }

        //

        @Override
        public IBinder onBind(Intent intent) 
        {
            Log.d(TAG, "onBind(" + intent + ")");
            return binder;
        }

        @Override
        public boolean onUnbind(Intent intent) 
        {
            Log.d(TAG, "onUnind(" + intent + ")");
            return super.onUnbind(intent);
        }
	
        //
	
        private FaeroesServiceBinder binder;
	
        //
	
        private static final String TAG = "FaeroesServiceBinderFactory";
    }

5.1.4 The Binder implementation

    package xper.service.faeroes;

    import android.os.RemoteException;

    final class FaeroesServiceBinder 
                extends
                    Faeroes.Stub
    {
        @Override
        public void doNothingInParticular() 
                    throws 
                        RemoteException 
        {
            service.doNothingInParticular();
        }
	
        //
	
        FaeroesServiceBinder(FaeroesService theService)
        {
            service = theService;
        }

	    //
	
        private FaeroesService service;
    }

5.1.5 The Android Manifest

    <?xml version="1.0" encoding="utf-8"?>
    <manifest xmlns:android="http://schemas.android.com/apk/res/android"
        package="xper.service.faeroes"
        android:versionCode="1"
        android:versionName="1.0" >

        <uses-sdk android:minSdkVersion="9" />

        <application
            android:icon="@drawable/ic_launcher"
            android:label="@string/app_name" android:name="FaeroesServiceApplication">
            <service android:name="FaeroesServiceBinderFactory">
                <intent-filter>
                    <action android:name="xper.service.faeroes.ACTION_FAEROES"/>
                </intent-filter>
            </service>
        </application>
    </manifest>

Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

June 1, 2012

Adventures In Distributed Garbage Collection: Part Seventeen – Binders And Garbage Collection

Filed under: Android, Android Internals, Java, Mobile Java, Services — Tags: , , , , — Simon Lewis @ 9:33 am

1.0 An Alternative Service implementation

Unlike the previous example the following Android Service implementation creates the Java Binder object returned from the onBind() method on demand and it does not retain a reference to it.

    package xper.service.malin;

    import android.app.Service;
    import android.content.Intent;
    import android.os.IBinder;
    import android.util.Log;

    public final class MalinService 
                       extends 
                           Service 
    {
        public MalinService()
        {
        }
	

        @Override
        public IBinder onBind(Intent intent) 
        {
            Log.d(TAG, "onBind(" + intent + ")");
            return new MalinServiceBinder(this);
        }
	
        //
	
        @Override
        public boolean onUnbind(Intent intent) 
        {
            Log.d(TAG, "onUnbind(" + intent + ")");
            return super.onUnbind(intent);
        }

        void doSomethingInParticular()
        {
        }

        //
	
        private static final String TAG = "MalinService";
    }

The MarlinServiceBinder object retains a reference to the MalinService object but that is all.

    package xper.service.malin;

    import android.os.RemoteException;
    import xper.service.malin.Malin.Stub;

    final class MalinServiceBinder 
                extends 
                    Stub 
    {
        public void doSomethingInParticular() 
                    throws 
                        RemoteException 
        {
            service.doSomethingInParticular();
        }

        //
	
        MalinServiceBinder(MalinService theService)
        {
            service = theService;
        }
	
        //
	
        private MalinService	service;
    }

Once it has been created there are no references to the MarlinServiceBinder object from any other Java objects.

We can show this using MAT and by examining the heap dump directly.

As in the ServiceRecord case MAT erroneously reports that the MalinServiceBinder object is referenced by a Native Stack GC Root but examination of the heap dump shows that, as expected, it is actually a JNI Global reference.

In this example, if the JNI Global reference did not exist then the MarlinServiceBinder object would be eligible for garbage collection. This is of course precisely why the Binder C++ code creates the JNI Global reference. It is to ensure that a Java Binder object that is otherwise unreferenced in the process in which it was created is not garbage collected whilest there are references to it from other processes.

2.0 Binder References

Explaining exactly how the Binder reference mechanism works more or less requires explaining exactly how Binders work full-stop, which would take a lot more space than is available in the margin of this post, so what follows is a somewhat simplified explanation.

2.2 Reference Counting

Binders are reference counted. Each BinderProxy which references a Binder constitutes a reference to that Binder.

The Binder reference counting mechanism is, necessarily, cross-process and entirely orthogonal to the per-process Java garbage collection mechanism.

2.2.1 Incrementing The Reference Count

Effectively each time a Binder is sent to another process and a BinderProxy is created for that Binder the reference count is incremented.

2.2.2 Decrementing The Reference Count

The finalize() method of a Java BinderProxy object, that is an instance of the class android.os.BinderProxy, calls the native method destroy() which decrements the reference count of the associated Binder object.

In other words, once a Java BinderProxy object has been created in a process then the reference from it to the associated Binder will exist until that Java BinderProxy object has both been garbage collected and finalized.

2.2 Interaction With Java GC

As we have seen the Binder reference mechanism creates a JNI Global reference to a Java Binder object to prevent it from being garbage collected whilest there remain BinderProxy objects in other processes that reference it.

The JNI Global reference is created the first time the Binder is referenced by a BinderProxy.

The JNI Global reference is destroyed when there are no longer any BinderProxies referencing the Binder as determined by the reference count.

That is the only interaction between the Binder reference mechanism and the Java garbage collection mechanism of the process in which the Binder object was created and it is completely opaque. That is the Binder reference mechanism creates a vanilla JNI Global reference that is indistinguishable from a JNI Global reference created by any other piece of native code.

2.4 Inspecting Per-Process Binder Related State At Runtime

It is possible to inspect the Binders and BinderProxies in use in a given process by looking at the contents of the file

    /proc/binder/proc/<PID>

where <PID> is the id of the process.

For example, entering

 cat /proc/binder/proc/368

in a shell running on the device after the client has bound to the Malin Service results in the following output

    binder proc state:
    proc 368
      thread 368: l 00
      thread 374: l 12
      thread 375: l 11
      node 3879: u0007ccf0 c00094f88 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      node 3904: u001a5bb8 c001a5b98 hs 1 hw 1 ls 0 lw 0 is 2 iw 2 proc 357 60
      ref 3874: desc 0 node 1 s 1 w 1 d (null)
      ref 3877: desc 1 node 135 s 1 w 1 d (null)
      ref 3882: desc 2 node 196 s 1 w 1 d (null)
      ref 3883: desc 3 node 131 s 1 w 1 d (null)
      ref 3884: desc 4 node 183 s 1 w 1 d (null)
      ref 3887: desc 5 node 3886 s 1 w 1 d (null)
      ref 3896: desc 6 node 23 s 1 w 1 d (null)
      ref 3899: desc 7 node 44 s 1 w 1 d (null)

In this case 368 is the id of the process running the Malin Service, 357 is the id of the process running the client, and 60 is the id of the System process.

The corresponding output for the process running the client looks like this

    binder proc state:
    proc 357
      thread 357: l 00
      thread 366: l 12
      thread 367: l 11
      node 3830: u000a5168 c001ad3c0 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      node 3754: u0014c9a0 c00094f88 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      node 3785: u001ad4f0 c001ae468 hs 1 hw 1 ls 0 lw 0 is 2 iw 2 proc 113 60
      node 3783: u001ae450 c001ae430 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      node 3791: u001ae6d0 c001ae6b0 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      node 3870: u001b3ba8 c001b3b88 hs 1 hw 1 ls 0 lw 0 is 1 iw 1 proc 60
      ref 3749: desc 0 node 1 s 1 w 1 d (null)
      ref 3752: desc 1 node 135 s 1 w 1 d (null)
      ref 3757: desc 2 node 196 s 1 w 1 d (null)
      ref 3758: desc 3 node 131 s 1 w 1 d (null)
      ref 3759: desc 4 node 183 s 1 w 1 d (null)
      ref 3762: desc 5 node 3761 s 1 w 1 d (null)
      ref 3772: desc 6 node 23 s 1 w 1 d (null)
      ref 3775: desc 7 node 44 s 1 w 1 d (null)
      ref 3781: desc 8 node 219 s 1 w 1 d (null)
      ref 3789: desc 9 node 3788 s 1 w 1 d (null)
      ref 3799: desc 10 node 3798 s 1 w 1 d (null)
      ref 3803: desc 11 node 3802 s 1 w 1 d (null)
      ref 3807: desc 12 node 3806 s 1 w 1 d (null)
      ref 3828: desc 13 node 272 s 1 w 1 d (null)
      ref 3843: desc 14 node 3821 s 1 w 1 d (null)
      ref 3866: desc 15 node 765 s 1 w 1 d c5ff8940
      ref 3907: desc 16 node 3904 s 1 w 1 d c469be40

2.4.1 Binder Entries

The lines with the prefix node describe Binder objects that have been created in the process.

The numbers after the proc keyword are the ids of the processes which hold references to that Binder, that is, there
are BinderProxy objects in those processes which reference the Binder.

For example, the entry

      node 3904: u001a5bb8 c001a5b98 hs 1 hw 1 ls 0 lw 0 is 2 iw 2 proc 357 60

from the first example above from the Service process, 368, specifies that the System process (60) and the client process (357) both have BinderProxy objects which reference this Binder in the service process.

By inference this entry must be for the MalinServiceBinder Java Binder object.

2.4.2 BinderProxy Entries

The lines with the prefix ref describe the BinderProxy objects that have been created in the process

The number after the node keyword is the id of the Binder the BinderProxy references.

For example, the entry

      ref 3907: desc 16 node 3904 s 1 w 1 d c469be40

from the second example above from the client process, 357, specfies the BinderProxy which references the Binder with id 3904 which as we have seen is a Binder in the service process, 368.

3.0 A Binder Cycle Made For Two

The way the Binder reference mechanism works means that it possible to create cross-process reference cycles such that none of the objects in the cycle can be garbage collected if they are Java objects or freed if they are C++ objects despite the fact that none of them are in use. In short a memory leak.

The simplest cycle requires just two Java Binder objects in two separate processes each of which holds a Java BinderProxy object referencing the other, like so


Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

May 30, 2012

Adventures In Distributed Garbage Collection: Part Sixteen – The Great ServiceRecord Mystery

1.0 The Mystery

Why does the ServiceRecord object representing a non-existent Service still exist in the Java heap of the System process ?

2.0 The Object Graph: A Recap

The object graph shows two references to the ServiceRecord object [0x406f89b8] from other objects in the graph but they are themselves only referenced from the ServiceRecord object itself and therefore cannot be responsible for its continued existence.

3.0 MAT: A Recap

Using the MAT

 Path To GC Roots > with all references

option on the errant ServiceRecord object produces this

There is a reference to the ServiceRecord object from a Native Stack GC Root.

The MAT Garbage Collection Roots documentation says

Native Stack

In or out parameters in native code, such as user defined JNI code or JVM internal code. This is often the case as many methods have native parts and the objects handled as method parameters become GC roots. For example, parameters used for file/network I/O methods or reflection.

The implication is that the ServiceRecord object has been passed to a native method as an argument which is executing at the point the Java heap dump was taken. This is unlikely but not impossible. However we have seen exactly the same thing in a different process. Again this is not impossible but it is suspicious and it does suggest that it might be a good idea to try and find out what exactly is going on in this particular case. One way to find out is to look at the contents of the hprof file directly.

4.0 The Heap Dump vs. MAT

Looking directly at the contents of the hprof file containing the dump of the Java heap turns up a single JNI Global reference (HPROF_GC_ROOT_JNI_GLOBAL) rather than a native stack reference (HPROF_GC_ROOT_NATIVE_STACK) which is odd.

A JNI Global reference is not the same as a reference from a native stack frame and in fact there is a separate entry for JNI Global in the MAT documentation for Garbage Collection Roots which reads

Global variable in native code, such as user defined JNI code or JVM internal code.

so why has MAT decided to conflate the two ?

The answer appears to lie in the following method which is defined in the class org.eclipse.mat.hprof.Pass1Parser (code slightly re-formatted)

    private void readDumpSegments(long length) throws IOException, SnapshotException
    {
        long segmentStartPos = in.position();
        long segmentsEndPos = segmentStartPos + length;

        while (segmentStartPos < segmentsEndPos)
        {
            long workDone = segmentStartPos / 1000;
            if (this.monitor.getWorkDone() < workDone)
            {
                if (this.monitor.isProbablyCanceled())
                    throw new IProgressListener.OperationCanceledException();
                this.monitor.totalWorkDone(workDone);
            }

            int segmentType = in.readUnsignedByte();
            if (verbose)
                System.out.println("    Read heap sub-record type "+segmentType+" at position 0x"+ \
                   Long.toHexString(segmentStartPos)); //$NON-NLS-1$ //$NON-NLS-2$
            switch (segmentType)
            {
                case Constants.DumpSegment.ROOT_UNKNOWN:
                    readGC(GCRootInfo.Type.UNKNOWN, 0);
                    break;
                case Constants.DumpSegment.ROOT_THREAD_OBJECT:
                    readGCThreadObject(GCRootInfo.Type.THREAD_OBJ);
                    break;
				
                case Constants.DumpSegment.ROOT_JNI_GLOBAL:
                    readGC(GCRootInfo.Type.NATIVE_STACK, idSize);
                    break;
				
                case Constants.DumpSegment.ROOT_JNI_LOCAL:
                    readGCWithThreadContext(GCRootInfo.Type.NATIVE_LOCAL, true);
                    break;
                case Constants.DumpSegment.ROOT_JAVA_FRAME:
                    readGCWithThreadContext(GCRootInfo.Type.JAVA_LOCAL, true);
                    break;
                case Constants.DumpSegment.ROOT_NATIVE_STACK:
                    readGCWithThreadContext(GCRootInfo.Type.NATIVE_STACK, false);
                    break;
                case Constants.DumpSegment.ROOT_STICKY_CLASS:
                    readGC(GCRootInfo.Type.SYSTEM_CLASS, 0);
                    break;
                case Constants.DumpSegment.ROOT_THREAD_BLOCK:
                    readGC(GCRootInfo.Type.THREAD_BLOCK, 4);
                    break;
                case Constants.DumpSegment.ROOT_MONITOR_USED:
                    readGC(GCRootInfo.Type.BUSY_MONITOR, 0);
                    break;
                case Constants.DumpSegment.CLASS_DUMP:
                    readClassDump(segmentStartPos);
                    break;
                case Constants.DumpSegment.INSTANCE_DUMP:
                    readInstanceDump(segmentStartPos);
                    break;
                case Constants.DumpSegment.OBJECT_ARRAY_DUMP:
                    readObjectArrayDump(segmentStartPos);
                    break;
                case Constants.DumpSegment.PRIMITIVE_ARRAY_DUMP:
                    readPrimitiveArrayDump(segmentStartPos);
                    break;
                default:
                    throw new SnapshotException(MessageUtil.format( \
                                  Messages.Pass1Parser_Error_InvalidHeapDumpFile,
                                    segmentType, segmentStartPos));
            }

            segmentStartPos = in.position();
        }
       
        ...
    }

Curiously the class org.eclipse.mat.snapshot.model.GCRootInfo does not define a NATIVE_GLOBAL constant for use in this case which would seem to imply that this is intentional if rather unhelpful behaviour.

5.0 Hunt The JNI Global Reference

So the ServiceRecord object is still in the Java heap because there is a JNI Global reference to it. Why ?

At this point it looks as though we may be stuck. There is obviously no record in the heap dump of how or why individual JNI Global references were created by native code nor could there be.

On the other hand for native code to have created a JNI Global reference to the ServiceRecord object, it must have been passed to a native method at some point. None of the ActivityManagerService methods discussed to date have been native. What about the ServiceRecord class ? It does not have any native methods, on the other hand its super class android.os.Binder does so what about them ?

6.0 android.os.Binder Native Methods

The first native method to be invoked in the lifecycle of a Binder object is the init() method which is invoked at the very start of the default constructor

    public Binder()

It is declared as follows

    private native final void init();

The actual implementation can be found in the file frameworks/base/core/jni/android_util_Binder.cpp, lines 633-643, and it looks like this

    static void android_os_Binder_init(JNIEnv* env, jobject clazz)
    {
        JavaBBinderHolder* jbh = new JavaBBinderHolder(env, clazz);
        if (jbh == NULL) {
            jniThrowException(env, "java/lang/OutOfMemoryError", NULL);
            return;
        }
        LOGV("Java Binder %p: acquiring first ref on holder %p", clazz, jbh);
        jbh->incStrong(clazz);
        env->SetIntField(clazz, gBinderOffsets.mObject, (int)jbh);
    }

Note that init() is an instance method, so the jobject argument clazz is the Java Binder object itself which is neither a class nor a clazz.

At first glance it does not look very promising. Certainly there is no direct call to the NewGlobalRef() JNI function which would result in the creation of a JNI Global reference. On the other hand the Java Binder object is being passed to the constructor of the C++ class JavaBBinderHolder. Maybe that creates one.

The C++ class JavaBBinderHolder is defined in the file frameworks/base/core/jni/android_util_Binder.cpp, lines 322-359.

Although none of the JavaBBinderHolder methods invoke NewGlobalRef() a JavaBBinderHolder object does retain the pointer to the Java Binder object passed to the constructor, so there is still hope.

In the get() method the pointer to the Java Binder object is passed to the constructor of the C++ class JavaBBinderr.

The C++ class JavaBBinder is also defined in the file frameworks/base/core/jni/android_util_Binder.cpp, lines 234-318.

The constructor is defined like this

   JavaBBinder(JNIEnv* env, jobject object)
        : mVM(jnienv_to_javavm(env)), mObject(env->NewGlobalRef(object))
    {
        LOGV("Creating JavaBBinder %p\n", this);
        android_atomic_inc(&gNumLocalRefs);
        incRefsCreated(env);
    }

and as we can see it duly creates a JNI Global reference to the Java Binder object.

Note

NewGlobalRef() is defined to return NULL if the system runs out of memory, so in general the usage in this method is bad practice. However the Dalvik implementation will blow up the VM if it cannot allocate the reference (see dalvik/vm/Jni.c NewGlobalRef() lines 2195-2207 and addGlobalReference() lines 742-866) so this code is safe for some value of safe.


Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

May 28, 2012

Adventures In Distributed Garbage Collection: Part Fifteen – Post Unbind System State

Filed under: Android, Android Internals, Java, Mobile Java, Services — Tags: , , , , — Simon Lewis @ 8:41 am

1.0 Object State

Once the client has unbound from the Fastnet Service then on the basis of the source code and given that it was the only client and that it specified the BIND_AUTO_CREATE flag when binding to the Fastnet Service, we would expect that

  • the ActivityManagerService would no longer hold a reference to the ServiceRecord representing the Fastnet Service via the mServices instance variable,

  • the ProcessRecord representing the Service process would no longer hold a reference to the ServiceRecord representing the Fastnet Service via
    services instance variable,

  • the ProcessRecord representing the Service client process would no longer hold a reference to the ConnectionRecord representing the connection to the Fastnet Service via the connections instance variable,

  • the ActivityRecord representing the Service client Activity would no longer hold a reference to the ConnectionRecord representing the connection to the Fastnet Service via the connections instance variable

  • the ServiceRecord object representing the Fastnet Service would no longer be present in the Java heap, nor would the associated IntentBindRecord and AppBindRecord objects

  • the ConnectionRecord object represnting the binding to the Fastnet Service from the Service client process would no longer be present in the Java heap.

2.0 dumpsys Output

Below are excerpts from the output of the dumpsys command run immediately after the client has unbound from the Fastnet Service.
In some cases the output has been slightly re-formatted for clarity.

2.1 ProcessRecords

2.1.1 Service Client ProcessRecord

    ...

    *APP* UID 10054 ProcessRecord{4067a100 293:xper.service.fastnet.client/10054}
      dir=/data/app/xper.service.fastnet.client-2.apk \
	  publicDir=/data/app/xper.service.fastnet.client-2.apk \
          data=/data/data/xper.service.fastnet.client
      packageList=[xper.service.fastnet.client]
      thread=android.app.ApplicationThreadProxy@406c1890 curReceiver=null
      pid=293 starting=false lastPss=0
      lastActivityTime=-6m18s641ms lruWeight=253489 keeping=true hidden=false empty=false
      oom: max=15 hidden=7 curRaw=0 setRaw=0 cur=0 set=0
      curSchedGroup=0 setSchedGroup=0
      setIsForeground=false foregroundServices=false forcingToForeground=null
      persistent=false removed=false
      adjSeq=248 lruSeq=57
      lastRequestedGc=-6m18s642ms lastLowMemory=-6m18s642ms reportLowMemory=false
      activities=[HistoryRecord{407ad2d8 xper.service.fastnet.client/.FastnetClientActivity}]
      conProviders={ContentProviderRecord{4051bc78 com.android.providers.settings.SettingsProvider}=1}
	
    ...

As expected the ProcessRecord [0x4067a100] representing the Service client process has no associated connections.

2.1.2 Service ProcessRecord

    ...

    *APP* UID 10053 ProcessRecord{4076e000 301:xper.service.fastnet/10053}
      dir=/data/app/xper.service.fastnet-2.apk \
	      publicDir=/data/app/xper.service.fastnet-2.apk data=/data/data/xper.service.fastnet
      packageList=[xper.service.fastnet]
      thread=android.app.ApplicationThreadProxy@4077e2a0 curReceiver=null
      pid=301 starting=false lastPss=0
      lastActivityTime=-6m11s99ms lruWeight=141059 keeping=false hidden=true empty=true
      oom: max=15 hidden=7 curRaw=7 setRaw=7 cur=7 set=7
      curSchedGroup=1 setSchedGroup=1
      setIsForeground=false foregroundServices=false forcingToForeground=null
      persistent=false removed=false
      adjSeq=248 lruSeq=60
      lastWakeTime=0 time used=0
      lastCpuTime=0 time used=0
      lastRequestedGc=-29s98ms lastLowMemory=-6m11s10ms reportLowMemory=false
	  
    ...

As expected the ProcessRecord [0x4076e000] representing the Service process has no associated services.

2.2 ActivityRecord

    Activities in Current Activity Manager State:
      * TaskRecord{4070aa90 #3 A xper.service.fastnet.client}
        clearOnBackground=false numActivities=1 rootWasReset=true
        affinity=xper.service.fastnet.client
        intent={act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] \
            flg=0x10200000 cmp=xper.service.fastnet.client/.FastnetClientActivity bnds=[83,316][157,395]}
        realActivity=xper.service.fastnet.client/.FastnetClientActivity
        lastActiveTime=253499 (inactive for 378s)
        * Hist #1: HistoryRecord{407ad2d8 xper.service.fastnet.client/.FastnetClientActivity}
            packageName=xper.service.fastnet.client processName=xper.service.fastnet.client
            launchedFromUid=10001 app=ProcessRecord{4067a100 293:xper.service.fastnet.client/10054}
            Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10200000 \
			    cmp=xper.service.fastnet.client/.FastnetClientActivity bnds=[83,316][157,395] }
            frontOfTask=true task=TaskRecord{4070aa90 #3 A xper.service.fastnet.client}
            taskAffinity=xper.service.fastnet.client
            realActivity=xper.service.fastnet.client/.FastnetClientActivity
            base=/data/app/xper.service.fastnet.client-2.apk/data/app/xper.service.fastnet.client-2.apk \
            data=/data/data/xper.service.fastnet.client
            labelRes=0x7f040001 icon=0x7f020000 theme=0x0
            stateNotNeeded=false componentSpecified=true isHomeActivity=false
                configuration={ scale=1.0 imsi=310/260 loc=en_US touch=3 keys=2/1/2 nav=3/1 orien=1 \
                layout=18 uiMode=17 seq=4}
            launchFailed=false haveState=false icicle=null
            state=RESUMED stopped=false delayedResume=false finishing=false
            keysPaused=false inHistory=true launchMode=0
            fullscreen=true visible=true frozenBeforeDestroy=false thumbnailNeeded=false idle=true
            waitingVisible=false nowVisible=true
            connections=[]
       ...

As expected the ActivityRecord [0x4070aa90] representing the Activity running in the Service client process has no associated connections.

3.0 Java Heap State

As before we already have the dumpsys output, but again we can use the the state of the Java heap of the System process to double-check our understanding of the source code.

This graph was generated from a Java heap dump of the System process taken after the Service client has unbound from the Fastnet Service .

Like its counterpart here it was generated using the ServiceRecord object [0x406f89b8] representing the Fastnet Service as the root. As before greyed out nodes have been filtered, that is, one or more instance variables have been omitted for clarity.

This is another problematic graph. If the ServiceRecord object [0x406f89b8] is no longer in the Java heap it should not be possible to generate a graph with it as the root.

Evidently the graph exists therefore the assumption is wrong. So what is referencing it ? According to MAT there is a single GC Root of type Native Stack.

This is an identical situation to the previous one and whilest it is possible that on one occasion there was a transient reference from native code at exactly the point the Java heap dump was taken, two occasions in two separate processes is beginning to look like carelessness on somebody’s part !


Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

May 25, 2012

Adventures In Distributed Garbage Collection: Part Fourteen – Post Unbind Service State

Filed under: Android, Android Internals, Java, Mobile Java, Services — Tags: , , , , — Simon Lewis @ 6:00 am

1.0 Object State

Once the client has unbound from the Fastnet Service then on the basis of the source code we would expect that in the Service process

  • there should no longer be a reference to the FastnetService object from the ActivityThread via its mServices instance variable

  • the FastnetService object should no longer exist

  • the FastnetServiceBinder object should no longer exist

The production of a diagram showing the absence of these things is left as an exercise for the reader.

2.0 Java Heap State

This graph was generated from a Java heap dump of the Service process taken after the client has unbound from the Fastnet Service.

Like its counterpart here it was generated using the FastnetService object [0x40517518] as the root. As before greyed out nodes have been filtered, that is, one or more instance variables have been omitted for clarity.

The existence of this graph is problematic. If the FastnetService object is no longer in the Java heap it should not be possible to generate
a graph with it as the root.

Evidently the graph does exist therefore the assumption that the FastnetService object no longer existing in the Java heap is wrong.

The graph shows two references to the FastnetService object

  • one from the FastnetServiceBinder object [0x40517fc0], and

  • one from the ContextImpl object [0x40517548]

However none of these objects are referenced from any other objects in the graph so this does not explain why the FastnetService object still exists. Either it, or something which references it, is being referenced by something else which is not reachable from any object in the graph.

According to MAT there are no other references to the FastnetService object but there is another reference to the FastnetServiceBinder object.

The reference is identified as Native Stack and it is a GC Root, that is, it is a reference that will ensure the FastnetServiceBinder object will not be garbage collected as long as the GC Root continues to exist.

The Native Stack GC Root is described in the MAT documentation as follows.

In or out parameters in native code, such as user defined JNI code or JVM internal code. This is often the case as many methods have native parts and the objects handled as method parameters become GC roots. For example, parameters used for file/network I/O methods or reflection.

It is not obvious what this reference from the native code might be but presumably it is transient. Possibly the continued existence of the FastnetService object is simply an anomaly that happens to have occurred on this occasion.


Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

May 24, 2012

Adventures In Distributed Garbage Collection: Part Thirteen – Post Unbind Service Client State

Filed under: Android, Android Internals, Java, Mobile Java, Services — Tags: , , , , — Simon Lewis @ 6:00 am

1.0 Object State

Once the client has unbound from the Fastnet Service then on the basis of the source code we would expect that in the Service client process

2.0 Java Heap State

This graph was generated from a Java heap dump of the Service client process taken after the client Activity has unbound from the Fastnet Service by calling the unbindService() method.

Like its counterpart here it was generated using the LoadedApk object [0x40513e90] as the root. As before greyed out nodes have been filtered, that is, one or more instance variables have been omitted for clarity.

The graph shows that as expected the LoadedApk object does not reference the ServiceDispatcher object [0x405213b8]

In addition the following objects no longer appear in the graph

  • the ServiceDispatcher object [0x405213b8]

  • the ConnectionInfo object [0x40521b40]

  • the InnerConection object [0x405216b8]

  • the FastnetServiceConnection object [0x405211f0]

This on its own does not prove conclusively that these objects are no longer in the Java heap, just that they are not reachable via the LoadedApk any more.

However, after loading the Java heap dump into MAT none of the objects can be found either which should mean that they have been garbage collected.

Inspecting the contents of the Java heap dump file directly confirms this.


Copyright (c) 2012 By Simon Lewis. All Rights Reserved.

Older Posts »

Theme: WordPress Classic. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: