Just An Application

October 24, 2014

So Swift Then: Fun With Mangled Names Continued

Continuing where we left off in last week’s thrilling installment, that is, with the encoding of function parameters in mangled function names.

While the return type of a function is necessarily a type and nothing but a type, Swift function parameters come in a variety of different flavours.

Some of these flavours affact how a function is called and what it can be called with.

Mangled function names are potentially used by the linker so the names must include an encoding of the relevant information about the function parameters which affect how the function can be called.

A basic vanilla function parameter is of the form

    'name' ":" 'type'

for example

    func cod(b: Bool)

In the case of a vanilla parameter only the type is significant so we would expect that the parameter would appear in a mangled function name as the encoding of its type.

A vanilla parameter can be augmented with an ‘external name’ which is declared before the ‘name’, for example

    func cod(e b: Bool)

The ‘external name’ must appear in any call to the function, for example

    cod(e: true)

In this case, since the external name is significant, we would expect both it and the type to appear in a mangled function name is some way.

By default function parameters are immutable. A function parameter can be made mutable using the

    var

keywood. for example

    func cod(var b: Bool)

The fact that a parameter of a function is mutable is not apparent to a caller of that function, consequently we would not expect the encoding of a mutable parameter to be any different from that of an immutable one.

A function parameter can be made mutable such that changes to it are visible to the caller using the

    inout

keywood. for example

    func cod(inout b: Bool)

A call to this function would look like this

    cod(&flag)

and flag must be mutable.

In this case we would expect the encoding to comprise the encoding of the parameter type plus something to indicate that it is an inout parameter.

A vanilla parameter can have a default value added, like so

    func cod(b: Bool = true)

A parameter with a default value must have an ‘external name’. If it does not have an explicit one, the ‘name’ is also used as the ‘external name’.

A call to the function above with a value would look like this

    cod(b:true)

and without one, so the default value is used, like this

    cod()

While the presence of a default value does affect how the function can be called, the effect is to define a function with two different entry points.

It is easier for the compiler to handle this by generating what looks like two different functions than for the linker to edit compiled code at the call site.

Hence, in this case we would only expect the ‘external name’ and the parameter type to appear in the parameter encoding.

A Swift function can be defined to take a variable number of parameters lke so

    func cod(flags: Bool...)

In the body of the function

   flags

has the type

   [Bool]

One candidate for the encoding of this kind of parameter would be the encoding of the appropriate array type.

Vanilla Parameters

Compiling

    func cod(b: Bool)
    {
    }

gives us the symbol

    __TF4xper3codFSbT_

Compiling

    func cod(b: Bool, c: Bool)
    {
    }

gives us the symbol

    __TF4xper3codFTSbSb_T_

Compiling

    func cod(b: Bool, c: Bool, d: Bool)
    {
    }

gives us the symbol

    __TF4xper3codFTSbSbSb_T_

On the basis of these three examples so far we can infer the following rules for the encoding of the parameters of a function with N vanilla parameters.

If N == 0 the parameters are represented by the encoding of the empty tuple type

   ()

If N == 1 the parameters are represented by the encoding of

    T

where T is the type of the single parameter.

If N > 1 then the parameters are represented by the encoding of the tuple type

   (T[0], ..., T[N-1])

where T[i] is the type of the i'th parameter.

As it stands the N == 1 case is a bit odd. Why is it a special case ? What if the single parameter has a tuple type ?

Compiling this

    func cod(b: (Bool))
    {
    }

gives us the symbol

    __TF4xper3codFSbT_

NOT

    __TF4xper3codFTSb_T_

which is surprising but it does mean that the rule for N == 1 holds.

Compiling this

    func cod(b: (Bool,Bool))
    {
    }

gives us the symbol

    __TF4xper3codFTSbSb_T_

which is even more surprising.

Two functions with different numbers of parameters end up with the same mangled name.

That cannot be right.

How can you have both of the functions in the same library ?

What happens if you compile both of them in the same file ?

    $ swiftc -module-name xper -emit-library funcs.swift
    Basic Block in function '_TF4xper3codFTSbSb_T_' does not have terminator!
    label %entry1
    LLVM ERROR: Broken function found, compilation aborted!

That's not good.

What about putting them in different files ?

    $ swiftc -module-name xper -emit-library func01.swift func02.swift
    duplicate symbol __TF4xper3codFTSbSb_T_ in:
        [..]/func01-b2b04f.o
        [..]/func02-5b9bf6.o
    ld: 1 duplicate symbol for architecture x86_64
    <unknown>:0: error: linker command failed with exit code 1 (use -v to see invocation)

That's not good either.

OK, them's the rules and they are broken.

Sidles away nonchalantly, hands in pockets, hoping no one is going to ask him to pay for the damage.

Parameter Type Substitution Syntax

There is one slight twist with the encoding of function parameter types even in the all vanilla parameters case.

Compiling

    func cod(c1: Character, c2: Character)
    {
    }

gives us the symbol

    __TF4xper3codFTOSs9CharacterS0__T_

rather than

    __TF4xper3codFTOSs9CharacterOSs9CharacterST_

If we read

    S0_

as substitute the 0th parameter type then it makes sense.

External Names

Adding an external name to our first example

    func cod(e b: Bool)
    {
    }

and compiling gives us the symbol

    __TF4xper3codFT1eSb_T_

We now have the function's parameters represented by

    T1eSb_

which looks like the encoding of the named tuple type

    (e:Bool)

Adding external names to our second example

    func cod(e b: Bool, f c: Bool)
    {
    }

and compiling gives us the symbol

    __TF4xper3codFT1eSb1fSb_T_

and we now have the function's parameters represented by an encoding of the named tuple type

    (e:Bool, f:Bool)

Adding external names to the first two parameters of our third example

    func cod(e b: Bool, f c: Bool, d: Bool)
    {
    }

and compiling gives us the symbol

    __TF4xper3codFT1eSb1fSbSb_T_

and we now have the function's parameters represented by an encoding of the hybrid named/unnamed tuple type

    (e:Bool, f:Bool, Bool)

as you might expect.

Mutable Parameters

Modifying our first example once more

    func cod(var b: Bool)
    {
    }

and compiling gives us the symbol

    __TF4xper3codFSbT_

as expected the presence of the var keyword has no effect on the encoding of the paramater.

inout Parameters

Compiling

    func cod(inout b: Bool)
    {
    }

gives us the symbol

    __TF4xper3codFRSbT_

and

    RSb

for the encoding of the parameter with an

    R

for 'Reference' or 'Recondite' or something.

An inout parameter can have an 'external name'.

Compiling

    func cod(inoute b: Bool)
    {
    }

gives us the symbol

    __TF4xper3codFT1eRSb_T_

and

    T1eRSb_

for the encoding of the parameter.

Default Values

Compiling

    func cod(b: Bool = true)
    {
    }

gives us the symbol

    __TF4xper3codFT1bSb_T_

and a second symbol

    __TIF4xper3codFT1bSb_T_A_

The first symbol is the same as the symbol for the function

    func cod(b b: Bool)
    {
    }

and would enable the linker to identify the entry point for calls to the function made with a value.

The second symbol presumably enables the linker to identify the entry point for the calls to the function made without a value.

Adding another parameter

    func cod(s:String, b: Bool = true)
    {
    }

and compiling gives us the two symbols

    __TF4xper3codFTSS1bSb_T_

and

    __TIF4xper3codFTSS1bSb_T_A0_

because there are still only two ways to call the function, for example

    cod("")

and

    cod("", b:false)

Compiling a function with two parameters both with default values

    func cod(i:Int = 2, b: Bool = true)
    {
    }

and compiling gives us three symbols

    __TF4xper3codFT1iSi1bSb_T_
    __TIF4xper3codFT1iSi1bSb_T_A0_
    __TIF4xper3codFT1iSi1bSb_T_A_

as there are three ways to call the function.

In all these examples it looks as though it is the 'A' suffix on the 'secondary' symbols which identifies the parameter value or values which are being defaulted.

Varadic Parameters

Compiling

    func cod(flags: Bool...)
    {
    }

gives us the symbol

    __TF4xper3codFtGSaSb__T_

and

   tGSaSb__

for the encoding of the parameter.

We have

   GSaSb_

for

  [Bool]

but 't' rather than 'T' for the tuple.

Adding another parameter

    func  cod(s: String, flags:  Bool...)
    {
    }

and compiling gives us the symbol

    __TF4xper3codFtSSGSaSb__T_

A varadic parameter can have an 'external name'

    func  cod(flags f:  Bool...)
    {
    }

Compiling this gives us the symbol

    __TF4xper3codFt5flagsGSaSb__T_

The 'external name' is now present as we would expect.

In all these examples the 't' remains resolutely lower-case so it looks as though it is connected with the presence of the varadic parameter.

Not The Summary

This is another post that is now way too long so its time to call a halt.

Coming up next time, what has the function name mangling machine got against single element type tuples ?


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog's author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

October 23, 2014

So Swift Then: Fun With Mangled Names

For a while writing about Swift was something of a Sisyphean task as features kept changing either just after I had posted about them, or just before I was about to, so in the end I decided to wait for the dust to settle before writing anything else.

With the advent of iOS 8.1, OS X 10.10 and Xcode 6.1 there is now a version of Swift which cannot keep changing.

I am not assuming that there will not be any more changes it just that for the moment I am basing everything on the version available in Xcode 6.1.

As is my wont I have started trying to do something ‘real’ in Swift and while rummaging around trying to work out how to do something I stumbled upon Swift’s version of mangled names.

These turn out to be quite interesting in what they reveal about how Swift works, or at least how it currently works, and they are also turn out to be a good way of gaining an understanding of certain aspects of Swift as a language.

Swift mangled names can be found by looking at the symbols in Swift dynamic libraries.

If you have a Swift file you can turn it into a Swift dynamic library by doing something like this

    swiftc -emit-library -module-name xper functions.swift

You can then use nm to look at the resulting symbols.

The simplest possible Swift function definition is this

    func cod() -> Void
    {
    }

which defines a function which takes no arguments and returns nothing.

When a Swift function returns Void the return type can be omitted so the simplest possible Swift function
definition is actually

    func cod()
    {
    }

Compiling this in the module xper gives us the symbol

    __TF4xper3codFT_T_

which unsurprisngly doesn't tell us a great deal, although we might conjecture that names are encoded as their length in ASCII followed by the characters, hence

    4xper

and

    3cod

What if we try returning something ?

Compiling the definition

    func cod() -> Bool
    {
        return true
    }

gives us the symbol

    __TF4xper3codFT_Sb

The trailing

    T_

has been replaced by

    Sb

so it looks like the return type is at the end

Lets try returning a Character instead.

Compiling the definition

    func cod() -> Character
    {
        return "A"
    }

gives us the symbol

    __TF4xper3codFT_OSs9Character

Not sure what that is about but the name 'Character' is encoded as we would expect and it is definitely at the end.

Trying some more return types

    func cod() -> Double
    {
        return 0.0
    }

gives us the symbol

    __TF4xper3codFT_Sd

while

    func cod() -> Int
    {
        return 0
    }

gives us the symbol

    __TF4xper3codFT_Si

and

    func cod() -> UInt
    {
        return 0
    }

gives us the symbol

    __TF4xper3codFT_Su

So far so good. Everything is at the end.

Some size specific integers.

    func cod() -> Int16
    {
        return 0
    }

gives us the symbol

    __TF4xper3codFT_VSs5Int16

and

    func cod() -> Int32
    {
        return 0
    }

gives us the symbol

    __TF4xper3codFT_VSs5Int32

OK, so not a lot like the Int case.

What about the unsigned versions ?

    func cod() -> UInt16
    {
        return 0
    }

gives us the symbol

   __TF4xper3codFT_VSs6UInt16

and

    func cod() -> UInt32
    {
        return 0
    }

gives us the symbol

   __TF4xper3codFT_VSs6UInt32

Not making a lot of progress right now. Although all the signed and unsigned integer types seem to be encoded as the same kind of 'something' it is not currently obvious what the 'something' is.

Lets try something different.

What about returning an array ?

    func cod() -> [Int]
    {
        return []
    }

gives us the symbol

    __TF4xper3codFT_GSaSi_

We've got an

    Si

at least.

Presumably the

    GSa

prefix is the code for 'array'

We have also got a

    _

suffix.

A dictionary ?

    func cod() -> [Int: Int]
    {
        return [Int: Int]()
    }

gives us the symbol

    __TF4xper3codFT_GVSs10DictionarySiSi_

Now we've got a

    VSs

again, albeit with a 'G' prefix.

We've also got

    Si

twice which does at least make sense, and another

    _

suffix.

How about a tuple ?

    func cod() -> (Int, Int)
    {
        return (0, 1)
    }

gives us the symbol

    __TF4xper3codFT_TSiSi_

We've got a

    T

prefix, followed by

    Si

twice, corresponding to the tuple element types, followed by a

    _

suffix, which is interesting, because if

    TSiSi_

encodes

    (Int, Int)

then presumably

    T_

encodes

    ()

Given that Void is simply an alias for the empty tuple

    ()

then we would would expect the return type of a function with a Void return type to be encoded as

    T_

and if we look at the first example we see that it is.

Note also that in every example to date the encoding of the return type has been preceded by

    T_

and in every example to date the function has no arguments.

Carrying on with return types for the moment.

What about returning a String ?

    func cod() -> String
    {
        return ""
    }

gives us the symbol

     __TF4xper3codFT_SS

String is analagous to Bool, Double , Int and UInt it would appear.

Time to try returning some non-builtin defined types

Given the class Thing then

    func cod() -> Thing
    {
        return Thing()
    }

gives us the symbol

    __TF4xper3codFT_CS_5Thing

which gives us

   C

for class presumably.

In addition to classes there are protocols, so lets return one.

Given the protocol ByteSource implemented by the class ByteBuffer then compiling

    func cod() -> ByteSource
    {
        return ByteBuffer()
    }

gives us the symbol

    __TF4xper3codFT_PS_10ByteSource_

so that's

   P

for 'protocol' then, except that we have a '_' suffix which implies that there can be more than one protocol name so its really 'protocols'

Compiling this, where ByteSink is an additional protocol and the class ByteBuffer now implements both ByteSink and ByteSource

    func cod() -> protocol<ByteSource,ByteSink>
    {
        return ByteBuffer()
    }

duly gives us the symbol

    __TF4xper3codFT_PS_8ByteSinkS_10ByteSource_

Then there is the 'no protocol' case

    func cod() -> protocol<>
    {
        return 0
    }

duly gives us the symbol

    __TF4xper3codFT_P_

If you are wondering what you can actually do with the result of that function the answer is anything that you can do with something of type Any.

The type

    Any

is simply an alias for

    protocol<>

But I digress.

Onwards with enums.

Given an enum Element then compiling

    func cod() -> Element
    {
        return Element.He
    }

gives us the symbol

    __TF4xper3codFT_OS_7Element

Interestingly we've seen something like this before.

The encoding for Character is

    OSs9Character

so we've got

    O S_ 7Element

and

    O Ss 9Character

If

    O

is the type prefix for an enum, then we have

    "O" 'something' 'enum name'

We've also seen

    "C" 'something' 'class name'

in the class example above, and

    "P" 'something' 'protocol name' "_"

in the protocol example above.

No idea about the 'something' as yet, so moving right along.

What about a struct ?

Given an empty struct AnotherThing

    func cod() -> AnotherThing
    {
        return AnotherThing()
    }

gives us the symbol

    __TF4xper3codFT_VS_12AnotherThing

We've seen some types with a 'V' prefix before, namely

  • VSs5Int16

  • VSs5Int32

  • VSs6UInt16

  • VSs6UInt32

as well as something that might have either a 'GV' or a 'V' prefix

    GVSs10DictionarySiSi_

We know that dictionaries and structs are passed by value so 'V' might mean value, but so are arrays and the array type encoding we have seen does not have a 'V' prefix

We also know that explicitly sized signed and unsigned integer types are actually structs so for the moment we will assume that

    V

is the type prefix for struct.

We now have four type encodings of the form

    'type prefix' 'something' 'type name'

In the class and protocol case the 'something' is

    S_

In the enum cases the 'something' is either

    S_

or

    Ss

The same thing is true in the struct cases

In all the examples to date the 'something' is always

    S_

when the type is local to the module and

    Ss

when it is a built-in type.

It looks as though 'something' might be the module name where

    S_

is 'this module' and

    Ss

is 'Swift'

We can try and confirm this by moving one of the local types into another module.

If we move the Element type into the module other and compile this

    import other
    
    func cod() -> other.Element
    {
        return other.Element.He
    }

we would expect the resulting symbol to be

    __TF4xper3codFT_O5other7Element

and it is.

Since types can be nested in Swift the type encodings are likely to actually be of the form

    'type prefix' 'fully quaified type name'

Compiling

    struct Node
    {
        enum Colour: UInt8
        {
            case Red   = 0
            case Black = 1
        }

        var colour = Colour.Red
    }

    func cod() -> Node.Colour
    {
        return Node.Colour.Black
    }

gives us the symbol

    __TF4xper3codFT_OVS_4Node6Colour

which gives us

    "O" 'fully qualified type name'

where the 'fully qualified name' is three elements long.

What else can a function return ?

There are optionals.

An optional Int

    func cod() -> Int?
    {
        return nil
    }

gives us the symbol

    __TF4xper3codFT_GSqSi_

This looks as though it follows the same pattern as the encoding for array, dictionary, and tuple types, namely

    'type prefix' 'element-type(s)' "_"

What about an optional array ?

    func cod() -> [Int]?
    {
        return nil
    }

gives us the symbol

    __TF4xper3codFT_GSqGSaSi__

which gives us a return type encoding of

    "GSq" 'array type encoding' "_"

as we would expect.

We now have three encodings, array, dictionary and optional, which have a 'G' prefix and a '_' suffix.

All three are generic types so it looks as though their encodings are instances of a more general generic type encoding

Defining the canonical generic type Stack<T> and compiling this

    func cod() -> Stack<Int>
    {
        return Stack<Int>()
    }

gives us the symbol

    __TF4xper3codFT_GVS_5StackSi_

which matches the form of the dictionary type encoding.

Where there is a '?' there is always a '!'

Compiling

    func cod() -> Int!
    {
        return nil
    }

gives us the symbol

    __TF4xper3codFT_GSQSi_

so SQ is to '!' as Sq is to '?'.

What about types ? Can you return a type ? You can access them so, so you should be able to return them.

Compiling

    func cod() -> UInt16.Type
    {
        return UInt16.self
    }

gives us the symbol

    __TF4xper3codFT_MVSs6UInt16

which gives us another type prefix

    M

for

    Meta

or something like that.

And of course, functions can return functions

Compiling the not terribly useful functions

    func zero() -> Int
    {
         return 0
    }
    
    func cod() -> () -> Int
    {
        return zero
    }

gives us the symbol

    __TF4xper3codFT_FT_Si

which would appear to give us

    F

as the type prefix for a function.

This post is already way too long so the encoding of function parameters will have to be the next post.

In the meantime here is a summary of what we know so far about how Swift types are encoded in mangled names in the form of an ad-hoc syntax diagram

    type-encoding            := builtin-type
                                |
                                class-type-encoding
                                |
                                enum-type-encoding
                                |
                                function-type-encoding
                                |
                                generic-type-encoding
                                |
                                meta-type-encoding
                                |
                                protocols-type-encoding
                                |
                                struct-type-encoding
                                |
                                tuple-type-encoding
    
                     
    builtin-type             := "SS"    // String
                                |
                                "Sb"    // Bool
                                |
                                "Sd"    // Double
                                |
                                "Si"    // Int
                                |
                                "Su"    // Uint
    
                    
    class-type-encoding      := "C" fully-qualified-name
    
    
    enum-type-encoding       := "O" fully-qualified-name
    
    
    function-type-encoding   := "F" ???? type-encoding
    
    
    generic-type-encoding    := "G" "Sa" type-encoding "_"                  // array
                                |
                                "G" class-type-encoding type-encoding+ "_"  // generic class
                                |
                                "G" enum-type-encoding type-encoding+ "_"   // generic enum
                                |
                                "G" struct-type-encoding type-encoding+ "_" // generic struct
                                |
                                "G" "SQ" type-encoding "_"                  // implicit optional
                                |
                                "G" "Sq" type-encoding "_"                  // optional
                                
                                
    meta-type-encoding       := "M" type-encoding                          // ???? conjecture based on one example ! ????
                                
                                
    protocols-type-encoding  := "P" fully-qualified-name* "_"
    
    
    struct-type-encoding     := "V" fully-qualified-name
    
    
    tuple-type-encoding      := "T" type-encoding* "_"

Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog's author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 30, 2014

Building The Android Runtime (ART) For Mac OS X: Part Eight — Testing, Testing

Having got the Android Runtime to build, eventually, does it actually work ?

Running art the immediate answer is no,

    $art
    E/JniInvocation(46765): Failed to dlopen libart.so: dlopen(libart.so, 2): image not found
    Failed to initialize JNI invocation API from libart.so

The art executable is a script which contains the following so it looks as though this problem is easily fixed.

    ...
    
    invoke_with=
    DALVIKVM=dalvikvm
    LIBART=libart.so
    
    ...

Trying again after modifying the art script appropriately we get

    $art
    art I 46796 66656 art/runtime/gc/space/image_space.cc:269] RelocateImage: \
        /Users/simon/Scratch/art/out/host/darwin-x86/bin/../bin/patchoat \
            --input-image-location=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.art \
            --output-image-file=/Users/simon/Scratch/xper/art/android-data46788/dalvik-cache/x86_64/Users@simon@Scratch@art@out@host@darwin-x86@bin@..@framework@core.art \
            --input-oat-location=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.oat \
            --output-oat-file=/Users/simon/Scratch/xper/art/android-data46788/dalvik-cache/x86_64/Users@simon@Scratch@art@out@host@darwin-x86@bin@..@framework@core.oat \
            --instruction-set=x86_64 --base-offset-delta=9420800
    patchoat W 46797 66657 art/runtime/gc/heap.cc:216] \
        Could not create image space with image file '/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.art'. \
        Attempting to fall back to imageless running. \
        Error was: Failed to load /system image '/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/x86_64/core.art': \
            Failed to mmap at expected address, mapped at 0x100b06000 instead of 0x60000000

and then it hangs/spins.

The address

    0x100b06000

in the error message from patchoat shows that it is a 64-bit executable, so it needs to be rebuilt as a 32-bit executable.

Modifying the file

    art/patchoat/Android.mk

in the same way as we did for dex2oat, rebuilding and trying again, remembering to specify that we want to use a 32-bit VM, we get

    $art --32
    art I 24835 34536 art/runtime/gc/space/image_space.cc:269] RelocateImage: \
        /Users/simon/Scratch/art/out/host/darwin-x86/bin/../bin/patchoat \
            --input-image-location=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.art \
            --output-image-file=/Users/simon/Scratch/xper/art/android-data24827/dalvik-cache/x86/Users@simon@Scratch@art@out@host@darwin-x86@bin@..@framework@core.art \
            --input-oat-location=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.oat \
            --output-oat-file=/Users/simon/Scratch/xper/art/android-data24827/dalvik-cache/x86/Users@simon@Scratch@art@out@host@darwin-x86@bin@..@framework@core.oat \
            --instruction-set=x86 \
            --base-offset-delta=-11399168
    Class name required

which is definitely an improvement.

Trying again with

   -help

to find out how to actually run something we get a long list of arguments we can use and an equally long list of arguments we cannot.

    $art --32 -help
    dalvikvm: [options] class [argument ...]

    The following standard options are supported:
      -classpath classpath (-cp classpath)
      -Dproperty=value
      -verbose:tag ('gc', 'jni', or 'class')
      -showversion
      -help
      -agentlib:jdwp=options

    The following extended options are supported:
      -Xrunjdwp:<options>
      -Xbootclasspath:bootclasspath
      -Xcheck:tag  (e.g. 'jni')
      -XmsN (min heap, must be multiple of 1K, >= 1MB)
      -XmxN (max heap, must be multiple of 1K, >= 2MB)
      -XssN (stack size)
      -Xint
    
    The following Dalvik options are supported:
      -Xzygote
      -Xjnitrace:substring (eg NativeClass or nativeMethod)
      -Xstacktracefile:<filename>
      -Xgc:[no]preverify
      -Xgc:[no]postverify
      -XX:HeapGrowthLimit=N
      -XX:HeapMinFree=N
      -XX:HeapMaxFree=N
      -XX:NonMovingSpaceCapacity=N
      -XX:HeapTargetUtilization=doublevalue
      -XX:ForegroundHeapGrowthMultiplier=doublevalue
      -XX:LowMemoryMode
      -Xprofile:{threadcpuclock,wallclock,dualclock}
        
    The following unique to ART options are supported:
      -Xgc:[no]preverify_rosalloc
      -Xgc:[no]postsweepingverify_rosalloc
      -Xgc:[no]postverify_rosalloc
      -Xgc:[no]presweepingverify
      -Ximage:filename
      -XX:+DisableExplicitGC
      -XX:ParallelGCThreads=integervalue
      -XX:ConcGCThreads=integervalue
      -XX:MaxSpinsBeforeThinLockInflation=integervalue
      -XX:LongPauseLogThreshold=integervalue
      -XX:LongGCLogThreshold=integervalue
      -XX:DumpGCPerformanceOnShutdown
      -XX:IgnoreMaxFootprint
      -XX:UseTLAB
      -XX:BackgroundGC=none
      -XX:LargeObjectSpace={disabled,map,freelist}
      -XX:LargeObjectThreshold=N
      -Xmethod-trace
      -Xmethod-trace-file:filename  -Xmethod-trace-file-size:integervalue
      -Xenable-profiler
      -Xprofile-filename:filename
      -Xprofile-period:integervalue
      -Xprofile-duration:integervalue
      -Xprofile-interval:integervalue
      -Xprofile-backoff:doublevalue
      -Xprofile-start-immediately
      -Xprofile-top-k-threshold:doublevalue
      -Xprofile-top-k-change-threshold:doublevalue
      -Xprofile-type:{method,stack}
      -Xprofile-max-stack-depth:integervalue
      -Xcompiler:filename
      -Xcompiler-option dex2oat-option
      -Ximage-compiler-option dex2oat-option
      -Xpatchoat:filename
      -X[no]relocate
      -X[no]dex2oat (Whether to invoke dex2oat on the application)
      -X[no]image-dex2oat (Whether to create and use a boot image)
        
    The following previously supported Dalvik options are ignored:
      -ea[:<package name>... |:<class name>]
      -da[:<package name>... |:<class name>]
      (-enableassertions, -disableassertions)
      -esa
      -dsa
      (-enablesystemassertions, -disablesystemassertions)
      -Xverify:{none,remote,all}
      -Xrs
      -Xint:portable, -Xint:fast, -Xint:jit
      -Xdexopt:{none,verified,all,full}
      -Xnoquithandler
      -Xjniopts:{warnonly,forcecopy}
      -Xjnigreflimit:integervalue
      -Xgc:[no]precise
      -Xgc:[no]verifycardtable
      -X[no]genregmap
      -Xverifyopt:[no]checkmon
      -Xcheckdexsum
      -Xincludeselectedop
      -Xjitop:hexopvalue[-endvalue][,hexopvalue[-endvalue]]*
      -Xincludeselectedmethod
      -Xjitthreshold:integervalue
      -Xjitcodecachesize:decimalvalueofkbytes
      -Xjitblocking
      -Xjitmethod:signature[,signature]* (eg Ljava/lang/String\;replace)
      -Xjitclass:classname[,classname]*
      -Xjitoffset:offset[,offset]
      -Xjitconfig:filename
      -Xjitcheckcg
      -Xjitverbose
      -Xjitprofile
      -Xjitdisableopt
      -Xjitsuspendpoll
      -XX:mainThreadStackSize=N

Writing a very simple ‘hello world’

    package xper.hw;
    
    public final class HelloWorldART
    {
        public static void main(String[] theArgs)
        {
            System.out.println("Hello World courtesy of the Android Runtime (ART)");
            System.exit(0);
        }
    }

and turning it to a .dex and trying again we get

    art --32 -classpath hw.dex xper.hw.HelloWorldART
    art I 24871 35716 art/runtime/gc/space/image_space.cc:269] RelocateImage: \
        /Users/simon/Scratch/art/out/host/darwin-x86/bin/../bin/patchoat \
            --input-image-location=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.art \
            --output-image-file=/Users/simon/Scratch/xper/art/android-data24863/dalvik-cache/x86/Users@simon@Scratch@art@out@host@darwin-x86@bin@..@framework@core.art\
            --input-oat-location=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.oat \
            --output-oat-file=/Users/simon/Scratch/xper/art/android-data24863/dalvik-cache/x86/Users@simon@Scratch@art@out@host@darwin-x86@bin@..@framework@core.oat --instruction-set=x86\
            --base-offset-delta=-4640768
    dex2oat I 24873 35749 art/dex2oat/dex2oat.cc:1266] \
        /Users/simon/Scratch/art/out/host/darwin-x86/bin/../bin/dex2oat \
            --runtime-arg -classpath \
            --runtime-arg hw.dex \
            --instruction-set=x86 \
            --instruction-set-features=default \
            --runtime-arg -Xrelocate
            --host \
            --boot-image=/Users/simon/Scratch/art/out/host/darwin-x86/bin/../framework/core.art \
            --dex-file=/Users/simon/Scratch/xper/art/hw.dex \
            --oat-fd=3 \
            --oat-location=/Users/simon/Scratch/xper/art/android-data24863/dalvik-cache/x86/Users@simon@Scratch@xper@art@hw.dex
    dex2oat I 24873 35749 art/dex2oat/dex2oat.cc:284] dex2oat took 246.341ms (threads: 8)
    Hello World courtesy of the Android Runtime (ART)
    art I 24871 35716 art/runtime/native/java_lang_Runtime.cc:41] System.exit called, status: 0

And there you have it. Possibly the longest and most convoluted way of getting a Java ‘hello world’ program to run short of writing your own Java runtime.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

Building The Android Runtime (ART) For Mac OS X: Part Seven — N’th Time Lucky

Obliterating the entire out directory and restarting the build for the n’th time results in the now familiar litany of compiler
warnings but eventually the final invocation of dex2oat completes

    ...
    
    
    host dex2oat: out/host/darwin-x86/framework/x86/core.art \
        (out/host/darwin-x86/framework/core-libart-hostdex.jar \
         out/host/darwin-x86/framework/conscrypt-hostdex.jar \
         out/host/darwin-x86/framework/okhttp-hostdex.jar \
         out/host/darwin-x86/framework/core-junit-hostdex.jar \
         out/host/darwin-x86/framework/bouncycastle-hostdex.jar \
         art/runtime/oat.cc \
         art/runtime/image.cc)
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1266] out/host/darwin-x86/bin/dex2oatd \
        --runtime-arg -Xms64m \
        --runtime-arg -Xmx64m \
        --image-classes=frameworks/base/preloaded-classes \
        --dex-file=out/host/common/obj/JAVA_LIBRARIES/core-libart-hostdex_intermediates/javalib.jar \
        --dex-file=out/host/common/obj/JAVA_LIBRARIES/conscrypt-hostdex_intermediates/javalib.jar \
        --dex-file=out/host/common/obj/JAVA_LIBRARIES/okhttp-hostdex_intermediates/javalib.jar \
        --dex-file=out/host/common/obj/JAVA_LIBRARIES/core-junit-hostdex_intermediates/javalib.jar \
        --dex-file=out/host/common/obj/JAVA_LIBRARIES/bouncycastle-hostdex_intermediates/javalib.jar \
        --dex-location=out/host/darwin-x86/framework/core-libart-hostdex.jar \
        --dex-location=out/host/darwin-x86/framework/conscrypt-hostdex.jar \
        --dex-location=out/host/darwin-x86/framework/okhttp-hostdex.jar \
        --dex-location=out/host/darwin-x86/framework/core-junit-hostdex.jar \
        --dex-location=out/host/darwin-x86/framework/bouncycastle-hostdex.jar \
        --oat-file=out/host/darwin-x86/framework/x86/core.oat \
        --oat-location= \
        --image=out/host/darwin-x86/framework/x86/core.art \
        --base=0x60000000 --instruction-set=x86 \
        --instruction-set-features= \
        --host \
        --android-root=out/host/darwin-x86 \
        --include-patch-information
    dex2oatd I 47906 155398 art/runtime/gc/heap.cc:2182] \
        Explicit concurrent mark sweep GC freed 15367(1476KB) AllocSpace objects, 0(0B) LOS objects, 43% free, 5MB/9MB, paused 83.421ms total 185.818ms
    dex2oatd I 47906 155398 art/runtime/gc/heap.cc:2182] \
        Explicit concurrent mark sweep GC freed 34415(2MB) AllocSpace objects, 0(0B) LOS objects, 56% free, 3MB/7MB, paused 62.147ms total 133.061ms
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511] compiler [Exclusive time] [Total time]
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]   0.579s/11.173s dex2oat Setup
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.238s LoadImageClasses
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.130s Resolve Types
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.029s Resolve MethodsAndFields
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.007s Resolve Types
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.008s Resolve MethodsAndFields
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.024s Resolve Types
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.002s Resolve MethodsAndFields
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.002s Resolve Types
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0s Resolve MethodsAndFields
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.081s Resolve Types
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.006s Resolve MethodsAndFields
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.385s Verify Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.047s Verify Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.039s Verify Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.004s Verify Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.185s Verify Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.224s InitializeNoClinit
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.022s InitializeNoClinit
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.001s InitializeNoClinit
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0s InitializeNoClinit
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.143s InitializeNoClinit
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.065s UpdateImageClasses
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     1.538s Compile Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.124s Compile Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.167s Compile Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.013s Compile Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0.654s Compile Dex File
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     0s/0.213s dex2oat OatWriter
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0s InitOatHeader
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0s InitOatDexFiles
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0s InitDexFiles
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0.022s InitOatClasses
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0.028s InitOatMaps
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0s InitOatCode
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]       0.162s InitOatCodeDexFiles
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     3.497s Writing ELF
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]     2.732s dex2oat ImageWriter
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511] compiler: end, 11.173s
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:1511]
    dex2oatd I 47906 155398 art/dex2oat/dex2oat.cc:284] dex2oat took 11.365s (threads: 8)
    
    \e[0;32m#### make completed successfully (01:16:12 (hh:mm:ss)) ####\e[00m

and that’s it.

Products

After all that what have we got ?

Directory: out/host/darwin-x86/bin

  • acp

  • art

  • dalvikvm -> dalvikvm64

  • dalvikvm32

  • dalvikvm64

  • dex2oat

  • dex2oatd

  • dx

  • oatdump

  • oatdumpd

  • patchoat

  • patchoatd

Directory: out/host/darwin-x86/framework/{x86,x86_64}

  • core.art

  • core.oat

Directory: out/host/darwin-x86/{lib,lib64}

  • libart-compiler.dylib

  • libart.dylib

  • llibartd-compiler.dylib

  • llibartd.dylib

  • libbacktrace_libc++.dylib

  • libc++.dylib

  • libcrypto-host.dylib

  • libexpat-host.dylib

  • libicui18n-host.dylib

  • libicuuc-host.dylib

  • libjavacore.dylib

  • liblog.dylib

  • libnativebridge.dylib

  • libnativehelper.dylib

  • libsigchain.dylib

  • libz-host.dylib


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 29, 2014

Building The Android Runtime (ART) For Mac OS X: Part Six — Once More With Feeling

Removing the dex2oat and dex2oatd executables from the directory

    out/host/darwin-x86/bin

and the object files from

    out/host/darwin-x86/obj/EXECUTABLES/dex2oat_intermediates

and

    out/host/darwin-x86/obj/EXECUTABLES/dex2oatd_intermediates

and restarting the build results in dex2oat getting rebuilt as a 32-bit executable.

As expected, at the point dex2oat is first invoked in the build it no longer spins, instead, and not as expected, it crashes.

    ...

    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] Fatal signal 10 (SIGBUS), code 2 (BUS_ADRERR) fault addr 0x2fb0ed6
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] OS: Darwin 13.4.0 (x86_64)
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] Cmdline: out/host/darwin-x86/bin/dex2oatd ...
    
    ...
    
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] Thread: 72220 "<unknown>"
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] Registers:
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299]     eax: 0x7eadf38a    ebx: 0x7da400e4    ecx: 0x00000076    edx: 0x00000b10
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299]     edi: 0x00000012    esi: 0x02fb0ed6    ebp: 0xbff54578    esp: 0xbff5456c
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299]     eip: 0x002db0e1                    eflags: 0x00010202 [ IF ]
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299]      cs: 0x0000001b     ds: 0x00000023     es: 0x00000023     fs: 0x0000001f
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299]      gs: 0x0000000f     ss: 0x00000023
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299] Backtrace:
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:299]
    dex2oatd F 24874 72220 art/runtime/runtime_linux.cc:313] Fault message:
    make: *** [out/host/darwin-x86/framework/x86_64/core.art] Error 1

Following the way of the printf once more reveals that the crash occurs in this function

File: $(ANDROID_SRC)/art/runtime/mem_map.cc

    ...
    
    static bool ContainedWithinExistingMap(uintptr_t begin,
                                           uintptr_t end,
                                           std::string* error_msg) {
      std::unique_ptr<BacktraceMap> map(BacktraceMap::Create(getpid(), true));
      if (map.get() == nullptr) {
        *error_msg = StringPrintf("Failed to build process map");
        return false;
      }
      for (BacktraceMap::const_iterator it = map->begin(); it != map->end(); ++it) {
        if ((begin >= it->start && begin < it->end)  // start of new within old
            && (end > it->start && end <= it->end)) {  // end of new within old
          return true;
        }
      }
      std::string maps;
      ReadFileToString("/proc/self/maps", &maps);
      *error_msg = StringPrintf("Requested region 0x%08" PRIxPTR "-0x%08" PRIxPTR " does not overlap "
                                "any existing map:\n%s\n",
                                begin, end, maps.c_str());
      return false;
    }
    
    ...

While this is clearly a very Mac OS X unfriendly function, it crashes a bit before the point where it might attempt to read the non-existent ‘file’

    /proc/self/maps

In fact it crashes at the point at which the for loop initialization is doing

      BacktraceMap::const_iterator it = map->begin();

The problem looks as though it might involve death by some combination of temporary variable/assignment operator/copy constructor C++ voodoo and the compiler getting horribly confused or something.

Various attempts at re-factoring to induce some non-crashing combination of temporary variable/assignment operator/copy constructor C++ voodoo all failed, so I tried this

    ...
    
    static bool ContainedWithinExistingMap(uintptr_t begin,
                                           uintptr_t end,
                                           std::string* error_msg) {
      std::unique_ptr<BacktraceMap> map(BacktraceMap::Create(getpid(), true));
      if (map.get() == nullptr) {
        *error_msg = StringPrintf("Failed to build process map");
        return false;
      }
    
      const backtrace_map_t* entry = map->Find(begin);
    
      if ((entry != NULL) && (end <= entry->end)) {
        return true;
      }

      std::string maps;
      ReadFileToString("/proc/self/maps", &maps);
      *error_msg = StringPrintf("Requested region 0x%08" PRIxPTR "-0x%08" PRIxPTR " does not overlap "
                                "any existing map:\n%s\n",
                                begin, end, maps.c_str());
      return false;
    }
    
    ...

which works.

Quite why it works I don’t know, since the BacktraceMap::Find method is defined like this

File: $(ANDROID_SRC)/system/core/libbacktrace/BacktraceMap.cpp

    ...
    
    const backtrace_map_t* BacktraceMap::Find(uintptr_t addr) {
      for (BacktraceMap::const_iterator it = begin();
           it != end(); ++it) {
        if (addr >= it->start && addr < it->end) {
          return &*it;
        }
      }
      return NULL;
    }
    
    ...

It too is using an iterator in exactly the same way. It just doesn’t crash when it does so.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 28, 2014

Building The Android Runtime (ART) For Mac OS X: Part Five — 64 Bits Considered 32 Bits Too Many

As we have seen, if mmap is not passsed the

    MAP_FIXED

flag, then, if the request is for memory somewhere in the low 4GB of the address space although the call succeeds the resulting memory is not in the low 4GB at all.

Setting the MAP_FIXED flag does not help. The call to mmap simply fails.

In short, the problem is that Mac OS X will not allocate memory in the low 4GB of the address space of a 64-bit executable, irrespective of the start address.

The fact that the low_4gb argument to the MemMap::MapAnonymous method is hard-wired to true when called from the Heap::MapAnonymousPreferredAddress method implies that this is an absolute requirement in this case. A possible implication of this is that the some or all of the code is written on the assumption that addresses are necessarily 32 bits so relaxing the constraint is probably not a good idea.

A quick check shows that other calls MemMap::MapAnonymous also pass true as the low_4gb argument, so those calls would also have to be changed as well.

Fortunately there is a workaround of sorts.

One way to get the OS to allocate memory in the first 4GB of the address space of a process is to ensure that the process only has a 4GB address space, i.e, turn dex2oat into a 32-bit executable.

There is probably a ‘proper’ way to do this, but the brute force way is to change this

    ...
        
    ifeq ($(ART_BUILD_HOST_NDEBUG),true)
      $(eval $(call build-art-executable,dex2oat,$(DEX2OAT_SRC_FILES),libart-compiler,art/compiler,host,ndebug))
    endif
    ifeq ($(ART_BUILD_HOST_DEBUG),true)
      $(eval $(call build-art-executable,dex2oat,$(DEX2OAT_SRC_FILES),libartd-compiler,art/compiler,host,debug))
    endif
        
    ...

in the file

    $(ANDROID_SRC)/art/dex2oat/Android.mk

to this

    ...
        
    ifeq ($(ART_BUILD_HOST_NDEBUG),true)
      $(eval $(call build-art-executable,dex2oat,$(DEX2OAT_SRC_FILES),libart-compiler,art/compiler,host,ndebug, 32))
    endif
    ifeq ($(ART_BUILD_HOST_DEBUG),true)
      $(eval $(call build-art-executable,dex2oat,$(DEX2OAT_SRC_FILES),libartd-compiler,art/compiler,host,debug, 32))
    endif
        
    ...

Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 27, 2014

Building The Android Runtime (ART) For Mac OS X: Part Four — MemMap::MapAnonymous

File: $(ANDROID_SRC)/art/runtime/mem_map.cc

Method: MemMap::MapAnonymous

    ...
    
    MemMap* MemMap::MapAnonymous(const char* name, byte* expected_ptr, size_t byte_count, int prot,
      bool low_4gb, std::string* error_msg) {
      if (byte_count == 0) {
        return new MemMap(name, nullptr, 0, nullptr, 0, prot, false);
      }
      size_t page_aligned_byte_count = RoundUp(byte_count, kPageSize);
    
      int flags = MAP_PRIVATE | MAP_ANONYMOUS;
      ScopedFd fd(-1);
    
    #ifdef USE_ASHMEM
    #ifdef HAVE_ANDROID_OS
      const bool use_ashmem = true;
    #else
      // When not on Android ashmem is faked using files in /tmp. Ensure that such files won't
      // fail due to ulimit restrictions. If they will then use a regular mmap.
      struct rlimit rlimit_fsize;
      CHECK_EQ(getrlimit(RLIMIT_FSIZE, &rlimit_fsize), 0);
      const bool use_ashmem = (rlimit_fsize.rlim_cur == RLIM_INFINITY) ||
         (page_aligned_byte_count < rlimit_fsize.rlim_cur);
    #endif
      if (use_ashmem) {
        // android_os_Debug.cpp read_mapinfo assumes all ashmem regions associated with the VM are
        // prefixed "dalvik-".
        std::string debug_friendly_name("dalvik-");
        debug_friendly_name += name;
        fd.reset(ashmem_create_region(debug_friendly_name.c_str(), page_aligned_byte_count));
        if (fd.get() == -1) {
          *error_msg = StringPrintf("ashmem_create_region failed for '%s': %s", name, strerror(errno));
          return nullptr;
        }
        flags = MAP_PRIVATE;
      }
    #endif
    
      // We need to store and potentially set an error number for pretty printing of errors
      int saved_errno = 0;
    
    #ifdef __LP64__
      // When requesting low_4g memory and having an expectation, the requested range should fit into
      // 4GB.
      if (low_4gb && (
          // Start out of bounds.
          (reinterpret_cast<uintptr_t>(expected_ptr) >> 32) != 0 ||
          // End out of bounds. For simplicity, this will fail for the last page of memory.
          (reinterpret_cast<uintptr_t>(expected_ptr + page_aligned_byte_count) >> 32) != 0)) {
        *error_msg = StringPrintf("The requested address space (%p, %p) cannot fit in low_4gb",
        expected_ptr, expected_ptr + page_aligned_byte_count);
        return nullptr;
      }
    #endif
    
      // TODO:
      // A page allocator would be a useful abstraction here, as
      // 1) It is doubtful that MAP_32BIT on x86_64 is doing the right job for us
      // 2) The linear scheme, even with simple saving of the last known position, is very crude
    #if USE_ART_LOW_4G_ALLOCATOR
      // MAP_32BIT only available on x86_64.
      void* actual = MAP_FAILED;
      if (low_4gb && expected_ptr == nullptr) {
        bool first_run = true;
    
        for (uintptr_t ptr = next_mem_pos_; ptr < 4 * GB; ptr += kPageSize) {
          if (4U * GB - ptr < page_aligned_byte_count) {
            // Not enough memory until 4GB.
            if (first_run) {
              // Try another time from the bottom;
              ptr = LOW_MEM_START - kPageSize;
              first_run = false;
              continue;
            } else {
              // Second try failed.
              break;
            }
          }
    
          uintptr_t tail_ptr;
    
          // Check pages are free.
          bool safe = true;
          for (tail_ptr = ptr; tail_ptr < ptr + page_aligned_byte_count; tail_ptr += kPageSize) {
            if (msync(reinterpret_cast<void*>(tail_ptr), kPageSize, 0) == 0) {
              safe = false;
              break;
            } else {
              DCHECK_EQ(errno, ENOMEM);
            }
          }
    
          next_mem_pos_ = tail_ptr;  // update early, as we break out when we found and mapped a region
    
          if (safe == true) {
            actual = mmap(reinterpret_cast<void*>(ptr), page_aligned_byte_count, prot, flags, fd.get(),
                          0);
            if (actual != MAP_FAILED) {
              // Since we didn't use MAP_FIXED the kernel may have mapped it somewhere not in the low
              // 4GB. If this is the case, unmap and retry.
              if (reinterpret_cast<uintptr_t>(actual) + page_aligned_byte_count < 4 * GB) {
                break;
              } else {
                munmap(actual, page_aligned_byte_count);
                actual = MAP_FAILED;
              }
            }
          } else {
            // Skip over last page.
            ptr = tail_ptr;
          }
        }
    
        if (actual == MAP_FAILED) {
          LOG(ERROR) << "Could not find contiguous low-memory space.";
          saved_errno = ENOMEM;
        }
      } else {
        actual = mmap(expected_ptr, page_aligned_byte_count, prot, flags, fd.get(), 0);
        saved_errno = errno;
      }
    
    #else
    #if defined(__LP64__)
      if (low_4gb && expected_ptr == nullptr) {
        flags |= MAP_32BIT;
      }
    #endif
    
      void* actual = mmap(expected_ptr, page_aligned_byte_count, prot, flags, fd.get(), 0);
      saved_errno = errno;
    #endif
    
      if (actual == MAP_FAILED) {
        std::string maps;
        ReadFileToString("/proc/self/maps", &maps);
    
        *error_msg = StringPrintf("Failed anonymous mmap(%p, %zd, 0x%x, 0x%x, %d, 0): %s\n%s",
                                  expected_ptr, page_aligned_byte_count, prot, flags, fd.get(),
                                  strerror(saved_errno), maps.c_str());
        return nullptr;
      }
      std::ostringstream check_map_request_error_msg;
      if (!CheckMapRequest(expected_ptr, actual, page_aligned_byte_count, error_msg)) {
        return nullptr;
      }
      return new MemMap(name, reinterpret_cast<byte*>(actual), byte_count, actual,
                        page_aligned_byte_count, prot, false);
    }
    
    ...

Although it is not entirely obvious, amongst the welter of #ifdefs and assorted detritus, there is a for loop hiding in the method.

It is only compiled in if

    USE_ART_LOW_4G_ALLOCATOR

is defined and not 0.

The definition, or otherwise, of USE_ART_LOW_4G_ALLOCATOR occurs in the file

    art/runtime/mem_map.h

and is itself conditional

    ...
    
    #if defined(__LP64__) && (!defined(__x86_64__) || defined(__APPLE__))
    #define USE_ART_LOW_4G_ALLOCATOR 1
    #else
    #define USE_ART_LOW_4G_ALLOCATOR 0
    #endif
    
    ...

so on a 64-bit Mac it will be defined as 1, so the loop is present,

The for loop is inside the if statement

      ...
    
      if (low_4gb && expected_ptr == nullptr) {
      
        ...
        
      }
    
      ...

where

    low_4gb

and

    expected_ptr

are both arguments to the method.

Looking at the caller in this case

File: $(ANDROID_SRC)/art/runtimegc/heap.cc

Method: Heap::MapAnonymousPreferredAddress

    ...
    
    MemMap* Heap::MapAnonymousPreferredAddress(const char* name, byte* request_begin, size_t capacity,
                                               int prot_flags, std::string* out_error_str) {
      while (true) {
        MemMap* map = MemMap::MapAnonymous(kMemMapSpaceName[0], request_begin, capacity,
                                           PROT_READ | PROT_WRITE, true, out_error_str);
        if (map != nullptr || request_begin == nullptr) {
          return map;
        }
        // Retry a  second time with no specified request begin.
        request_begin = nullptr;
      }
      return nullptr;
    }
    
    ...

we can see that the

    low_4gb

argument is always true, and the

    expected_ptr

argument is the value of the method’s

    request_begin

argument, or

    nullptr

so one way or another the MemMap::MapAnonymous method can definitely end up in the loop.

What the method is trying to do in the loop is get the OS to allocate a continuous area of memory of a given size somewhere in the low 4GB of the process address space, starting at the 64K mark and working its way upwards a ‘page’ at a time.

The worst case outcome is that the necessary memory is not available. In this case the loop will iterate over the entire address space, so although
it will terminate it will not do so for a very long time.

Here is a simplified version of the loop in the form of a standalone program.


    static constexpr int kPageSize = 4096;

    static constexpr size_t KB = 1024;
    static constexpr size_t GB = KB * KB * KB;

    static constexpr uintptr_t LOW_MEM_START = 64 * KB;

    int main(int argc, const char * argv[])
    {
        void*     actual = MAP_FAILED;
        size_t    nBytes = 16 * kPageSize;

        for (uintptr_t ptr = LOW_MEM_START; ptr < 4 * GB; ptr += kPageSize)
        {
            uintptr_t tail_ptr;
    
            // Check pages are free.
            bool safe = true;
    
            for (tail_ptr = ptr; tail_ptr < ptr + nBytes; tail_ptr += kPageSize)
            {
                if (msync(reinterpret_cast<void*>(tail_ptr), kPageSize, 0) == 0)
                {
                    safe = false;
                    break;
                }
                else
                if (errno == ENOMEM)
                {
                    // not in use
                }
                else
                {
                    // some other error
                    return 1;
                }
            }
            if (safe == true)
            {
                actual = mmap(reinterpret_cast<void*>(ptr), nBytes, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);
                if (actual != MAP_FAILED)
                {
                    fprintf(stdout, "actual == %p\n", actual);
                    // Since we didn't use MAP_FIXED the kernel may have mapped it somewhere not in the low
                    // 4GB. If this is the case, unmap and retry.
                    if (reinterpret_cast<uintptr_t>(actual) + nBytes < 4 * GB)
                    {
                        break;
                    }
                    else
                    {
                        munmap(actual, nBytes);
                        actual = MAP_FAILED;
                    }
                }
            }
            else
            {
                // Skip over last page.
                ptr = tail_ptr;
            }
        }
        return 0;
    }

and here’s what it prints when it is run

    actual == 0x10cbf5000
    actual == 0x10cbf5000
    actual == 0x10cbf5000
    actual == 0x10cbf5000
    actual == 0x10cbf5000
    actual == 0x10cbf5000
    actual == 0x10cbf5000
    actual == 0x10cbf5000
    
    ... 

effectively forever, or certainly for a lot longer than I am prepared to wait for it to iterate approximately 4G/4K times then terminate.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 26, 2014

Building The Android Runtime (ART) For Mac OS X: Part Three — Let’s Build ! Again !

Restarting the build after the modification to $(ANDROID_SRC)/art/Android.mk results in another unhealthy number of compiler warnings followed, eventually, by this (reformatted for clarity)

    ...
    
    dex2oatd I  2426 823971 art/dex2oat/dex2oat.cc:1266] out/host/darwin-x86/bin/dex2oatd \
       --runtime-arg -Xms64m \
       --runtime-arg -Xmx64m \
       --image-classes=frameworks/base/preloaded-classes \
       --dex-file=out/host/common/obj/JAVA_LIBRARIES/core-libart-hostdex_intermediates/javalib.jar \
       --dex-file=out/host/common/obj/JAVA_LIBRARIES/conscrypt-hostdex_intermediates/javalib.jar \
       --dex-file=out/host/common/obj/JAVA_LIBRARIES/okhttp-hostdex_intermediates/javalib.jar \
       --dex-file=out/host/common/obj/JAVA_LIBRARIES/core-junit-hostdex_intermediates/javalib.jar \
       --dex-file=out/host/common/obj/JAVA_LIBRARIES/bouncycastle-hostdex_intermediates/javalib.jar \
       --dex-location=out/host/darwin-x86/framework/core-libart-hostdex.jar \
       --dex-location=out/host/darwin-x86/framework/conscrypt-hostdex.jar \
       --dex-location=out/host/darwin-x86/framework/okhttp-hostdex.jar \
       --dex-location=out/host/darwin-x86/framework/core-junit-hostdex.jar \
       --dex-location=out/host/darwin-x86/framework/bouncycastle-hostdex.jar \
       --oat-file=out/host/darwin-x86/framework/x86_64/core.oat \
       --oat-location= \
       --image=out/host/darwin-x86/framework/x86_64/core.art \
       --base=0x60000000 \
       --instruction-set=x86_64 \
       --instruction-set-features= \
       --host \
       --android-root=out/host/darwin-x86 \
       --include-patch-information

It all goes ominously quiet until you get this

    dex2oatd W 2426 823972 dex2oat did not finish after 300 seconds

and then, after an interminable wait, you get this

    dex2oatd F 2426 823972 dex2oat did not finish after 1800 seconds
    make: *** [out/host/darwin-x86/framework/x86_64/core.art] Error 1

and everything stops.

Obviously dex2oatd is very busy doing something or other and it never gets a chance to finish, except that if you look at the process stats during the interminable wait for something to happen, it isn’t. The CPU load is very light and nothing else is happening.

It looks in fact as though dex2oatd is simply spinning and doing no work at all.

Using the arcane debugging technique known to those of us who worked on the OS formerly known as S*****n as The Way Of The printf reveals that the dex2oatd process appears to be sat in the method

    MemMap::MapAnonymous

which has been called from the method

    Heap::MapAnonymousPreferredAddress

which in turn has been called from the constructor

    Heap::Heap

which in turn has been called from

    Runtime::Init

In short, dex2oatd hasn’t even started, its just spent 1800 seconds attempting to initialize its Runtime.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

September 25, 2014

Building The Android Runtime (ART) For Mac OS X: Part Two — Let’s Build !

To do the build

One

Copy the necessary Android ‘source’ to a directory ($(ANDROID_SRC)) on the case-sensitive file system.

Two

Set the environment variable

    DEVELOPER_DIR

to the path to the Developer directory within the version of the Xcode.app to use to do the build, for example

    export DEVELOPER_DIR=/Users/simon/Dev/xcode/Xcode.app/Contents/Developer

Three

Set the environment variable JAVA_HOME to the Home directory of the version of JDK 7 to use to do the build, for example

    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home

Four

Change to the directory containing the Android source and ‘source’ the file build/envsetup.sh to set up the Android build environment

    . build/envsetup.sh

Five

Start the build

    m build-art-host

What Happens Next ?

Eventually, after an unhealthy number of compiler warnings, this happens

    make: *** No rule to make target `out/host/darwin-x86/lib/libjavacore.so', needed by `build-art-host'.  Stop.

A quick check reveals that there is a

    libjavacore.dylib

in

    out/host/darwin-x86/lib

but not a

    libjavacore.so

which is what you would expect on Mac OS X.

Searching for references to libjavacore.so in the art build system turns up

    ...
    
    # ART_HOST_DEPENDENCIES depends on Android.executable.mk above for ART_HOST_EXECUTABLES
    ART_HOST_DEPENDENCIES := \
	    $(ART_HOST_EXECUTABLES) \
	    $(HOST_OUT_JAVA_LIBRARIES)/core-libart-hostdex.jar \
	    $(ART_HOST_OUT_SHARED_LIBRARIES)/libjavacore$(ART_HOST_SHLIB_EXTENSION)
    ART_TARGET_DEPENDENCIES := \
	    $(ART_TARGET_EXECUTABLES) \
	    $(TARGET_OUT_JAVA_LIBRARIES)/core-libart.jar \
	    $(TARGET_OUT_SHARED_LIBRARIES)/libjavacore.so
    ifdef TARGET_2ND_ARCH
    ART_TARGET_DEPENDENCIES += $(2ND_TARGET_OUT_SHARED_LIBRARIES)/libjavacore.so
    endif
    ifdef HOST_2ND_ARCH
    ART_HOST_DEPENDENCIES += $(2ND_HOST_OUT_SHARED_LIBRARIES)/libjavacore.so
    endif
    
    ...

in the file

    $(ANDROID_SRC)/art/Android.mk

It looks as though

    ...
    
    ifdef HOST_2ND_ARCH
    ART_HOST_DEPENDENCIES += $(2ND_HOST_OUT_SHARED_LIBRARIES)/libjavacore.so
    endif
    
    ...

should probably be

    ...
    
    ifdef HOST_2ND_ARCH
    ART_HOST_DEPENDENCIES += $(2ND_HOST_OUT_SHARED_LIBRARIES)/libjavacore$(ART_HOST_SHLIB_EXTENSION)
    endif
    
    ...

Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

Building The Android Runtime (ART) For Mac OS X: Part One — Prerequisites

Mac OS X

I am using Mac OS X 10.9.5.

I am not sure whether the exact version of the OS matters.

Xcode And Xcode Command Line Tools

To build am using Xcode 5.1 plus the corresponding command line tools.

JDK 7

To build I am using JDK 7. The version reported by java -version is

    1.7.0_45

It has to be JDK 7.

A Case Sensitive File System

You will need a case-sensitive file system which probably isn’t what you have by default, so you will need an external drive, a USB stick or possibly a .dmg that has been formatted appropriately.

I have successfully used an external drive and a USB stick, both formatted as

    Mac OS Extended(Case-sensitive, Journaled)

I haven’t tried the .dmg approach.

Some Android Source

I am using the the following specific subset of the Android ‘source’ all obtained within the last couple of days courtesy
of repo sync.

  • abi

  • art

  • bionic

  • build

  • dalvik

  • external/bouncycastle

  • external/conscrypt

  • external/expat

  • external/fdlibm

  • external/gtest

  • external/icu

  • external/junit

  • external/libcxx

  • external/libcxxabi

  • external/libphonenumber

  • external/llvm

  • external/okhttp

  • external/openssl

  • external/stlport

  • external/valgrind

  • external/vixl

  • external/zlib

  • frameworks/base

  • frameworks/compile/mclinker

  • frameworks/native

  • libcore

  • libnativehelper

  • prebuilts/clang/darwin-x86/host/3.5

  • prebuilts/misc

  • prebuilts/sdk

  • system/core

I think this is the absolute minimum subset you need to build with.

I assume it would also work if you simply have the entire source tree but I haven’t tried it.


Copyright (c) 2014 By Simon Lewis. All Rights Reserved.

Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and owner Simon Lewis is strictly prohibited.

Excerpts and links may be used, provided that full and clear credit is given to Simon Lewis and justanapplication.wordpress.com with appropriate and specific direction to the original content.

Older Posts »

The WordPress Classic Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: