GC on Android SIGSEGV

I'm trying to build android NativeActivity application with ldc. And I have strange errors with GC

-mtriple=i686-none-linux-android

I/NativeActivity( 2948): _nativeInit
D/NativeActivity( 2948): _jni_nativeInit
D/NativeActivity( 2948): _jni_nativeInit 32 bit
D/NativeActivity( 2948): _android_app_thread
D/NativeActivity( 2948): core.runtime.rt_init
D/NativeActivity( 2948): initSections
D/NativeActivity( 2948): initSections LDC mbeg = 0xa16f96f0, mend = 0xa16faa40
D/NativeActivity( 2948): _jni_nativeInit ok
D/NativeActivity( 2948): initSections LDC pbeg = 0xa16a8a64, pend = 0xb7714658
D/NativeActivity( 2948): initSections LDC pbeg = 0xa16a8a64, pend = 0xb7714658
I/NativeActivity( 2948): _nativeInit ok
I/NativeActivity( 2948): WebViewNativeActivity.onCreate ok
D/NativeActivity( 2948): initTLSRanges

here i'm trying to allocate 25 bytes
char[1024] buf;
....
return buf[0 .. len].idup; // len = 25

on idup it crashes

D/NativeActivity( 2948): _d_newarrayU(length = x19, size = 1)
D/NativeActivity( 2948): __arrayAlloc size = 25
D/NativeActivity( 2948): __arrayAlloc typeInfoSize = 0
D/NativeActivity( 2948): __arrayAlloc padSize = 1
D/NativeActivity( 2948): __arrayAlloc padded_size = 26
D/NativeActivity( 2948): __arrayAlloc GC.qalloc padded_size = 26
D/NativeActivity( 2948): GC.qalloc sz = 26
D/NativeActivity( 2948): gc/proxy/gc_qalloc sz = 26
D/NativeActivity( 2948): gc/impl/proto/gc qalloc 26
D/NativeActivity( 2948): gc_init()
D/NativeActivity( 2948): gc_init() !isInstanceInit
D/NativeActivity( 2948): createGCInstance conservative
D/NativeActivity( 2948): createGCInstance entry.name = conservative ok
D/NativeActivity( 2948): gc/impl/conservative/gc private GC initialize()
D/NativeActivity( 2948): gc/impl/conservative/gc private GC initialize() 1
D/NativeActivity( 2948): gc/impl/conservative/gc private GC initialize() ok
D/NativeActivity( 2948): 0xb3d93f80.Gcx::addRange(0xa16a8a64, 0xb7714658)
D/NativeActivity( 2948): gc/impl/proto/gc qalloc 26
D/NativeActivity( 2948): gc/proxy/gc_qalloc sz = 26
D/NativeActivity( 2948): conservative/gc/qalloc 26
D/NativeActivity( 2948): GC::malloc(gcx = 0xb3d93f80, size = 26 bits = a, ti = TypeInfo_a)
D/NativeActivity( 2948): Gcx.fullcollect()
D/NativeActivity( 2948): startScanThreads: 1 threads per CPU
D/NativeActivity( 2948): preparing mark.
D/NativeActivity( 2948): 	collect stacks.
D/NativeActivity( 2948): collectRoots
D/NativeActivity( 2948): collectRoots pbot = 0xa0ff4050, ptop = 0xa0ff9000
D/NativeActivity( 2948): collectRoots minAddr = 0x0, memSize = 0
D/NativeActivity( 2948): scanTLSRanges
D/NativeActivity( 2948): collectRoots
D/NativeActivity( 2948): collectRoots pbot = 0xb3fc8100, ptop = 0xb3fc81f8
D/NativeActivity( 2948): collectRoots minAddr = 0x0, memSize = 0
D/NativeActivity( 2948): 	collect roots[]
D/NativeActivity( 2948): 	collect ranges[]
D/NativeActivity( 2948): 		0xa16a8a64 .. 0xb7714658
D/NativeActivity( 2948): collectRoots
D/NativeActivity( 2948): collectRoots pbot = 0xa16a8a64, ptop = 0xb7714658
D/NativeActivity( 2948): collectRoots minAddr = 0x0, memSize = 0
--------- beginning of crash

and when I try to run with -mtriple=aarch64-none-linux-android the result is more strange:
the size of memory 25 changed to 26 (it is ok), bun then it changed to 10???, and then size = -309990416 (in function conservative/gc/qalloc). Why???

D/NativeActivity: _d_newarrayU(length = x19, size = 1)
D/NativeActivity: __arrayAlloc size = 25
D/NativeActivity: __arrayAlloc typeInfoSize = 0
D/NativeActivity: __arrayAlloc padSize = 1
D/NativeActivity: __arrayAlloc padded_size = 26
D/NativeActivity: __arrayAlloc GC.qalloc padded_size = 26
D/NativeActivity: GC.qalloc sz = 26
D/NativeActivity: gc/proxy/gc_qalloc sz = 26
D/NativeActivity: gc/impl/proto/gc qalloc 10
D/NativeActivity: gc_init()
D/NativeActivity: gc_init() !isInstanceInit
D/NativeActivity: createGCInstance conservative
D/NativeActivity: createGCInstance entry.name = conservative ok
D/NativeActivity: gc/impl/conservative/gc private GC initialize()
D/NativeActivity: gc/impl/conservative/gc private GC initialize() ok
D/NativeActivity: gc/impl/proto/gc qalloc 10
D/NativeActivity: gc/proxy/gc_qalloc sz = 10
D/NativeActivity: conservative/gc/qalloc -309990416, _add = -320560320

With -mtriple=armv7-none-linux-androideabi all works good without errors.

If I turn off the GC by adding in code
extern(C) pragma(mangle, "rt_options") __gshared string[] rt_options = [ "gcopt=disable:1" ];
all works good on all android targets.

I test it in Android studio emulator, Android 7.0, API 24

[kinke]

This should already be fixed in branch merge-2.091 master by #3332, at least the AArch64 issue.

Do you have a D main in the first object file fed to the linker? (see https://wiki.dlang.org/Build_D_for_Android#Changes_for_Android)

Do you have a D main in the first object file fed to the linker?

Yes, this is my build script

$HOME/ldc.2.1.20.0.linux/bin/ldc2
    -c
    -of=$PROJECT/obj/android/testapp-aarch64.o
    --d-debug
    -g
    -w
    -I=$FRAMEWORK/src
    -i=framework
    -I=$PROJECT/src
    -i=testapp
    -mtriple=aarch64-none-linux-android
    --d-version=Application
    --d-version=AndroidNativeActivity
    $PROJECT/src/testapp/main.d

$HOME/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
    -o $PROJECT/obj/android/lib/arm64-v8a/libtestapp.so
    -shared
    -Wl,-soname,libtestapp.so
    -fuse-ld=bfd
    -target aarch64-none-linux-android
    -gcc-toolchain $HOME/android-ndk-r21/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64
    -Wl,--fix-cortex-a53-835769
    -Wl,--fix-cortex-a53-843419
    -no-canonical-prefixes
    -Wl,--gc-sections
    -Wl,--no-undefined
    -Wl,-z,nocopyreloc
    -Wl,-z,noexecstack
    -Wl,-z,relro
    -Wl,-z,now
    -Wl,--warn-shared-textrel
    -Wl,--fatal-warnings
    --sysroot=$HOME/android-ndk-r21/platforms/android-21/arch-arm64
    $PROJECT/obj/android/testapp-aarch64.o
    $FRAMEWORK/c/tables/tables.android-arm64.o
    $HOME/ldc.runtime.android-arm64/lib/libdruntime-ldc-debug.a
    $FRAMEWORK/c/zlib/libz.android-aarch64.a
    -lc
    -lm
    -ldl
    -llog
    -lgcc
    -landroid
    $FRAMEWORK/c/pcre-8.42/libpcre.android-aarch64.a
    $PROJECT/obj/android/res-aarch64/res.o

I build ldc and ldc runtime from master, and now i have an error:

ld.bfd: _tlsstart: TLS definition in testapp-aarch64.o section .tdata mismatches non-TLS reference in ldc.runtime.android-arm64/lib/libdruntime-ldc-debug.a(sections_android.o)

file ldc/driver/codegenerator.cpp placed this symbol into ".tdata" segment

[kinke]

Do you use an official prebuilt package from GitHub? You need our forked LLVM for Android TLS emulation.

[kinke]

(You can also use a prebuilt CI package: https://github.com/ldc-developers/ldc/releases/tag/CI.)

You need our forked LLVM for Android TLS emulation.

I rebuilded last version LDC from git with this LLVM.

-mtriple=armv7-none-linux-androideabi - ok
-mtriple=aarch64-none-linux-android - now ok thank you, it works!
-mtriple=i686-none-linux-android - still failed with SIGSEGV
-mtriple=x86_64-none-linux-android - still failed with SIGSEGV

call stack for x86_64:

_D2gc4impl12conservativeQw3Gcx12collectRootsMFNbNlPvQcZv+102
_D2gc4impl12conservativeQw3Gcx15collectAllRootsMFNbbZ14__foreachbody3MFNbKS4coreQDa11gcinterface5RangeZi+46
_D2rt4util9container5treap__T5TreapTS4core2gc11gcinterface5RangeZQBk7opApplyMFNbMDFNbKQByZiZ9__lambda2MFNbKxSQCuQCsQCsQCiZi+12
_D2rt4util9container5treap__T5TreapTS4core2gc11gcinterface5RangeZQBk13opApplyHelperFNbxPSQDjQDjQDhQDa__TQCxTQCuZQDf4NodeMDFNbKxSQDnQDlQDlQDbZiZi+103
_D2rt4util9container5treap__T5TreapTS4core2gc11gcinterface5RangeZQBk7opApplyMxFNbMDFNbKxSQCaQByQByQBoZiZi+31
_D2rt4util9container5treap__T5TreapTS4core2gc11gcinterface5RangeZQBk7opApplyMFNbMDFNbKQByZiZi+39
_D2gc4impl12conservativeQw3Gcx15collectAllRootsMFNbbZv+136
_D2gc4impl12conservativeQw3Gcx12markParallelMFNbbZv+117
_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm+352
_D2gc4impl12conservativeQw3Gcx10smallAllocMFNbmKmkxC8TypeInfoZPv+632
_D2gc4impl12conservativeQw3Gcx5allocMFNbmKmkxC8TypeInfoZPv+60
_D2gc4impl12conservativeQw14ConservativeGC12mallocNoSyncMFNbmkKmxC8TypeInfoZPv+76
_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs12mallocNoSyncMFNbmkKmxC8TypeInfoZPvS_DQEgQEgQEeQEp10mallocTimelS_DQFiQFiQFgQFr10numMallocslTmTkTmTxQCzZQFcMFNbKmKkKmKxQDsZQDl+69
_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+145
_DThn16_2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+26
gc_qalloc+76
_D2gc4impl5protoQo7ProtoGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+58
_DThn16_2gc4impl5protoQo7ProtoGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+26
gc_qalloc+76
_D4core6memory2GC6qallocFNaNbmkxC8TypeInfoZSQBqQBo8BlkInfo_+41
_D2rt8lifetime12__arrayAllocFNaNbmxC8TypeInfoxQlZS4core6memory8BlkInfo_+365
_d_newarrayU+208
_D6object__T4_dupTaTyaZQlFNaNbAaZAya+31
_D6object__T11_trustedDupTaTyaZQtFNaNbNeAaZAya+28
_D6object__T4idupTaZQiFNaNbNdNfAaZAya+28

here i'm trying to call "idup"

_D9framework2os6Config25_sharedStaticCtor_L160_C1FZv+38
_D2rt5minfo__T14runModuleFuncsSQBdQBd11ModuleGroup8runCtorsMFZ9__lambda2ZQChMFAxPyS6object10ModuleInfoZv+88
_D2rt5minfo11ModuleGroup8runCtorsMFZv+60
_D2rt5minfo13rt_moduleCtorUZ14__foreachbody1MFKSQBu16sections_android12SectionGroupZi+48
_D2rt16sections_android12SectionGroup7opApplyFMDFKSQBxQBxQBiZiZi+35
rt_moduleCtor+19
rt_init+91
_android_app_thread+220
__pthread_start(void*)+46
__start_thread+11
__bionic_clone+53

[kinke]

Glad it works for AArch64 now at least. - Looks like the GC is scanning an invalid range, probably returned in rt/sections_android.d's initSections() (output of &{_tlsstart,_tlsend,__bss_end__} and comparison against ARM output could be interesting, as well as the failing address). Adding -v to the clang linker cmdliner could also be interesting for the underlying ld cmdline, to see if it does something different for x86 architectures.

Adding -v to the clang linker cmdliner could also be interesting for the underlying ld cmdline

arm/aarch64 linker has only 2 aaditional flags -EL and -X

rt/sections_android.d's initSections() (output of &{_tlsstart,_tlsend,bss_end}

The main difference x86 has no bss_end

I made an interesting observation, when I build console program for i686-none-linux-android, GC works. But when i build shared .so for i686-none-linux-android GC crashes.

initSections log in console application

initSections LDC mbeg = 0xb77cc488, mend = 0xb77cd864
initSections LDC pbeg = 0xb77a7874, pend = 0xb77cdf14

bend - pbeg = 153Kb, i think it is good

initSections log in shared .so with NativeActivity

initSections LDC mbeg = 0xa16f9660, mend = 0xa16fa9b0
initSections LDC pbeg = 0xa16a8c14, pend = 0xb7747658

pend - pbeg = 352Mb, too big to be real.

[kinke]

Looks promising. - It's probably time to revamp sections_android.d and get rid of the dependence on these magic symbols. IIRC, __bss_end__ (for ARM archs) is the reason we need ld.bfd for Android (and e.g. ld.gold doesn't work). And _tls{start,end} are emitted manually by the compiler in the object file containing D main, and that's why it needs to be the first object in the linker cmdline (and why even .so libs need a D main). Joakim wanted to get rid of this and parse the ELF headers instead, but he was waiting for dlang/druntime#2330, which was recently merged (after ~1.5 years) and is already available in our 2.091 branch (https://github.com/ldc-developers/druntime/tree/ldc-merge-2.091/src/core/internal/elf, see #3333).

[kinke]

Could you please refetch master (now based on 2.091), rebuild and then test the following?

import core.internal.elf.dl;
import core.internal.elf.io;
import core.stdc.stdio;
import core.sys.linux.elf;

version (Android)
{
    extern __gshared extern(C)
    {
        version (X86_Any)
        {
            size_t _end;
            alias __bss_end__ = _end;
        }
        else
        {
            size_t __bss_end__;
        }

        int _tlsstart;
        int _tlsend;
    }
}

import ldc.attributes;
@section("mySection") __gshared size_t myGlobal;

void main()
{
    version (Android)
    {
        printf("&_tlsstart:   %p\n", &_tlsstart);
        printf("&_tlsend:     %p\n", &_tlsend);
        printf("&__bss_end__: %p\n", &__bss_end__);
    }

    SharedObject object;

    if (!SharedObject.findForAddress(&myGlobal, object))
        assert(0, "cannot find SharedObject");

    printf("ELF object base: %p\n", object.baseAddress);

    scanSegments(object);
    scanSections(object);

    printf("&myGlobal: %p\n", &myGlobal);
}

void scanSegments(const ref SharedObject object)
{
    foreach (ref phdr; object)
    {
        if (phdr.p_type == PT_LOAD && (phdr.p_flags & PF_W)) // writeable data segment
        {
            auto beg = object.baseAddress + (phdr.p_vaddr & ~(size_t.sizeof - 1));
            printf("data segment range: %p - %p\n", beg, beg + phdr.p_memsz);
        }
    }
}

void scanSections(const ref SharedObject object)
{
    ElfFile file;
    if (!ElfFile.open(object.name().ptr, file))
        assert(0, "cannot open ELF file");

    void checkSection(string name)
    {
        ElfSectionHeader sectionHeader;
        if (file.findSectionHeaderByName(name, sectionHeader))
        {
            auto beg = object.baseAddress + sectionHeader.shdr.sh_addr;
            printf("found section %.*s: %p - %p\n",
                cast(int) name.length, name.ptr, beg, beg + sectionHeader.shdr.sh_size);
        }
    }

    checkSection(".tdata");
    checkSection(".tbss");
    checkSection("mySection");
}

This works in my Linux VM, printing something like:

ELF object base: 0x561d063ac000
data segment range: 0x561d063aed10 - 0x561d063af038
found section mySection: 0x561d063af018 - 0x561d063af020
&myGlobal: 0x561d063af018

-mtriple=i686-none-linux-android, shared .so, NativeActivity

&_tlsstart:   0xa17aca5c
&_tlsend:     0xa17acb54
&__bss_end__: 0xb7757658
ELF object base: 0xa1104000
data segment range: 0xa17aca5c - 0xa17ff6d4
found section .tdata: 0xa17aca5c - 0xa17aca60
found section .tbss: 0xa17aca60 - 0xa17acb58
found section mySection: 0xa17fedcc - 0xa17fedd0
&myGlobal: 0xa17fedcc

variable "program_invocation_name" in file core/internal/elf/dl.d not exists on android, I changed it to __progname

And at line
if (!ElfFile.open(object.name().ptr, file))
function object.name() when I run it as NativeActivity returns only basename of shared library without full path.
I found solution here https://stackoverflow.com/questions/1681060/library-path-when-dynamically-loaded/54764051#54764051

For example my version:

private @nogc nothrow bool __get_this_so_path(char *path)
{
	import core.stdc.string;
	import core.sys.posix.strings;
	import core.sys.posix.libgen;
	import core.sys.linux.dlfcn;

	Dl_info info;
	if (!dladdr(&__tls_get_addr, &info))
	{
		printf("dladdr failed");
		return false;
	}

	if (strrchr(info.dli_fname, '/') != null)
	{
		strcpy(path, info.dli_fname);
		return true;
	}


	FILE *fp = fopen("/proc/self/maps", "r");
	if (!fp) return false;

	enum BUFFER_SIZE = 256;
	char[BUFFER_SIZE] buffer;

	while (fgets(buffer.ptr, BUFFER_SIZE, fp))
	{
		if (sscanf(buffer.ptr, "%*llx-%*llx %*s %*s %*s %*s %s", path) == 1)
		{
			char *bname = basename(path);
			printf("check basename[%s]", bname);
			if (strcasecmp(bname, info.dli_fname) == 0)
			{
				fclose(fp);
				return true;
			}
		}
	}
	fclose(fp);
	return false;
}

[kinke]

Thx for testing, seems to work. Is that enough for you to attempt a sections_android.d PR? [Wrt. resorting to /proc/self/maps, it'd probably be nicer to check the address range to overcome potential issues wrt. duplicate .so names. Also note that the .tdata and .tbss sections might not exist anymore, unlike now, where the _tls{start,end} symbols enforce both sections.]

[kinke]

I've started refactoring stuff in ldc-developers/druntime#178; I'll take a look at sections_android.d tomorrow if you don't beat me to it.