How to link runtime into shared library?

[trivialfis]

Originally the question was posted on SO. I failed to link AOT generated object into a shared object. I extracted a minimum example:

// file: impl.cc
#include <Halide.h>
#include <vector>

namespace ft {

class Test : public Halide::Generator<Test> {
 public:
  Test() : vars(4) {}

  Input<Halide::Buffer<>> lhs{"lhs"};
  Input<Halide::Buffer<>> rhs{"rhs"};
  Output<Halide::Buffer<>> out{"out"};

  std::vector<Var> vars;

  void generate() {
    out(vars) = lhs(vars) + rhs(vars);
  }
};

}  // namespace ft

HALIDE_REGISTER_GENERATOR(ft::Test, test_generator)

// file: lib.cc
#include <Halide.h>
#include "test_generator.h"

void test() {
  Halide::Runtime::Buffer<uint8_t> input(640, 480);
  test_generator(input, input, input);
}

# file: makefile
export CXX = clang++
export LD = lld

all:
	$(CXX) -fno-rtti -o impl.generator impl.cc GenGen.cpp -fpie -L/usr/local/bin -lHalide
	LD_LIBRARY_PATH=/usr/local/bin ./impl.generator \
		-e o,h	\
		-g test_generator \
		-o . target=x86-64-linux-no_runtime  \
		lhs.type=float32 lhs.dim=4 rhs.type=float32 \
		rhs.dim=4 out.type=float32
	LD_LIBRARY_PATH=/usr/local/bin ./impl.generator \
		-e o,h \
		-g test_generator \
		-o . \
		-r halide_runtime_x86-64-linux target=x86-64-linux \
		lhs.type=float32 lhs.dim=4 rhs.type=float32 rhs.dim=4 out.type=float32
	$(CXX) -c lib.cc -fPIC
	$(CXX) -o \
		libtest.so lib.o test_generator.o halide_runtime_x86-64-linux.o \
		-L/usr/local/bin/ -lHalide -pthread -ldl -shared -fuse-ld=$(LD)

clean:
	-rm *.bc *.o *.s *.so
	-rm impl.generator
	-rm test_generator.cpp
	-rm test_generator.h

In the above example I used llvm stack with lld. g++ with normal ld is also tried, here are the error messages from using g++ with default ld:

/usr/bin/ld: test_generator.o: relocation R_X86_64_PC32 against symbol `_ZN6Halide7Runtime8Internal13custom_mallocE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status

And with clang++/lld:

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_malloc; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_custom_malloc)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_malloc; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_custom_malloc)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_free; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_custom_free)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_free; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_custom_free)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_malloc; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_malloc)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_free; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_free)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::error_handler; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_error)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::error_handler; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_error_handler)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::error_handler; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_error_handler)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_print; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_print)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_print; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_custom_print)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::custom_print; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_set_custom_print)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol halide_reference_clock_inited; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_start_clock)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol halide_reference_clock; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_start_clock)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol halide_reference_clock_inited; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_start_clock)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol halide_reference_clock; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_current_time_ns)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::work_queue; recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_default_do_par_for)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::Synchronization::mutex_parking_control_validate(void*, Halide::Runtime::Internal::Synchronization::validate_action&); recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_mutex_lock)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::Synchronization::mutex_parking_control_unpark(void*, int, bool); recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_mutex_lock)

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol Halide::Runtime::Internal::Synchronization::parking_control_before_sleep(void*); recompile with -fPIC
>>> defined in test_generator.o
>>> referenced by posix_allocator.cpp
>>>               test_generator.o:(halide_mutex_lock)

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [makefile:8: all] Error 1

As hinted by @steven-johnson PIC is the default generated code model (thanks). But I haven't found any example usage of building shared library in Halide's source tree. Is there any flag I should put into the generator?

Halide is at 2e874da
No compile flag is specified during build.

OS: Ubuntu 18.10
gcc: 8.2.0
llvm/clang 7.0.1

[trivialfis]

Sorry for the previous incorrect flags, now is updated.

[trivialfis]

@steven-johnson Hi, I added -fPIC flag to these two lines and things started to work:

Halide/src/CMakeLists.txt

Line 173 in 549bf23

COMMAND ${CLANG} ${CXX_WARNING_FLAGS} ${RUNTIME_DEBUG_FLAG} -DDEBUG_RUNTIME -O3 -fno-ms-compatibility -ffreestanding -fno-blocks -fno-exceptions -fno-unwind-tables -std=gnu++98 -m${j} -target "${TARGET}" "-I${TARGET_RUNTIME_DIR}" -DCOMPILING_HALIDE_RUNTIME "-DLLVM_VERSION=${LLVM_VERSION}" -DBITS_${j} -emit-llvm -S "${SOURCE}" -o "${LL_D}"

Halide/src/CMakeLists.txt

Line 182 in 549bf23

COMMAND ${CLANG} ${CXX_WARNING_FLAGS} -O3 -fno-ms-compatibility -ffreestanding -fno-blocks -fno-exceptions -fno-unwind-tables -std=gnu++98 -m${j} -target "${TARGET}" "-I${TARGET_RUNTIME_DIR}" -DCOMPILING_HALIDE_RUNTIME "-DLLVM_VERSION=${LLVM_VERSION}" -DBITS_${j} -emit-llvm -S "${SOURCE}" -o "${LL}"

May I ask is not adding -fPIC to clang by design or by mistake?

[steven-johnson]

May I ask is not adding -fPIC to clang by design or by mistake?

Good question. Not sure. AFAICT we don't explicitly add -fPIC to the build options for either Make or CMake, and I'm not aware of other people reporting the same issue here, so something is unusual about your setup. It would be good to figure out what that is. I'm not sure about the implications of adding -fPIC everywhere.

[derek-gerstmann]

Hi! I just wanted to add that we ran into the exact same problem! The compiled runtime bits are not relocatable ... we had to add the -fPIC command line flags to the lines listed above in order for the objects/archives created by CLANG for the runtime to be linked into a shared object on Linux.