[C++] Default thread count should count process affinity into account

[pitrou]

Describe the enhancement requested

When using something like taskset to reduce the number of CPUs actually usable by the current process, Arrow C++ should also limit its number of CPU threads to avoid thread oversubscription:

Compare with what Python does:

$ ipython
>>> import os
>>> os.process_cpu_count()
24
>>> from multiprocessing.pool import ThreadPool
>>> pool = ThreadPool()
>>> pool._processes
24
>>> import pyarrow as pa
>>> pa.cpu_count()
24

$ taskset -c 1,2,3 ipython
>>> import os
>>> os.process_cpu_count()
3
>>> from multiprocessing.pool import ThreadPool
>>> pool = ThreadPool()
>>> pool._processes
3
>>> import pyarrow as pa
>>> pa.cpu_count()
24

Component(s)

C++

[pitrou]

On Linux at least, we can use https://linux.die.net/man/2/sched_getaffinity. Other platforms could be supported later if necessary.

[MuteBardTison]

Hi! I'd love to work on this issue as my first contribution to Arrow. Let me know if there's anything specific you'd like me to keep in mind. Thanks!

[pitrou]

@MuteBardTison Just for the record, the current code is here:

arrow/cpp/src/arrow/util/thread_pool.cc

Lines 732 to 748 in bfaa612

	int ThreadPool::DefaultCapacity() {
	int capacity, limit;
	capacity = ParseOMPEnvVar("OMP_NUM_THREADS");
	if (capacity == 0) {
	capacity = std::thread::hardware_concurrency();
	}
	limit = ParseOMPEnvVar("OMP_THREAD_LIMIT");
	if (limit > 0) {
	capacity = std::min(limit, capacity);
	}
	if (capacity == 0) {
	ARROW_LOG(WARNING) << "Failed to determine the number of available threads, "
	"using a hardcoded arbitrary value";
	capacity = 4;
	}
	return capacity;
	}

[MuteBardTison]

Thanks @pitrou!
I opened a PR for the affinity logic: GH-45860

[pitrou]

@MuteBardTison Are you still willing to work on the PR and address the comments that were posted? Otherwise, I think someone else could pick this up.

[pitrou]

cc @AntoinePrv

[pitrou]

Issue resolved by pull request 47152
#47152