Skip to content

Conversation

@wjones127
Copy link
Member

In #12339 we added one, which enabled joining one table to one dataset using use_threads=false. However, I found that joining two datasets hit the thread limit.

There are plans to find a long-term fix that can run these operations synchronously with fewer threads, but that won't be ready for the next release.

As a temporary fix for 8.0.0, I propose just bumping up the local_states_ capacity.

@github-actions
Copy link

github-actions bot commented Apr 8, 2022

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@wjones127 wjones127 changed the title [C++] Increase thread limit to work around thread issues ARROW-15718: [C++] Increase thread limit to work around thread issues Apr 8, 2022
@github-actions
Copy link

github-actions bot commented Apr 8, 2022

@github-actions
Copy link

github-actions bot commented Apr 8, 2022

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

Copy link
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for realizing we still needed to patch this up. This looks good to me but I think we can be just a touch safer (the cost of allocating these extra states should be insignificant).

@wjones127 wjones127 marked this pull request as ready for review April 15, 2022 22:33
@wjones127
Copy link
Member Author

@wjones127 wjones127 requested a review from westonpace April 18, 2022 20:57
Copy link
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this.

@wjones127 wjones127 deleted the ARROW-15718-multiple-datasets branch April 21, 2022 14:41
@ursabot
Copy link

ursabot commented Apr 23, 2022

Benchmark runs are scheduled for baseline = 1763622 and contender = 08ab8b0. 08ab8b0 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Failed ⬇️0.75% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.67% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/566| 08ab8b0f ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/554| 08ab8b0f test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/552| 08ab8b0f ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/564| 08ab8b0f ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/565| 1763622b ec2-t3-xlarge-us-east-2>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/553| 1763622b test-mac-arm>
[Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/551| 1763622b ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/563| 1763622b ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants