Skip to content

[CALCITE-2322] Add fetch row count support#148

Closed
zacMode wants to merge 9 commits intoapache:mainfrom
zacMode:zac/calcite-2322-add-fetch-size-support
Closed

[CALCITE-2322] Add fetch row count support#148
zacMode wants to merge 9 commits intoapache:mainfrom
zacMode:zac/calcite-2322-add-fetch-size-support

Conversation

@zacMode
Copy link
Copy Markdown

@zacMode zacMode commented Jun 28, 2021

This resurrects #49, updating @kminder 's existing work per guidance from CALCITE-2322.

The purpose of this change is to add support for configurable "fetch size"/ "fetch row count". Currently, setting fetchSize has no effect, because avatica always uses a hardcoded default value. This PR resolves that issue.

HostnameVerification.class, false),

/** The number of rows to fetch per call, default is 100 rows. */
FETCH_ROW_COUNT("fetch_row_count", Type.NUMBER, AvaticaStatement.DEFAULT_FETCH_SIZE, false);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of default_fetch_size?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would second @julianhyde 's feedback here:

not "default_fetch_row_count" (because it is understood that this is a default that can be overridden per-statement; furthermore, there is a default default, which is 100);

@zacMode zacMode requested a review from vlsi June 30, 2021 16:41
Copy link
Copy Markdown
Contributor

@vlsi vlsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo the blank line in AvaticaStatement

Comment thread core/src/main/java/org/apache/calcite/avatica/AvaticaStatement.java Outdated
@zacMode
Copy link
Copy Markdown
Author

zacMode commented Jul 16, 2021

@F21 @zabetak @vlsi - any update on when this can make it into a release? Thanks!

Copy link
Copy Markdown
Member

@zabetak zabetak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, is it possible to add some unit tests verifying the default and overriden values?

private class FetchIterator implements Iterator<Object> {
private final AvaticaStatement stmt;
private final QueryState state;
private final int fetchRowCount;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be fetchSizeRows as per the property or there is a reason that the naming is different?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for using the same terminology

Copy link
Copy Markdown
Member

@joshelser joshelser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit-picking over some naming, but the change makes sense (and is likely an oversight).

Did you try to mock out enough of this such that we could write a unit test? I can imagine this would be not very straightforward.

private class FetchIterator implements Iterator<Object> {
private final AvaticaStatement stmt;
private final QueryState state;
private final int fetchRowCount;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for using the same terminology

HostnameVerification.class, false),

/** The number of rows to fetch per call, default is 100 rows. */
FETCH_SIZE_ROWS("fetch_size_rows", Type.NUMBER, AvaticaStatement.DEFAULT_FETCH_SIZE, false);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Playing Devil's Advocate: if Avatica is focused as a JDBC abstraction layer, should we just call the client configuration property fetch_size instead of fetch_size_rows? I think the javadoc here and in ConnectionConfig appropriately explains what fetchSize means.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. The property should be called fetch_size. JDK has a method 'ResultSet.setFetchSize(int rows)' and so 'fetch size' is unambiguously a row count.

@gabor-hargitai-privitar
Copy link
Copy Markdown

Hello, what is holding up this feature being merged? It would be very useful.

@zabetak zabetak self-assigned this Oct 17, 2022
@johndww
Copy link
Copy Markdown

johndww commented Oct 31, 2022

Hi @zabetak / @zacMode . Any idea when this could be merged or if I could help move this through? I ran into this issue (fetch size stuck at 100) as well which isn't working for very large size results over HTTP.

@zabetak zabetak closed this in 5047b56 Dec 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants