Skip to content

Fix pagination beyond page 2#29

Merged
ChristopherDavenport merged 1 commit intodavenverse:masterfrom
timo-schmid:fix-pagination
Oct 15, 2019
Merged

Fix pagination beyond page 2#29
ChristopherDavenport merged 1 commit intodavenverse:masterfrom
timo-schmid:fix-pagination

Conversation

@timo-schmid
Copy link
Copy Markdown
Contributor

@timo-schmid timo-schmid commented Oct 6, 2019

In the process of changing Search.repositories to be paginated (#30) I noticed that I never get results beyond page 2.

I discovered that the Link header contains multiple links separated by , in one header line.
For example

curl -v https://api.github.com/search/repositories\?q\=scala\&sort\=stars\&page\=2 -O /dev/null

wiill return

Link: <https://api.github.com/search/repositories?q=scala&sort=stars&page=1>; rel="prev", <https://api.github.com/search/repositories?q=scala&sort=stars&page=3>; rel="next", <https://api.github.com/search/repositories?q=scala&sort=stars&page=34>; rel="last", <https://api.github.com/search/repositories?q=scala&sort=stars&page=1>; rel="first"

On the first page it works, because the first header there is the rel="next" link.

To fix it, I rewrote the logic for getNextUri.

The spec (page 5) seems to indicate this is not a standard, but it appears in the examples (page 9). So maybe it should be fixed ustream in http4s. I'd like to hear your thoughts on that.

While fixing the above I also noticed that runPaginatedRequest seemed to ignore invalid HTTP status codes. This change is kind of optional because the parsing of the response will later fail anyway, but I think it's a bit nicer to check the status code of the response first and the error message is much clearer.

@ChristopherDavenport
Copy link
Copy Markdown
Collaborator

Lets get an issue on http4s, and make sure we're handling this where most appropriate and then coping with it here in the process.

val req = auth.fold(baseReq)(setAuth(_)(baseReq))
c.run(req).use{resp =>
resp.as[B].map{
c.fetch(req) { resp =>
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we switch from run to fetch. run is clearer use semantics in my opinion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No clear reason - I just didn't know how to do it using f.
But looking at it now, I think it should also work using use as well.
I guess this part can be changed back - I can do it if you want.

@ChristopherDavenport ChristopherDavenport merged commit ce8e72d into davenverse:master Oct 15, 2019
@timo-schmid timo-schmid deleted the fix-pagination branch October 17, 2019 11:31
@ChristopherDavenport
Copy link
Copy Markdown
Collaborator

I think this may be caught by http4s/http4s#2967

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants