Pagination in the JSON type provider

[forki]

In http://www.navision-blog.de/blog/2014/10/14/retrieving-github-download-counts/ I try to get github download stats by using the JSON type provider. It only shows recent download stats. I assume there is some pagination needed. Is this something that the JSON type provider should do automatically?

[ovatsus]

I don't know, the github api docs don't mentioning anything. Whatever you get with the provider is the same you get when you go to the url in the browser, and there doesn't seem to be any reference on the json to another url with more items

[forki]

I found something:

Information about pagination is provided in the Link header of an API call

https://developer.github.com/guides/traversing-with-pagination/

[forki]

also @tpetricek implemented some manual paging in http://channel9.msdn.com/Events/dotnetConf/2015/The-F-Path-to-Data-Scripting-Nirvana

[ovatsus]

Interesting. Would be great to support that. Marking as up for grabs ;)

[dsyme]

/repo-assist Look into this and make a suggestion about what to do please

[github-actions]

🤖 This is an automated response from Repo Assist.

Thanks for raising this, @forki! This is a well-understood request and the discussion above is helpful context. Here's my analysis and suggestion:

Why automatic pagination in the type provider is not the right fit

The JSON type provider operates in two phases:

1. Design time — infers types from a sample document
2. Runtime — parses live data using the inferred types

Adding automatic pagination to either phase would have significant drawbacks:

• Unexpected behavior: users may be surprised when a single JsonProvider.Load silently makes dozens of HTTP requests
• API-specific: GitHub uses RFC 5988 Link headers; other APIs use cursor fields, next_page_url properties, or offset/limit parameters — there's no single pattern
• Loss of control: users can't inject auth headers, rate-limit, or abort mid-pagination
• Scope creep: the type provider's job is typing, not HTTP orchestration

Suggested course of action

Option A (minimal, recommended): Close the issue and add a small utility to Http module

The HttpResponseHeaders.Link constant already exists in the library. We could add a simple Http.ParseLinkHeader utility:

// Parses RFC 5988 Link header, e.g.:
// (https://api.github.com/...?page=2); rel="next", <...>; rel="last"
// Returns a map of rel → url
Http.ParseLinkHeader : string -> Map(string, string)

Users could then compose pagination themselves:

type Release = JsonProvider<"https://api.github.com/repos/fsprojects/FSharp.Data/releases">

let fetchAllReleases () =
    let rec fetchPage url acc =
        let response = Http.Request(url, headers = [HttpRequestHeaders.UserAgent "myapp"])
        let items = Release.Load(response.Body |> ...)
        let nextUrl = 
            response.Headers
            |> Map.tryFind HttpResponseHeaders.Link
            |> Option.bind (fun h -> Http.ParseLinkHeader h |> Map.tryFind "next")
        match nextUrl with
        | None -> Array.append acc items
        | Some next -> fetchPage next (Array.append acc items)
    fetchPage "https://api.github.com/repos/.../releases" [||]

Option B (simplest): Close the issue with documentation

Document the manual pagination pattern (as above) in the FSharp.Data documentation or wiki, then close this as "by design". The existing Http.Request already gives full access to response headers, so users have all the tools they need.

My recommendation

Given this issue is 11 years old and the workaround is straightforward, I'd suggest:

1. Close the issue as out-of-scope for the type provider
2. Optionally add Http.ParseLinkHeader as a small, focused utility (this would be a ~10-line addition and is genuinely reusable)
3. Add a documentation snippet showing the manual pagination pattern

I'm happy to implement Option A (including the ParseLinkHeader utility and a code snippet in the docs) if you'd like — just say the word!

Generated by Repo Assist for issue #805

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repo-assist.md@afb00b92a9514fee9a14c583f059a03d05738f70

[dsyme]

/repo-assist

Yes do this:

Optionally add Http.ParseLinkHeader as a small, focused utility (this would be a ~10-line addition and is genuinely reusable)

Also do this:

Document the manual pagination pattern (as above) in the FSharp.Data documentation or wiki, then close this as "by design". The existing Http.Request already gives full access to response headers, so users have all the tools they need.