JSON downloading isn't checking for rate limits

[joewiz]

To reproduce the error I just got: clear the import directory, start http://localhost:8080/exist/apps/twitter/jobs/download-recent-twitter-json.xq. Errors out after 180 requests, with the following error:

err:XPTY0004 expression to the left of a lookup operator needs to be a sequence of maps or arrays [at line 145, column 44, source: /db/apps/twitter/modules/twitter-downloader.xqm] In function: twitter-dl:download-last-json(xs:unsignedLong?)

Submitting a subsequent user-timeline() request in eXide returns the following response indicating why the download failed:

<hc:response xmlns:hc="http://expath.org/ns/http-client" status="429" message="Too Many Requests" spent-millis="241">
    <hc:header name="content-length" value="56"/>
    <hc:header name="content-type" value="application/json;charset=utf-8"/>
    <hc:header name="date" value="Fri, 10 Jun 2016 14:29:59 GMT"/>
    <hc:header name="server" value="tsa_b"/>
    <hc:header name="strict-transport-security" value="max-age=631138519"/>
    <hc:header name="x-connection-hash" value="e5771896d2cb604eaf24fdb8620977ae"/>
    <hc:header name="x-rate-limit-limit" value="180"/>
    <hc:header name="x-rate-limit-remaining" value="0"/>
    <hc:header name="x-rate-limit-reset" value="1465569208"/>
    <hc:header name="x-response-time" value="7"/>
    <hc:body media-type="application/json"/>
</hc:response>

I wonder too if it might be more efficient to request more than 1 tweet at a time (as I understand the code), and store the json at the same time we store the xml, since now we have to submit duplicate requests.