Update the pollSince time based on latest activation start time

[chetanmeh]

Change the pollSince time for each loop by basing it on the latest Start time seen in activation result

See #390 for background details

Testing Approach

While the fix is small it was tricky to validate the change. This PR add a test HTTP rever proxy which is used to intercept the calls between the cli client and the controller. The test then intercepts the request to check if since time gets changed or not

[chetanmeh]

@csantanapr @rabbah @dubee Please review

[csantanapr]

LGTM

[dariosm]

Hi, about this change: https://github.com/apache/incubator-openwhisk-cli/pull/396/files#diff-ad777399e121d471d84d4fd5bf456bceR390
@chetanmeh I believe it affects nested cloud functions. I’m experiencing activation logs missing while running activation poll and invoking a cloud function that triggers others. Actually, I can see inner cloud functions logs but never the logs of the main function than executed them.
We’ve discussed this with @mdeuser (thanks him for pointing me to this PR). I’ve reported the issue before here: https://ibm-cloud-tech.slack.com/team/UGVBSBWGH)

[mdeuser]

the activation polling scenario in question is:

1. activation A starts and continues to run
2. activation B starts and quickly completes
3. activation poll reports B completed and updates the since time
4. activation A completes
5. activation poll does not report activation A as being completed.

i believe the GET /activations api's since parameter is based off the start time, not the end time... but @rabbah will know for sure

[rabbah]

It’s possible to miss activations because there’s no guarantee the activations are committed to the datastore in order. The cursor in the cli advances based on the last fetched activations. The polling doesn’t back fill.

[mdeuser]

@rabbah - thanks fr the input. does the backend use the activation start time or end time when processing the activation list since parameter?

while not a guarantee, i'm thinking the activation order will likely be maintained most of the time - especially for activations completing in sequence and seconds apart. a polled-for, long running sequence action will probably run into this same issue.

[rabbah]

The activations in the database are indexed using the activation start time. The controller queries the database using the start key equal to the since parameter, and the end key equal to the upto parameter.