Feat- Use cloud buckets for caching by naveensrinivasan · Pull Request #215 · ossf/scorecard

naveensrinivasan · 2021-02-23T18:20:05Z

Please check if the PR fulfills these requirements

Tests for the changes have been added (for bug fixes/features)

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
Feature
What is the current behavior? (You can also link to an open issue here)
The present implementation provides only disk-based caching.
What is the new behavior (if this is a feature change)?
- Use cloud buckets for HTTP cache.
- The implementation uses https://github.com/google/go-cloud for it to be cloud vendor agnostic.
- Uses the GCP Env for testing E2E with GitHub Actions
- Fixed a bug on handling the tarball httpcache Fix revalidation of gzipped JSON responses gregjones/httpcache#86

Cache

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
None
Other information:

github-actions · 2021-02-23T18:22:50Z

Integration tests failure for d62373d5bd811940da6476d791d56fa073e2a975

github-actions · 2021-02-23T18:27:46Z

Integration tests failure for e82c083da1490d3fae38cf66d9f7200820464062

github-actions · 2021-02-23T20:59:42Z

Integration tests failure for 78a9f9427f41cae4308004a620172b1c0b8c3efb

github-actions · 2021-02-23T21:05:56Z

Integration tests failure for 0153fc09a2e215e94728ee3aa647e6863881acfe

github-actions · 2021-02-23T21:14:01Z

Integration tests failure for f562c7afde831f2efac00a6c2b418158d4c07709

github-actions · 2021-02-23T21:21:15Z

Integration tests failure for 040003a6ddfc968a7fae3c0e3a650452222b9828

github-actions · 2021-02-23T21:28:38Z

Integration tests failure for 98d81c780243a8de3a5595924c6d3037512fdf74

github-actions · 2021-02-24T03:37:56Z

Integration tests success for 13f89aea035f845264bdb9d45d6b6e466992ebb7

github-actions · 2021-02-24T03:43:12Z

Integration tests success for 530a0017ef68768d78af43816b21cf62c9c41302

github-actions · 2021-02-24T03:47:45Z

Integration tests failure for 723524c89fe4ae37363eb56f13fde7e447d2ed2c

github-actions · 2021-02-24T03:53:09Z

Integration tests success for 723524c89fe4ae37363eb56f13fde7e447d2ed2c

naveensrinivasan · 2021-02-24T04:00:10Z

            go get github.com/onsi/ginkgo/ginkgo@v1.14.2
-            go mod download
            make ci-e2e
+            sleep 1m # this required to avoid 403 from GitHub API's


This will be addressed when we fix this issue #206

naveensrinivasan · 2021-02-24T04:00:18Z

 	mkdir -p cache
 	USE_DISK_CACHE=1 DISK_CACHE_PATH="./cache" ./scorecard --repo=https://github.com/ossf/scorecard --show-details --metadata=openssf  --format json > ./bin/results.json
-	ginkgo -p  -v -cover  ./...
+	@sleep 30


This will be addressed when we fix this issue #206

naveensrinivasan · 2021-02-24T04:00:44Z

 	USE_DISK_CACHE=1 DISK_CACHE_PATH="./cache" ./scorecard --repo=https://github.com/ossf/scorecard --show-details --metadata=openssf  --format json > ./bin/results.json
-	ginkgo -p  -v -cover  ./...
+	@sleep 30
+	ginkgo -p  -v -cover --skip="E2E TEST:blob"  ./...


Ignoring blob test when running disk cache tests

Use cloud buckets for httpcache. The implementation uses https://github.com/google/go-cloud for it to be cloud vendor agnostic.

github-actions · 2021-02-24T04:08:48Z

Integration tests success for 9db47ee40cb69230cb076d3ff29056876bd3ebdf

naveensrinivasan · 2021-02-24T04:13:28Z

 	github.com/kr/text v0.2.0 // indirect
-	github.com/naveensrinivasan/httpcache v1.2.1
-	github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
+	github.com/naveensrinivasan/httpcache v1.2.2


The fix for handling tarball gregjones/httpcache#86.

As the next steps, I will fork this repo into OSSF org.

It's fine to keep it in your personal one actually. Any reason to put it in the org?

Risk of me blowing off the repository 😉

dlorenc

Nicely done!

naveensrinivasan self-assigned this Feb 23, 2021

naveensrinivasan force-pushed the feat/caching-bucket branch from d62373d to e82c083 Compare February 23, 2021 18:24

naveensrinivasan force-pushed the feat/caching-bucket branch from e82c083 to 78a9f94 Compare February 23, 2021 20:57

naveensrinivasan force-pushed the feat/caching-bucket branch from 78a9f94 to 0153fc0 Compare February 23, 2021 21:02

naveensrinivasan force-pushed the feat/caching-bucket branch from 0153fc0 to f562c7a Compare February 23, 2021 21:10

naveensrinivasan force-pushed the feat/caching-bucket branch from f562c7a to 040003a Compare February 23, 2021 21:17

naveensrinivasan force-pushed the feat/caching-bucket branch from 040003a to 98d81c7 Compare February 23, 2021 21:24

naveensrinivasan force-pushed the feat/caching-bucket branch from 98d81c7 to 13f89ae Compare February 24, 2021 03:32

naveensrinivasan force-pushed the feat/caching-bucket branch from 13f89ae to 530a001 Compare February 24, 2021 03:38

naveensrinivasan force-pushed the feat/caching-bucket branch from 530a001 to 723524c Compare February 24, 2021 03:44

naveensrinivasan requested a review from inferno-chromium February 24, 2021 03:53

naveensrinivasan marked this pull request as ready for review February 24, 2021 03:53

naveensrinivasan requested a review from dlorenc February 24, 2021 03:53

naveensrinivasan linked an issue Feb 24, 2021 that may be closed by this pull request

Reducing GitHub API calls to scale scanning repositories #202

Closed

naveensrinivasan removed a link to an issue Feb 24, 2021

Reducing GitHub API calls to scale scanning repositories #202

Closed

naveensrinivasan commented Feb 24, 2021

View reviewed changes

Feat- Use cloud buckets for caching

9db47ee

Use cloud buckets for httpcache. The implementation uses https://github.com/google/go-cloud for it to be cloud vendor agnostic.

naveensrinivasan force-pushed the feat/caching-bucket branch from 723524c to 9db47ee Compare February 24, 2021 04:03

naveensrinivasan commented Feb 24, 2021

View reviewed changes

naveensrinivasan mentioned this pull request Feb 24, 2021

Bump github.com/naveensrinivasan/httpcache from 1.2.1 to 1.2.2 #216

Closed

dlorenc approved these changes Feb 24, 2021

View reviewed changes

naveensrinivasan merged commit cab29a2 into main Feb 24, 2021

naveensrinivasan deleted the feat/caching-bucket branch February 24, 2021 16:17

naveensrinivasan linked an issue Feb 25, 2021 that may be closed by this pull request

Move to more reliable and scalable caching solution #205

Closed

naveensrinivasan mentioned this pull request Feb 25, 2021

Move to more reliable and scalable caching solution #205

Closed

Conversation

naveensrinivasan commented Feb 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cache

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 23, 2021

Uh oh!

github-actions Bot commented Feb 24, 2021

Uh oh!

github-actions Bot commented Feb 24, 2021

Uh oh!

github-actions Bot commented Feb 24, 2021

Uh oh!

github-actions Bot commented Feb 24, 2021

Uh oh!

naveensrinivasan Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

naveensrinivasan Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

naveensrinivasan Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 24, 2021

Uh oh!

naveensrinivasan Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

dlorenc Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

naveensrinivasan Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

dlorenc left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

naveensrinivasan commented Feb 23, 2021 •

edited

Loading