tests, scripts: cleanup tests and dev environment things

[flakey5]

Alternative title: making the tests & local dev environment not annoying and messy

This is an attempt to make the tests and local dev environment a lot nicer to work with.

Some of the main problems with them right now are:

1. No Sentry = no error visibility, nothing is logged to console
2. You can't run the worker in development unless it's in wrangler's remote mode. This requires a Cloudflare account & a R2 bucket to be setup on that account which is annoying
3. Tests are rather messy and haven't really been updated like the rest of the codebase has been. This was good when we were changing a lot so we could ensure we didn't break anything, but, it's time now to clean these up imo

This pr:

1. Rewrites tests to be a lot cleaner & conform with the style of the rest of the codebase
2. Adds an example of the structure of the dist-prod bucket (called dev-bucket)
   • Replaces the bucket data at tests/e2e/test-data/R2_BUCKET for e2e tests
   • Can be used in a dev script to populate a bucket for local testing (see 3)
3. npm run dev - starts a locally running version of the worker with a populated R2 bucket
   • Populated from the dev-bucket directory

What's TODO:

• Finish implementing e2e tests
• Go through code to see if there's anything that should be tested that isn't
• Make the local dev script nicer

Some current annoyances:

• The local dev script uses Miniflare since there isn't actually a way to populate an R2 bucket locally with Wrangler, so any niceness that that has we need to reimplement or go without. Hot reloading might not be possible either since Miniflare works off of the compiled worker.js file. I do have some ideas, but not entirely sure if they're gonna work.
• Ideally we would be able to use Workers' vitest integration, but it's not really possible since we're doing things like mocking an S3 api in our e2e tests. Vitest patches a lot of modules and globals making things like node:http unavailable. Workaround for this was found, so we can use the Vitest integration now

[ovflowd]

@flakey5 -- do you need a review now or is this still WIP?

[flakey5]

Still a wip

[flakey5]

Gonna go ahead and mark this for review now.

This unfortunately doesn't resolve the annoyances with running the worker locally though, there's still some things that I need to play around with to see if I can get it working in an ideal way.

[ovflowd]

Can this be exported to somewhere else?

[flakey5]

Wdym?

[ovflowd]

What's this for?

[flakey5]

Reading the contents of the dev-bucket folder and loading them into memory for the tests

[ovflowd]

OOC, why all these .gitkeep files? Is this like a "test" recration of a folder structure? Instead of this, could we make GHA create all these folders with mkdir before tests? :)

[flakey5]

Is this like a "test" recration of a folder structure

Yeah, this serves two purposes:

1. Helps with making local dev easier since a structurally accurate example is provided. This is especially useful for newcomers or for those that don't have R2 access. Ideally this will be reused in the local dev script I mentioned in the pr description but haven't gotten around to making it work yet. This is also why I think we shouldn't generate these files via GHA
2. Makes managing the contents of the bucket used during tests easier since it's not just a bunch of binary files

[ovflowd]

It took me so long to review this and for that, I'm sorry! I left a few comments, but the PR looks good :)

[ovflowd]

cc @flakey5 thanks for bearing with me 🙇

[ovflowd]

SGTM!

[ovflowd]

@flakey5 OOC are you switching from Wrangler to the Vite Plugin?

[ovflowd]

Ah sorry, nvm, you just added Vitest. Forgot of that.