cloudfront update with lambda function for redirects #15151

crazy-max · 2022-07-18T20:25:02Z

follow-up and take 2 of #15118

As replace directive for an s3 bucket is limited to 50 we can't use them. It's also really slow to propagate each of them with the AWS CLI (~30m).

This pr adds a new function to releaser that will update a lambda edge function to handle redirects directly in our cloudfront distribution.

I have created a lab branch on my fork and also a S3 bucket with a Lambda Edge function attached to a new Cloudfront distribution on my personal AWS account to test this feature and so far so good:

$ curl -s -D - https://dom3zg2ev0we5.cloudfront.net/get-started/part2/ -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 18 Jul 2022 20:15:47 GMT
location: /get-started/02_our_app/
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 d0229dbe69f77738f3ccab386a045ad8.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG52-P2
x-amz-cf-id: _6ds2VEHsPPvW2w8IuT38Gqw8GmPoJCnMx4rN9MiU9OR1S72LUXDTQ==

Also looking at #15133, I have added a new redirects-prefixes.json file that handles redirects with prefixes so we can get rid of replaces in our s3 bucket with hardcoded domain inside it and have everything in place in our lambda edge function:

$ curl -s -D - https://dom3zg2ev0we5.cloudfront.net/ee/sqdsqdsqdsqds -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 18 Jul 2022 20:16:33 GMT
location: /
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 8b20ff9a1799265d378bf510ac3db6de.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG52-P2
x-amz-cf-id: KyHvFjoVvM9doj9iuCuAv4CYaxNoIG2GlhhB7KTwEcm2ntQLKDsVqQ==

Cloudfront URL: https://dom3zg2ev0we5.cloudfront.net/
Deployment demo: https://github.com/crazy-max/docker.github.io/runs/7397172669?check_suite_focus=true

#16 0.271 INFO: updating lambda function "DockerDocsRedirectFunction"
#16 0.374 INFO: lambda function updated successfully (arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction)
#16 0.374 INFO: waiting for lambda function to be processed
#16 3.549 INFO: lambda function version "2022-07-18-20:12:27" published successfully (arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction:8)
#16 3.701 INFO: cloudfront distribution "E2Q9X128R7SWCF" loaded
#16 3.749 INFO: cloudfront distribution configuration loaded
#16 3.749 INFO: cloudfront distribution viewer request function ARN found: "arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction:7"
#16 3.749 INFO: updating cloudfront config with viewer request function ARN "arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction:8"
#16 5.342 INFO: cloudfront config updated successfully
#16 DONE 5.6s

Keeping in draft for now as we have to create the Lambda Edge function and additional perms on AWS as well as getting the current cloudfront ID for each environment (cc @StefanScherer).

Signed-off-by: CrazyMax crazy-max@users.noreply.github.com

netlify · 2022-07-18T20:27:10Z

✅ Deploy Preview for docsdocker ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`1b93493`
🔍 Latest deploy log	https://app.netlify.com/sites/docsdocker/deploys/62fcd54b0aa8c300091ad95b
😎 Deploy Preview	https://deploy-preview-15151--docsdocker.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

crazy-max · 2022-08-08T15:21:38Z

Labs environment is ready and is using Cloudfront and Lambda Edge to handle redirections. Thanks @VictorBersy!!

$ curl -s -D - https://docs-labs.docker.com/get-started/part2/ -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 08 Aug 2022 15:08:46 GMT
location: /get-started/02_our_app/
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 f41c2361062c4fc74c645f4e4fddd2de.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C2
x-amz-cf-id: MpXwuPV4N_Kd8sVy_AGNc2Rbi2oniKOVRMDUrBsTFIcbxtiH4Sunbw==

$ curl -s -D - https://docs-labs.docker.com/ee/sqdsqdsqdsqds -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 08 Aug 2022 15:10:30 GMT
location: /
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 9891f2220bf61a27cb1f26085ab3703c.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C2
x-amz-cf-id: Fl9PI6vyWYRIy5kjHpjOEbtJI1kTCf7Onenhtm9wsXxN2_JrFuOQRw==

I will update the PR to reflect changes made on lab branch.

crazy-max · 2022-08-11T13:28:16Z

Ready for review. I have just enabled OIDC auth and cloudfront redirects with lambda edge func for our lab environment as it's the only one supporting it atm.

Next step I will open a PR to prepare the work for stage and prod environment so that we will be ready when @VictorBersy has created the new stage and prod environments that support OIDC auth and Cloudfront with Lambda Edge redirections as well.

@VictorBersy Let me know when we can catch up on this by giving me the IAM role ARN for OIDC auth and cloudfront id with lambda edge function name as well to use for stage and prod.

cc @thaJeztah @usha-mandya

AlyxPractice · 2022-08-12T18:08:01Z

We finally managed to make it work with Terraform on the lab branch. Next week I will be working on deploying this to stage.
I will create all brand new cloud resources, and make the switch through DNS, to avoid any downtime on stage and to learn how it behaves before hitting production.

Thanks a lot @crazy-max for your support and help 🙇

crazy-max · 2022-08-16T12:08:37Z

As discussed with @VictorBersy, I have created a dedicated workflow called deploy that will use the new way to deploy our docs with Cloudfront and Lambda Edge so we don't disturb the current one (publish) while testing migration for stage and prod environments.

crazy-max · 2022-08-16T12:09:39Z

.github/workflows/deploy.yml

+          if [ "${{ github.ref }}" = "refs/heads/master" ]; then
+            DOCS_URL="https://docs-stage2.docker.com"
+            DOCS_AWS_IAM_ROLE=""
+            DOCS_S3_BUCKET=""
+            DOCS_S3_CONFIG="s3-config.json"
+            DOCS_CLOUDFRONT_ID=""
+            DOCS_LAMBDA_FUNCTION_REDIRECTS=""
+            DOCS_SLACK_MSG="Successfully deployed docs-stage2 from master branch. https://docs-stage2.docker.com/"


@VictorBersy Let me know when you have the IAM role, bucket, cloudfront ID and lambda name to use for stage env. I already set DOCS_URL="https://docs-stage2.docker.com" as discussed.

Here's the requested information:

DOCS_AWS_IAM_ROLE="arn:aws:iam::710015040892:role/stage-docs-docker.github.io-20220816140248629900000003" DOCS_S3_BUCKET="stage-docs-docker.github.io" DOCS_CLOUDFRONT_ID="E1R7CSW3F0X4H8" DOCS_LAMBDA_FUNCTION_REDIRECTS="DockerDocsRedirectFunction-stage"

Thanks! Added stage aws config in last commit. Let me know when you have prod ready and I update the config for it too.

And for prod:

DOCS_AWS_IAM_ROLE="arn:aws:iam::710015040892:role/prod-docs-docker.github.io-20220816161549883800000001" DOCS_S3_BUCKET="prod-docs-docker.github.io" DOCS_CLOUDFRONT_ID="E228TTN20HNU8F" DOCS_LAMBDA_FUNCTION_REDIRECTS="DockerDocsRedirectFunction-prod"

@VictorBersy Thanks, I updated the config for prod. Is https://docs2.docker.com the right domain?

Yes it is, we use docs2.docker.com for now.

crazy-max · 2022-08-17T09:04:24Z

Perfect thanks @VictorBersy! Looks like we are ready to merge. Can you PTAL @thaJeztah? Thanks

thaJeztah

overall this looks great! Left some comments/questions

thaJeztah · 2022-08-17T10:02:29Z

.github/workflows/deploy.yml

+          if [ "${{ github.ref }}" = "refs/heads/master" ]; then
+            DOCS_URL="https://docs-stage2.docker.com"
+            DOCS_AWS_IAM_ROLE="arn:aws:iam::710015040892:role/stage-docs-docker.github.io-20220816140248629900000003"
+            DOCS_S3_BUCKET="stage-docs-docker.github.io"


Probably too late now (?) to rename these buckets but we should really try to get rid of that docker.github.io everywhere, and use docs.docker.com (also rename the repository back to docs or documentation or docs.docker.com at some point)

I guess it's to match the name of the repository. Am I right @VictorBersy?

Yes, it was to match the repo name, but I don't mind renaming them. It's not too late for that.
I can name everything as docs.docker.com. Ideally, it would match the future name of the repository. Let me know 🙇

I don't think we have a final "verdict" on the repository name (but anything else than docker.github.io 😂)

docker/docs would work for me (with the caveat that we used to have a repository with that name that's now archived; https://github.com/docker-archive/docs.docker.com

Oh, LOL, and which apparently also was renamed to docs.docker.com 😂 so either way we would break a redirect (not really important, that stuff is pre-historic).

Let me know when you have one, ideally before the production release, as it would make the migration more tedious 😊

@VictorBersy if it's ok with you that potentially the name of the bucket doesn't match the name of the repository, I'm inclined to name the buckets after the domain they're for (docs.docker.com / docs-stage.docker.com etc) as those are unlikely to change.

@crazy-max WDYT?

Oh absolutely, no problem with that!

I'm fine with it

.github/workflows/deploy.yml

.github/workflows/publish.yml

_releaser/aws.go

_releaser/cloudfront-lambda-redirects.js

docker-bake.hcl

thaJeztah · 2022-08-17T10:36:47Z

Oh! Take note that GitHub is hiding some comments (so may need a "click to load more comments")

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max · 2022-08-17T11:59:55Z

Should be good, thanks for the review @thaJeztah

thaJeztah

LGTM (to the extend of my knowledge 😅)

thaJeztah · 2022-08-17T13:58:52Z

okay; let's get this one in.

@VictorBersy for a follow-up; perhaps you have an answer on #15151 (comment)

Probably too late now (?) to rename these buckets but we should really try to get rid of that docker.github.io everywhere, and use docs.docker.com (also rename the repository back to docs or documentation or docs.docker.com at some point)

I guess it's to match the name of the repository. Am I right @VictorBersy?

crazy-max · 2022-08-17T15:07:28Z

So far so good for stage2: https://github.com/docker/docker.github.io/runs/7880300358?check_suite_focus=true

#15 0.396 INFO: updating lambda function "DockerDocsRedirectFunction-stage"
#15 0.583 INFO: lambda function updated successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage)
#15 0.584 INFO: waiting for lambda function to be processed
#15 3.824 INFO: lambda function version "2022-08-17T14:05:57Z" published successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage:2)
#15 4.181 INFO: cloudfront distribution "E1R7CSW3F0X4H8" loaded
#15 4.259 INFO: cloudfront distribution configuration loaded
#15 4.259 INFO: cloudfront distribution viewer request function ARN found: "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage:1"
#15 4.259 INFO: updating cloudfront config with viewer request function ARN "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage:2"
#15 6.175 INFO: cloudfront config updated successfully

See https://docs-stage2.docker.com

crazy-max · 2022-08-18T14:57:22Z

docs2 has been deployed too: https://github.com/docker/docker.github.io/runs/7901183427?check_suite_focus=true#step:9:229

#15 0.475 INFO: updating lambda function "DockerDocsRedirectFunction-prod"
#15 0.707 INFO: lambda function updated successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod)
#15 0.707 INFO: waiting for lambda function to be processed
#15 3.933 INFO: lambda function version "2022-08-18T14:51:50Z" published successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod:2)
#15 4.272 INFO: cloudfront distribution "E228TTN20HNU8F" loaded
#15 4.353 INFO: cloudfront distribution configuration loaded
#15 4.353 INFO: cloudfront distribution viewer request function ARN found: "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod:1"
#15 4.353 INFO: updating cloudfront config with viewer request function ARN "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod:2"
#15 5.333 INFO: cloudfront config updated successfully
#15 DONE 5.9s

See https://docs2.docker.com/

thaJeztah · 2022-08-18T15:07:14Z

ooh, and now I'm wondering if we could optimize things further;

curl -v -HEAD https://docs2.docker.com/go/linux-credentials
 TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 302
< content-type: text/html; charset=utf-8
< content-length: 313
< x-amz-error-code: Found
< x-amz-error-message: Resource Found
< x-amz-request-id: QE9X3EZJDS08QN4E
< x-amz-id-2: +YIPXP6llntxLYjqdeVgHk0AJUYqJuNN2gdn5ZNRuzn50KnGsIx3uroaQG5DqM80/WRZJxiODhE=
< location: /go/linux-credentials/
< date: Thu, 18 Aug 2022 15:01:06 GMT
< server: AmazonS3
< x-cache: Hit from cloudfront
< via: 1.1 bdbb0d922c29917c00cfed799f55e7c2.cloudfront.net (CloudFront)
< x-amz-cf-pop: AMS1-C1
< x-amz-cf-id: XBv-kKOMyKAF6x0h0rC2Qrxp5M2WipYWJtINYMU8AaZOjSamtQ7UAQ==
< age: 170
<
<html>
<head><title>302 Moved Temporarily</title></head>
<body>
<h1>302 Moved Temporarily</h1>
<ul>
<li>Code: Found</li>
<li>Message: Resource Found</li>
<li>RequestId: QE9X3EZJDS08QN4E</li>
<li>HostId: +YIPXP6llntxLYjqdeVgHk0AJUYqJuNN2gdn5ZNRuzn50KnGsIx3uroaQG5DqM80/WRZJxiODhE=</li>
</ul>
<hr/>
</body>
</html>

That's a (default?) redirect from s3 to redirect URLs with no trailing /

However, in case of the above URL, that means a double redirect;

curl -v -HEAD https://docs2.docker.com/go/linux-credentials/
..
< location: /desktop/get-started/#credentials-management-for-linux-users

So

https://docs2.docker.com/go/linux-credentials -> https://docs2.docker.com/go/linux-credentials/
https://docs2.docker.com/go/linux-credentials/ -> https://docs2.docker.com/desktop/get-started/#credentials-management-for-linux-users

Would be cool if our redirect function would be able to skip the first one.

crazy-max · 2022-08-18T15:21:14Z

That's a (default?) redirect from s3 to redirect URLs with no trailing /

Yes indeed it seems to be returned by s3 and not cloudfront so cannot be handled by our redirect function:

curl -s -D - https://docs2.docker.com/go/linux-credentials -o /dev/null
HTTP/2 302 
content-type: text/html; charset=utf-8
content-length: 313
x-amz-error-code: Found
x-amz-error-message: Resource Found
x-amz-request-id: YHM9DHTDJCZDDW4X
x-amz-id-2: dI4JS/+f1y6O9SVsnLQ/OkK2TFp486XW0B0Ml61RScOj4nHycBLZUrLLg60pEGXhnbir/3XBiiQ=
location: /go/linux-credentials/
date: Thu, 18 Aug 2022 15:19:20 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 5732b8336788d04c0d6cb18b0b2aa3c2.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C1
x-amz-cf-id: FSbbAeGX8wN0UgLyRE0UmfaQ0XPAZ5EmWqLMDNWsck8MxtRikM5ldg==

See x-cache: Miss from cloudfront.

I guess it needs some tuning on s3 but looks like this is the same behavior currently right?:

curl -s -D - https://docs.docker.com/go/linux-credentials -o /dev/null
HTTP/2 302 
content-type: text/html; charset=utf-8
content-length: 313
x-amz-error-code: Found
x-amz-error-message: Resource Found
location: /go/linux-credentials/
date: Thu, 18 Aug 2022 15:20:32 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 d6bff47a79bb5fa9800d9ee4b2b92146.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C2
x-amz-cf-id: OWQjQSIJRAbTpvzMWyJAsEohk6JWHgkwR0ZBp_u9I0aCP-Y_tZUT-g==

crazy-max · 2022-08-18T15:24:21Z

Actually I see that we could use our redirect function for this: https://stackoverflow.com/questions/67967925/how-to-redirect-internal-urls-to-end-with-trailing-slashes-with-s3-websites

But I'm afraid we could potentially break access to assets (images, etc..)

thaJeztah · 2022-08-18T15:31:48Z

But I'm afraid we could potentially break access to assets (images, etc..)

Yes, so I was wondering; it could possibly still work as long as we base our redirects on the https://docs.docker.com/redirects.json; in that case we can look if there's a match for the source URL (after appending a / if there is none) 🤔

The redirects.json doesn't have redirects for assets (at least not currently)

crazy-max · 2022-08-18T15:33:10Z

Yes, so I was wondering; it could possibly still work as long as we base our redirects on the https://docs.docker.com/redirects.json; in that case we can look if there's a match for the source URL (after appending a / if there is none) 🤔

Hum I see, yes that could work.

crazy-max · 2022-08-18T16:01:27Z

I changed the redirect function to also check for uri without trailing slash but it doesn't pass through 😞

$ curl -s -D - https://docs-labs.docker.com/go/linux-credentials -o /dev/null
HTTP/2 302 
content-type: text/html; charset=utf-8
content-length: 313
x-amz-error-code: Found
x-amz-error-message: Resource Found
x-amz-request-id: 4FR6HZ1Q4XMGRHBB
x-amz-id-2: ZBCpILJxcFcyWbgFd7g+ZF++/2uWYxphk6KwdOyTi0VWqEHSumUQ7WbnAPv1cBuE9aM5lz+R5Pg=
location: /go/linux-credentials/
date: Thu, 18 Aug 2022 15:57:59 GMT
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 c0c888b299b9797c37778648bae22064.cloudfront.net (CloudFront)
x-amz-cf-pop: BRU50-C1
x-amz-cf-id: 3e180Ew3TgspHoj_AaPKG47KFLYsTkMJdIQpWTU9WG5NYsHn-UIoTQ==
age: 17

As I can see from the request, Cloudfront cache that request because it handles itself the redirection without going to the viewer request function. So this is not an S3 config but something in the Cloudfront distribution to change.

As Cloudfront manages this redirection and handles its internal cache for them I think this is fine but we could improve this in a follow-up and let our lambda function do this redirection.

thaJeztah · 2022-08-18T16:18:47Z

Ah, that's a pity; thanks for trying!

crazy-max requested review from thaJeztah and usha-mandya July 18, 2022 20:25

crazy-max mentioned this pull request Jul 18, 2022

move s3 redirects to jekyll redirect_from #15133

Closed

This was referenced Jul 19, 2022

allow search engines from crawling only on production #15153

Merged

ci: add lab branch to publish workflow #15179

Merged

crazy-max mentioned this pull request Jul 30, 2022

dockerfile: switch location and release notes #15110

Closed

crazy-max force-pushed the cloudfront-lambda-redirects branch 2 times, most recently from 0191320 to 86b99f9 Compare August 4, 2022 15:41

crazy-max force-pushed the cloudfront-lambda-redirects branch 2 times, most recently from 4ecf7b9 to 3d212a2 Compare August 11, 2022 13:19

crazy-max marked this pull request as ready for review August 11, 2022 13:28

crazy-max force-pushed the cloudfront-lambda-redirects branch from 3d212a2 to 0f4039a Compare August 12, 2022 17:37

crazy-max force-pushed the cloudfront-lambda-redirects branch from 0f4039a to c694e2d Compare August 16, 2022 12:03

crazy-max commented Aug 16, 2022

View reviewed changes

thaJeztah requested changes Aug 17, 2022

View reviewed changes

crazy-max force-pushed the cloudfront-lambda-redirects branch from 7a60be2 to 8a95cd1 Compare August 17, 2022 11:43

crazy-max added 5 commits August 17, 2022 13:47

lab: cloudfront update with lambda edge function for redirects

92948cb

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

lab: move s3 redirects to lambda function

a0c0aec

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

ci: create deploy workflow for new way to deploy docs

4429162

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

ci: set aws config for stage env

73a7690

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

ci: set aws config for prod env

1b93493

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the cloudfront-lambda-redirects branch from 8a95cd1 to 1b93493 Compare August 17, 2022 11:47

crazy-max requested a review from thaJeztah August 17, 2022 12:00

thaJeztah approved these changes Aug 17, 2022

View reviewed changes

thaJeztah requested a review from AlyxPractice August 17, 2022 13:50

thaJeztah merged commit 91de5f5 into docker:master Aug 17, 2022

crazy-max deleted the cloudfront-lambda-redirects branch August 17, 2022 15:06

This was referenced Aug 17, 2022

chore: fix s3 redirects with prefixes #15404

Merged

ci: remove publish workflow #15406

Closed

crazy-max mentioned this pull request Aug 22, 2022

ci: switch stage2 to stage environment #15443

Merged

crazy-max mentioned this pull request Aug 30, 2022

Switch prod2 to prod environment #15525

Merged

cloudfront update with lambda function for redirects #15151

cloudfront update with lambda function for redirects #15151

Uh oh!

Conversation

crazy-max commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for docsdocker ready!

Uh oh!

crazy-max commented Aug 8, 2022

Uh oh!

crazy-max commented Aug 11, 2022

Uh oh!

AlyxPractice commented Aug 12, 2022

Uh oh!

crazy-max commented Aug 16, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlyxPractice Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crazy-max Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crazy-max commented Aug 17, 2022

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thaJeztah commented Aug 17, 2022

Uh oh!

crazy-max commented Aug 17, 2022

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Aug 17, 2022

Uh oh!

crazy-max commented Aug 17, 2022

Uh oh!

crazy-max commented Aug 18, 2022

Uh oh!

thaJeztah commented Aug 18, 2022

Uh oh!

crazy-max commented Jul 18, 2022 •

edited

Loading

netlify bot commented Jul 18, 2022 •

edited

Loading

AlyxPractice Aug 16, 2022 •

edited

Loading

crazy-max Aug 16, 2022 •

edited

Loading

crazy-max commented Aug 18, 2022 •

edited

Loading