Feature/refetch

[tcfdev]

This PR is to merge the changes necessary to implement the Traffic Ops related changes for the Refetch Blueprint (PR 5910).

TLDR; Modified the /jobs endpoints for Traffic Ops to take an InvalidationType string that must be either 'REFRESH' or 'REFETCH'. There was some extra work around this modification, but that was its primary purpose.

The changes primarily fall on schema changes for the database and Traffic Ops API changes. Currently this is targeting Traffic Ops API 4.0.

TO Database:

• Remove unused tables and columns (job_status table, job_agent table, various columns in the job table)
• Modify the data within the job table to reflect the new schema and goal (e.g. remove the keyword column and change the parameters column to ttl_hr, etc.)
• TODO/Outstanding: Update app/db/seeds.sql and app/db/patches.sql to account for db schema changes Done

TO API:

• Add v4.0 InvalidationJob endpoints and structs to accomplish the requirements in the blueprint/refetch goals.
• Modify the previous endpoints to ensure compatibility with the DB schema changes

TO clients:

• Modify the v4 clients to take advantage of the TO v4.0 API and tc structs
• Ensure the tests pass

TP:

• Modify TP to use the stable (3.1) version of the /jobs endpoint

T3C (formerly ORT):

• Slight modification to ensure tests pass with the DB schema

Documentation:

• TODO/Outstanding: Want to ensure we are actually going to merge this into v4.0 before modifying all the documentation to reflect this.

---

Which Traffic Control components are affected by this PR?

• Documentation
• Traffic Control Cache Config (T3C, formerly ORT)
• Traffic Control Go Client v4
• Traffic Ops
• Traffic Portal

What is the best way to verify this PR?

To test whether REFETCH works specifically or not, you will need to set the parameter refetch_enabled to true for the global config file in the DB parameter table. Otherwise only REFRESH jobs will be allowed.

The TO database migration scripts will need to be run to ensure the transition from a previous schema works to the new. This is accomplished by using the db/admin utility as outlined here.

Once the DB migration has taken place, the Traffic Ops client API integration tests can be run. You will need TO to be up and running with the to_test database set up as described here.

• To test all currently supported APIs (ensures backwards compatibility as well) you can do something akin to:

cd github.com/apache/trafficcontrol/traffic_ops/testing/api
# default location is github.com/apache/trafficcontrol/traffic_ops/testing/api/conf/traffic-ops-test.conf
go test --cfg <location for your test config> ./...

• To test only the invalidation job specific changes for API 4.0 you could run something akin to:

cd github.com/apache/trafficcontrol/traffic_ops/testing/api/v4
# default location is github.com/apache/trafficcontrol/traffic_ops/testing/api/conf/traffic-ops-test.conf
go test -run Jobs --cfg <location for your test config>

Once the Traffic Ops Integration tests confirm the new and previous implementations of the /jobs endpoints still work then you can proceed to running and testing the TP integration tests still work. This will verify that TP is utilizing the previous stable version of the TO API (v3.1) and not the new /jobs endpoints.

Ensure a local version of TO is up and running with the correct DB schema. This should already be the case based on the TO client integration tests from before.

Ensure Traffic Portal is building and running as outline (here)[https://traffic-control-cdn.readthedocs.io/en/latest/development/traffic_portal.html#traffic-portal].

By default, the TP integration tests are set up to run against CIAB. If you're running this locally you will need to modify the config file located at github.com/apache/trafficcontrol/traffic_portal/test/integration/config.json. The params section will need to be modified:

  "params": {
    "apiUrl": " https://localhost:8443/api/4.0", 
    "baseUrl": "https://localhost:9090",
    "login": {
      "username": "admin",
      "password": "twelve12"
    },

Where
apiUrl = Running Traffic Ops API URL (This can and should be set to 4.0)
baseUrl = Running Traffic Portal URL
login.username = existing administrator account username
login.password = existing administrator account password
You can also, optionally, remove the --headless flag to see the integration tests as they run in Chrome Browser

cd github.com/apache/trafficcontrol/traffic_portal/test/integration
npm i
npm run start-webdriver
npm run test

PR submission checklist

• This PR has tests
• This PR has documentation - This is delayed for the time being until accepted or rejected in the current RC (6.0) for ATC.
• This PR has a CHANGELOG.md entry
• This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

[ocket8888]

I tried to look kind of broad-strokes at the design; there's more to say but it's not worth saying yet since this is just a draft.

Biggest thing that stood out to me was the CRUDer rearing its ugly head again

[rawlinp]

Biggest thing that stood out to me was the CRUDer rearing its ugly head again

This PR doesn't appear to be using the CRUDer any more than the original /jobs implementation. It still uses it for Read it appears, but create, update, and delete are all as CRUDer-free as the prior version. I'm guessing the prior version functions were mostly just copied then modified to work with the new changes we want.

[ocket8888]

I guess it's not fair to expect him to do more work than he needs to. It's just so terrible right now I can't believe he thought any of it was worth keeping.

[rawlinp]

Didn't manage to complete this review, but here are 22 comments in the meantime. A lot of them are related to using a "major version alias", so let me know if I need to explain that better. The basic idea is that it should be easy to add new minor version structs without having to update hundreds of function signatures all over the place.

[rawlinp]

Almost done reviewing this entire thing -- just need to finish the 2nd half of traffic_ops/traffic_ops_golang/invalidationjobs/invalidationjobs.go .

[rawlinp]

Ok, I've finally finished going through this entire thing.

[zrhoffman]

#6118's TP GHA failed due to #6283 needs to be rebased onto master now that #6284 is merged in order to pass

[rawlinp]

I think the DB migrations may need to be updated to a more recent timestamp: traffic_ops/app/db/migrations/2021081321025500_refetch.down.sql

Because there have been other migrations that have come in with more recent timestamps, this one won't be run in dev environments that have already run those more recent migrations.

[rawlinp]

TP integration test failure appears to be a transient issue unrelated to this PR, merging ahead.