Squash database migrations

[ocket8888]

Squashes Traffic Ops and Traffic Vault migrations that were in 6.1.0 into their respective create_tables.sql scripts.

Fixes #6698

---

Which Traffic Control components are affected by this PR?

• Traffic Ops
• Traffic Vault

What is the best way to verify this PR?

Make sure nothing's broken. All existing tests should still pass.
Upgrade a TO system from 6.1.0 to master and verify the databases still work and all tests pass.

PR submission checklist

• This PR has tests
• This PR has documentation
• This PR has a CHANGELOG.md entry
• This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY

[mitchell852]

sounds like this can only be merged after #6717

[mattjackson220]

Looks good to me! Will need #6717 to be merged first then this is good to go

[zrhoffman]

Looks good to me! Will need #6717 to be merged first then this is good to go

#6717 is merged

[zrhoffman]

#6802 seems to add some columns that don't exist on the master branch

--- apache.sql  2022-05-16 12:34:55.666481904 -0600
+++ 6802.sql    2022-05-16 12:35:08.473392689 -0600
@@ -58,6 +58,8 @@
     deliveryservice text NOT NULL,
     cdn text NOT NULL,
     version text NOT NULL,
+    provider text NOT NULL,
+    expiration timestamp with time zone DEFAULT now() NOT NULL,
     last_updated timestamp with time zone DEFAULT now() NOT NULL
 );

@@ -102,7 +104,7 @@
 -- Data for Name: sslkey; Type: TABLE DATA; Schema: public; Owner: traffic_ops
 --

-COPY public.sslkey (data, deliveryservice, cdn, version, last_updated) FROM stdin;
+COPY public.sslkey (data, deliveryservice, cdn, version, provider, expiration, last_updated) FROM stdin;
 \.

Current test version in #6802 (comment)

Test (click to expand)

import_and_dump_tv_schema() {
    output_file="$1";
    <<'DOCKER_COMMANDS' docker run --rm -i \
        -v "$(pwd)/traffic_ops/app/db/trafficvault/create_tables.sql:/create_tables.sql" \
        -e POSTGRES_USER=traffic_ops \
        -e POSTGRES_PASSWORD=twelve \
        -e PGPASSWORD=twelve \
        -e POSTGRES_DB=traffic_ops \
        postgres:13.6-alpine bash >"$output_file"
    trap 'echo "Error on line ${LINENO} of ${0}" >/dev/stderr; exit 1' ERR
    set -o errexit
    docker-entrypoint.sh postgres >/dev/stderr &
    echo 'Waiting for Postgres to start...' >/dev/stderr
    sleep 10
    psql -Utraffic_ops -fcreate_tables.sql >/dev/stderr
    pg_dump -Utraffic_ops
DOCKER_COMMANDS
};
git checkout apache/master
import_and_dump_tv_schema apache.sql
git checkout apache/pull/6802/merge
import_and_dump_tv_schema 6802.sql
diff -u apache.sql 6802.sql

[ocket8888]

You don't appear to have run the migrations based on that shell session

[zrhoffman]

Oh true

[ocket8888]

real indentation > spaces

but the rest of create_tables.sql is fake-indented, so it's consistent.

[zrhoffman]

When performing a db dump after running create_tables.sql, and the up migrations, role_id_seq sequence's currentval is different on this branch than on the master branch:

--- apache.sql	2022-05-16 13:22:03.797158150 -0600
+++ 6802.sql	2022-05-16 13:22:33.451033729 -0600
@@ -3563,7 +3279,7 @@
 -- Name: role_id_seq; Type: SEQUENCE SET; Schema: public; Owner: traffic_ops
 --
 
-SELECT pg_catalog.setval('public.role_id_seq', 7, true);
+SELECT pg_catalog.setval('public.role_id_seq', 1, false);
 
 
 --

[zrhoffman]

Also, the 00000000000000_init migrations existed in 6.1.x, so they should be removed, too. A user upgrading to ATC 7 will need to upgrade to 6.1 first, but IMO that is a reasonable requirement.

[ocket8888]

IMO that's a reasonable requirement

It's also been a long-standing actual requirement, to my knowledge.

I didn't collapse those because I thought they had some special sauce that just always needed to exist. I'll rectify that.

[zrhoffman]

To reference in another comment, this is the test I use to compare DB dumps from master and #6802 (click to expand)

import_and_dump_schema() {
    output_file="$1";
    <<'DOCKER_COMMANDS' docker run --rm -i \
        -v "$(pwd)/traffic_ops/app":/app \
        -w /app \
        -e POSTGRES_USER=traffic_ops \
        -e POSTGRES_PASSWORD=twelve \
        -e PGUSER=traffic_ops \
        -e PGPASSWORD=twelve \
        -e POSTGRES_DB=traffic_ops \
        postgres:13.6-alpine bash |
    set -o errexit
    (trap 'echo "Error on line ${LINENO} of Docker stdin"; exit 1' ERR
    docker-entrypoint.sh postgres &
    echo 'Waiting for Postgres to start...'
    sleep 10
    db/admin -env=production load_schema
    db/admin -env=production upgrade) >/dev/stderr
    pg_dump
DOCKER_COMMANDS
  sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]+\+[0-9]{2}/A_TIMESTAMP/g' >"$output_file"
};
CGO_ENABLED=0 go build -o traffic_ops/app/db/ traffic_ops/app/db/admin.go
git fetch apache
git checkout apache/master
import_and_dump_schema apache.sql
git checkout apache/pull/6802/merge
import_and_dump_schema 6802.sql
diff -u apache.sql 6802.sql

[zrhoffman]

Looks good! The only non-white-space difference post-squash is that the admin role does not have some permissions

--- 1-apache    2022-06-14 15:47:29.501648659 -0600
+++ 2-6802      2022-06-14 15:47:33.371722449 -0600
@@ -1,7 +1,4 @@
 1      ALL     A_TIMESTAMP
-1      DNS-SEC:DELETE  A_TIMESTAMP
-1      DNS-SEC:READ    A_TIMESTAMP
-1      PARAMETER-SECURE:READ   A_TIMESTAMP
 2      ASN:READ        A_TIMESTAMP
 2      ASYNC-STATUS:READ       A_TIMESTAMP
 2      CACHE-GROUP:READ        A_TIMESTAMP

which should be okay, because it has the ALL permission.