Hotfix: Program Type Dataloss

[elipe17]

Summary of Changes

• Updated Postgres views to leverage program type for correct file querying and versioning
• Update the remove_all_old_versions celery task to leverage program type for correct querying (also cleaned up ugly nested loops)
• Add function to create backwards compatible Parser log file paths
• Updated all newly parsed datafiles to leverage program type in their s3 path
• Updated error report s3 path to leverage program type

How to Test

For local testing, to prove what was going on with delete old versions task, follow the steps below:

Creating the Error State (data loss)

• Checkout release tag v4.4.0
• Submit small_ssp_section1.txt and cat_4_edge_case.txt twice for the same stt on TDP v4.4.0
• You should have 4 datafiles in the DB and a total of 54 records
• Bring docker down but preserve volumes
• Switch to develop and start the containers
• Run remove_all_old_df_versions management command
• Notice that you only have 8 records in the database now. The expected would have been 27 records.

Proving the Task Runs Correctly

• Now destroy docker AND docker volumes
• Checkout release tag v4.4.0
• Submit small_ssp_section1.txt and cat_4_edge_case.txt twice for the same stt on TDP v4.4.0
• You should have 4 datafiles in the DB and a total of 54 records
• Bring docker down but preserve volumes
• Switch to hotfix/program-type-dataloss and start the containers
• Run remove_all_old_df_versions management command
• Notice that you now have the expected 27 records and they are associated with the latest versions of the files

[jtimpe]

Looks good! Thank you for your work on this.

[ADPennington]

thank you @elipe17 @jtimpe