Scale ClamAV and App for March#4762
Merged
Merged
Conversation
Contributor
|
Terraform plan for meta No changes. Your infrastructure matches the configuration.✅ Plan applied in Deploy to Development and Management Environment #963 |
Contributor
|
Terraform plan for dev Plan: 1 to add, 0 to change, 1 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.dev.module.cors.null_resource.cors_header must be replaced
-/+ resource "null_resource" "cors_header" {
!~ id = "*******************" -> (known after apply)
!~ triggers = { # forces replacement
!~ "always_run" = "2025-03-11T15:53:15Z" -> (known after apply)
}
}
Plan: 1 to add, 0 to change, 1 to destroy.✅ Plan applied in Deploy to Development and Management Environment #963 |
asteel-gsa
force-pushed
the
scale-for-march
branch
from
c590f98 to
49f58fa
Compare
Contributor
|
gsa-suk
pushed a commit
that referenced
this pull request
Mar 14, 2025
gsa-suk
pushed a commit
that referenced
this pull request
Mar 14, 2025
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Apr 7, 2025
* Creating/Saving Audits * Audit migration command * Linting and feedback - Address feedback where we no longer need`_convert_additional_fields()` - it is a duplicate of a method in `utils.py`. - Minor cleanup on `_populate_accesses()`. I no longer felt it needed to be an isolated function while addressing the above bullet point * More linting and more feedback - Supplied two new arguments, "--disseminated" and "--intake". The first will only fetch SAC data that is disseminated, while the second will only fetch SAC data that has not yet disseminated. - Rather than iterate through 50k SACs at a time, this change introduces a while condition that will continuously migrate batches of 100 until all relevant SACs are migrated. This should mean we only need to run the command once at a time. * Exclusive mgmt command from Bandit Bandit does not like potential SQL injection with string-based query construction, however this logic is only accessed internally. * Remove deprecated "Schema" field * Updating mixins to account for audits * New migration and migration logic cleanup - Added `migrated_to_audit` flag on `SingleAuditChecklist` for determining which SACs have not yet been migrated. - Added some logic (for local testing ONLY) which cleans up audit data and references to mimic a clean slate. * Search on Audits, with some refactoring * Minor bug fixes * Missed the General Information tab * First start at cross validation * Migration script revisions - Updated `generate_audit_indexes` to catch an issue where `auditee_fiscal_period_end` is an empty string for some records. - `migrate_audits` command now uses ORM instead of raw queries for performance improvements. - Improved logging of `migrate_audits` for better monitoring of progress. - Inserted `null=True` into `migrated_to_audit` booleanfield for SACs. - Setting `migrated_to_audit=True` for SACs that get created during intake process. This is because an Audit already gets created with the SAC during this step. * Removing template tag * Adding blurb about how to use search * Updating the excel file handling to save records into the history table. Refactoring. * Migration corrections - `0019_audit_auditee_name` and `0019_singleauditchecklist_migrated_to_audit` were migration files that overlapped because they were created from separate branches. - This change moves `migrated_to_audit` sequentially after `audit_auditee_name`. This should be the extent of the changes needed to correct the issue, as the migration files do not conflict with one another. * Update instructions for testing migration script * Initial checkin for cog/over migration to Audit table * Cleanup * Added _get_cog_over * Update * Fix migration conflict * Create History when adding accesses Currently this is only generating SubmissionEvents. We want the History table to replicate these. * Revert Access change This is not needed in light of the `source-of-truth/events` PR. (https://github.com/GSA-TTS/FAC/pull/4784/files) * Creating/Saving Audits * Audit migration command * Linting and feedback - Address feedback where we no longer need`_convert_additional_fields()` - it is a duplicate of a method in `utils.py`. - Minor cleanup on `_populate_accesses()`. I no longer felt it needed to be an isolated function while addressing the above bullet point * More linting and more feedback - Supplied two new arguments, "--disseminated" and "--intake". The first will only fetch SAC data that is disseminated, while the second will only fetch SAC data that has not yet disseminated. - Rather than iterate through 50k SACs at a time, this change introduces a while condition that will continuously migrate batches of 100 until all relevant SACs are migrated. This should mean we only need to run the command once at a time. * Exclusive mgmt command from Bandit Bandit does not like potential SQL injection with string-based query construction, however this logic is only accessed internally. * Remove deprecated "Schema" field * New migration and migration logic cleanup - Added `migrated_to_audit` flag on `SingleAuditChecklist` for determining which SACs have not yet been migrated. - Added some logic (for local testing ONLY) which cleans up audit data and references to mimic a clean slate. * Missed the General Information tab * Updating mixins to account for audits * Search on Audits, with some refactoring * Minor bug fixes * Removing template tag * Adding blurb about how to use search * First start at cross validation * Migration script revisions - Updated `generate_audit_indexes` to catch an issue where `auditee_fiscal_period_end` is an empty string for some records. - `migrate_audits` command now uses ORM instead of raw queries for performance improvements. - Improved logging of `migrate_audits` for better monitoring of progress. - Inserted `null=True` into `migrated_to_audit` booleanfield for SACs. - Setting `migrated_to_audit=True` for SACs that get created during intake process. This is because an Audit already gets created with the SAC during this step. * Migration corrections - `0019_audit_auditee_name` and `0019_singleauditchecklist_migrated_to_audit` were migration files that overlapped because they were created from separate branches. - This change moves `migrated_to_audit` sequentially after `audit_auditee_name`. This should be the extent of the changes needed to correct the issue, as the migration files do not conflict with one another. * Update instructions for testing migration script * Updating the excel file handling to save records into the history table. Refactoring. * Fix migration conflict * Create History when adding accesses Currently this is only generating SubmissionEvents. We want the History table to replicate these. * Revert Access change This is not needed in light of the `source-of-truth/events` PR. (https://github.com/GSA-TTS/FAC/pull/4784/files) * Updated Events for Access. * Tested management command * Monitor A Bunch More Endpoints (#4764) * Add Audit Management Endpoints * Add unlock after certification to monitoring * Adjust row/column count for management endpoints * Mispelling is a crime punished with loading times * UNCLEAR COMMAS * Resolve copy paste error in a monitoring query. * Monitor even even more endpoints Add submission elibility (steps 1-3) and information form endpoints * Dependency updates (#4768) * Scale ClamAV and App for March (#4762) * Terraform module updates (#4692) * Initial Legacy Provider Update Sets the new provider into the terraform providers.tf and runs a dual provider setup, forcing any resource that does not natively interact with the new provider using: provider = cloudfoundry-community * ClamAV version 2.0.0 - Removes app_name_or_id for network policies - Reintroduces external network policy for clamav -> proxy - Deployed successfully around 9a with v2.0.0 clamav module and was able to successfully validate a file at around 916a without any issues * Clamav Module Update * Preliminary setup for v2.1.0 upgrade * Upgrade resources * Add upgrade docs * Add app -> clamav network policy * Update gitignore * Update gitignore * Update prepare-proxy.sh * Meta Provider Update * Update managed providers * prevent globbing * Move sandbox imports * Live commands * Documentation on all commands * Meta Module Upgrade * Dev and Preview Module Upgrade * Add routes.tf * Staging Module Upgrade * Production Module Upgrade * Update prod instances * Update bootstrap provider version * Update sandbox provider version * Upgrade sandbox shared modules provider * Updating the excel file handling to save records into the history table. Refactoring. * Fix migration conflict * Create History when adding accesses Currently this is only generating SubmissionEvents. We want the History table to replicate these. * Revert Access change This is not needed in light of the `source-of-truth/events` PR. (https://github.com/GSA-TTS/FAC/pull/4784/files) * Updated Events for Access. * Added _get_cog_over * Fixes after rebase with jr/source-of-truth/main * Updating the excel file handling to save records into the history table. Refactoring. * Scale ClamAV and App for March (#4762) * Terraform module updates (#4692) * Initial Legacy Provider Update Sets the new provider into the terraform providers.tf and runs a dual provider setup, forcing any resource that does not natively interact with the new provider using: provider = cloudfoundry-community * ClamAV version 2.0.0 - Removes app_name_or_id for network policies - Reintroduces external network policy for clamav -> proxy - Deployed successfully around 9a with v2.0.0 clamav module and was able to successfully validate a file at around 916a without any issues * Clamav Module Update * Preliminary setup for v2.1.0 upgrade * Upgrade resources * Add upgrade docs * Add app -> clamav network policy * Update gitignore * Update gitignore * Update prepare-proxy.sh * Meta Provider Update * Update managed providers * prevent globbing * Move sandbox imports * Live commands * Documentation on all commands * Meta Module Upgrade * Dev and Preview Module Upgrade * Add routes.tf * Staging Module Upgrade * Production Module Upgrade * Update prod instances * Update bootstrap provider version * Update sandbox provider version * Upgrade sandbox shared modules provider * Deleted to match jr/source-of-truth/main * Using only the audits with status SUBMITTED or IN_PROGRESS * Admin and migration changes - Now displaying Audit and Audit History in django admin. - Updated comments on `migrate_audits` mgmt command. - Now creating `History` for every `SubmissionEvent` tied to a SAC in the `migrate_audits` command. * Delete migrations - These should not still be here. * Correcting linting * Rename function, exit when no federal awards * Added year 2025, processing audits with SUBMITTED or DISSEMINATED status * My Submissions Beta Page * Update * Add arg for individual report_ids to migration script * Fix for auditee_ein * Updates per discussion * Removed initialize_db * Adding a beta page for Summary View * Remived CognizantAssignment * Removed unused imports * Removed unused imports * Tweak verbose names for Audit and Audit History * Fix issue with non-disseminated audits * Fix typo with showing download button * Fix notes to sefa count * Fix for counting of findings to be in-line with previous method * Update ein * Fix for counting of findings to be in-line with previous method * Limited number of rows read from audit table so container memory will not max out * This provides repeatable data loading for test * Updating README * Update menu.bash Line 54 was reading from the file (presumably untouched) when we wanted the script to read from a piped version of the file that removes `transaction_timeout`. * Update backend/util/load_public_dissem_data/README.md * File Uploads connection to Audit. Refactoring. * Assign file references to audit in migration script * Updates the script to handle additional utilities Resets the `migrated_to_audit` field after data load. Allows for truncating the `audit_audit` table during testing. * Minor tweak to the migration script * Adding api version 1.2.0 for SoT * Adding API test code Still needs more README, and perhaps not a `lib`. To be merged and tested with the new API code. * Updated api version ref in remaining sql files * Fixing an import * Adds some initial debugging This will want a flag to turn it on/off for production use. * Added api version 1.2.0 to live list * Export audit data * Export audit data * Export audit data * Fixed bugs and improved SQL scripts * Fixed the audit type underscore issue * Updating the downloads to use the audit table. * Initial commit - Created new model: `AuditValidationWaiver`. Operates the same as `SacValidationWaiver` but focuses on Audits. - Adjusted `test_admin` to account for the new model as well as testing actions on the audit instead of the SAC. - Created new method in `viewflow.py` for transitioning state of an Audit. - linting. * Include staffuser permissions for new models * Updates * Updates * Added Export Audit Data to CSV * Initial checkin * Updates * Updates * Update * Update * Copy over datasource during migration * PR Feedback * Refactor save_model() for AuditValidationWaiverAdmin * Further refactoring to save_model() and logical error fixes * Minor update to make code run smoothly * Added input parameters allowing audit_year or all options * Added year parameter * Testing * Testing * Testing * Testing * Testing * Fixed range * Updating the API to account for audit table, and refactoring. * Updating created/updated for the migration script * Implemented changes suggested by Matt * Removed print statement * Updated for full cycle This should allow us to take raw(ish) data from prod, clean it up further, and prepare it for reuse. More work would be needed to work directly with prod data. We (hopefully) will not have a need to re-run this process again. If we do, we will extend the script "further back" so that it is easy to replicate. * Minor changes to print statements and comments Nothing of note. No core logic changes. * In progress * Updating the previous references validation to use Audit table * Added logic to print results in CSV * Added exports for Federal year * Paired fixes We fixed a number of things in the comparison script, and like where we are. One fix in the API. Ready for iteration. * Fix admin test with deleting flagged records * Cleaning up the logging for search a bit * Bug fixing * Fix cog/over and history date on migration * Fixing html linting * Linting. * Linting. * Linting * bandit fixes Ignoring the test file, adding a timeout to `get()`. * mypy typing fixes * mypy fix * minor fix for api 1.2 * Do the date calculations for fac-accepted-date * Code improvement and completed api view for general table --------- Co-authored-by: Bobby Novak <176936850+rnovak338@users.noreply.github.com> Co-authored-by: Matthew Jadud <matthew.jadud@gsa.gov> Co-authored-by: Sudha Kumar <sudha.kumar@gsa.gov> Co-authored-by: James Person <james.person@gsa.gov> Co-authored-by: Phil Dominguez <142051477+phildominguez-gsa@users.noreply.github.com> Co-authored-by: Alex Steel <130377221+asteel-gsa@users.noreply.github.com> Co-authored-by: Sudha Kumar <135276194+gsa-suk@users.noreply.github.com> Co-authored-by: Hassan D. M. Sambo <hassandeme.mamasambo@gsa.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request to scale prod instances.