Skip to content

Conversation

@samseth-git
Copy link
Contributor

@samseth-git samseth-git commented Dec 17, 2025

Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], or [CONTRIB]. If a new feature introduces breaking changes for the or configuration files, please also add [BREAKING].

Description

  1. Adding new runtime overrides for formatter, interpolators, api_url.
  2. Making infile more robust and support overriding file-paths for multiple sources, API url_params for file-types, output.
  3. Added encoding for reader/writer with a default encoding if not passed.

Changes Made

  1. Added new column_formatters - constant_condition & override.
  2. Added new interpolator $override to read from cmdline and replace api/file-paths.
  3. Added encoding for reader and writer files with a new parameter to be passed from config, with default of utf-8
  4. Added cmdline argument --override_params to pass list of key=values space separated to provide overrides.
  5. Updated cmdline argument --inline to accept single string value, or list of key=values to override multiple file-paths.
  6. Updated testcases

Definition of Done

Before submitting this pull request, please ensure that the following criteria have been met:

  • All automated tests have passed successfully.
  • All manual tests have passed successfully.
  • Code has been reviewed by at least one other team member.
  • Code has been properly documented and commented as needed.
  • All new and existing code adheres to our project's coding standards.
  • All dependencies have been added or removed from the project's README or other documentation as needed.
  • Any relevant documentation or help files have been updated to reflect the changes made in this pull request.
  • Any necessary database migrations have been run.
  • Any relevant UI changes have been reviewed and approved by the UI/UX team.

Additional Notes

[Add any additional notes or context for the reviewer or future maintainers of this code.]

##Testing evidence
Input metadata file - config.yml
config1
config2

Command to test - /ingen/examples/config.yml --infile input1=/ingen/examples/sample_input1.csv input2=/ingen/examples/sample_input2.csv output=/ingen/examples/test.csv runDate=/ingen/examples/date.csv --override_params skip_col=N col_value=OVERRIDE purpose=TEST source=CLOUD

Testing LOGS-
run_logs

run_logs2

Skip constant_condition if override is matched
output_Y

Apply constant_condition if override is not matched
output_N

Thank you for submitting!

samseth-git and others added 2 commits December 16, 2025 17:39
Signed-off-by: Saambhavi <saambhavi01@gmail.com>
Signed-off-by: Saambhavi <saambhavi01@gmail.com>

Signed-off-by: Saambhavi <saambhavi01@gmail.com>
@samseth-git samseth-git changed the title Adding runtime overrides for formatter, interpolators, api_url and infile for multiple sources, url_params and encoding [FEATURE] Adding runtime overrides for formatter, interpolators, api_url and infile for multiple sources, url_params and encoding Dec 17, 2025
Signed-off-by: Saambhavi <saambhavi01@gmail.com>

Signed-off-by: Saambhavi <saambhavi01@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant