feat: implement centralized logging configuration and integrate into …#37
feat: implement centralized logging configuration and integrate into …#37
Conversation
…preprocessing script #5
|
Keeping this as a draft until python orchestration is done. Thinking we can implement logging across pipeline and add to this PR as shell is refactored into python? Reviewing as we go? |
| file_handler = RotatingFileHandler(log_file, maxBytes=5*1024*1024, backupCount=5) | ||
| console_handler = logging.StreamHandler() | ||
|
|
||
| # Create formatters and add them to handlers |
There was a problem hiding this comment.
can we have more context for debugging
ex: '%(asctime)s - %(levelname)s - %(name)s - %(funcName)s:%(lineno)d - %(message)s'
There was a problem hiding this comment.
Sorry @eswarchandravidyasagar I had this as a draft PR because I am not done yet!
| logger.setLevel(level) | ||
|
|
||
| # Create handlers - stream handler for stdout and rotating file handler for file logging | ||
| file_handler = RotatingFileHandler(log_file, maxBytes=5*1024*1024, backupCount=5) |
There was a problem hiding this comment.
do you think it would be helpful to define encoding format ?like UTF-8 to prevent encoding errors
There was a problem hiding this comment.
What happens if multiple logger instances called at the same time will it create multiple tiny log files or one continued log file?
is it possible that Timestamped log files might break log continuity
|
Closing this PR. Irrelevant with refactor. |
2 participants
This pull request introduces a centralized logging configuration for the pipeline and refactors the logging throughout
scripts/preprocess.pyto use the new logger. The update replaces scattered and inconsistent logging and print statements with a uniform logger, improving maintainability and log management.Centralized logging setup:
logger_config.pythat provides aget_loggerfunction for consistent logger configuration, including log formatting, log level, and rotating file handlers.Refactoring and standardization in
preprocess.py:loggingandprintstatements with calls to the centralized logger, ensuring consistent log output for info, warning, and error messages. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]Code cleanup:
loggingmodule frompreprocess.py, relying instead on the shared logger Log file organization #5