Data migration system example

This set of scripts and classes is the example of subsystem responsible for the data migration
from the legacy MSSQL database to the new system

The code is mostly striped for the obvious reasons, still allowing to showcase overall structure and approach

File structure

• legacy_migration.py main entry point, launch specified, parralel jobs in chunks
• src/services/data_migration/data collection of classes responsible to collect data from the legacy systems
• src/services/data_migration/handler collection of classes responsible to process source data and import to the new system
• src/service/data_migration/schemas - data schemas

CLI interface

$ python legacy_migration.py --help
usage: legacy_migration.py [-h] [--concurrency CONCURRENCY] [--chunk-size CHUNK_SIZE] [--sources SOURCES] [--log-level LOG_LEVEL]

Migrate legacy data

options:
  -h, --help            show this help message and exit
  --concurrency CONCURRENCY
                        Concurrency level
  --chunk-size CHUNK_SIZE
                        Chunk size to process in one thread
  --sources SOURCES     Comma separated list of sources to migrate. all to migrate all of them
  --log-level LOG_LEVEL
                        Logger level