feat(worker): add exponential backoff for failed task retries

[lukemarsden]

Tasks that fail during processing now use exponential backoff before retrying. This prevents failed tasks (e.g., cloning a non-existent repo) from immediately retrying in a tight loop. The backoff starts at 5 seconds and doubles with each retry, capping at 5 minutes.

Description

Adds exponential backoff retry logic for failed tasks in the indexing worker. When a task fails (e.g., cloning a repo that doesn't exist), it's now scheduled for retry with increasing delays: 5s → 10s → 20s → 40s → 80s → 160s → 300s (cap).

Changes:

• Added retry_count and next_retry_at fields to the Task entity (persisted to DB)
• Task repository next() now skips tasks waiting for retry
• Worker service marks failed tasks for retry instead of immediately re-processing
• Database migration adds the new columns to the tasks table

Related Issue

N/A

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Other (please describe)

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

Additional Notes

• Added 17 new tests covering backoff calculation, retry behavior, and persistence
• Retry state is persisted to survive restarts (tasks maintain their backoff schedule across deploys)
• Migration is backwards-compatible (new columns have defaults)
• Documentation updated in docs/reference/sync/index.md

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/kodit
_version.py	18	4	75%	15–19
app.py	78	78	0%	3–176
cli.py	51	17	67%	57–84, 99–101, 107
config.py	111	28	70%	24, 179, 182–185, 198, 207–209, 213–215, 222–225, 245–249, 255–263
database.py	42	26	35%	25–62, 71, 76–84, 88–89, 93
log.py	125	24	76%	60, 74, 142–146, 156–>160, 205–>198, 207–208, 213–223, 242, 244–253, 263–266, 276
mcp.py	159	114	22%	52–56, 73–77, 177–226, 231, 244–284, 317–341, 374–394, 427–451, 484–506, 539–559, 591
middleware.py	35	35	0%	3–75
src/kodit/application/factories
reporting_factory.py	21	11	48%	20–24, 31–36
server_factory.py	248	116	43%	199–201, 205–209, 213–215, 219–223, 229–235, 239–241, 245–249, 253–257, 261–263, 267–276, 280–282, 286–290, 294–298, 302–309, 313–524, 528–538, 542, 557–559, 565–567, 571–575, 579–583, 587–589, 593–598, 602–606, 610–612, 616–625, 629–633, 637–641, 645–649, 653–659, 663–669, 673–679
src/kodit/application/handlers
base.py	4	1	75%	11
registry.py	11	5	46%	12, 16, 20–22
src/kodit/application/handlers/commit
api_docs.py	51	39	20%	19–22, 38–43, 47–100
architecture_discovery.py	43	30	26%	27, 46–52, 56–117
commit_description.py	53	33	34%	23, 59–65, 69–132
cookbook.py	79	62	19%	30–33, 51–59, 63–164, 173–175
create_bm25_index.py	19	11	43%	15, 30–32, 36–53
create_code_embeddings.py	36	24	30%	20, 36–39, 43–75, 81–93
create_example_code_embeddings.py	36	24	30%	20, 36–39, 43–75, 83–93
create_example_summary.py	43	29	28%	22, 48–52, 56–118
create_example_summary_embeddings.py	50	35	26%	30, 48–53, 57–119, 127–137
create_summary_embeddings.py	50	35	26%	30, 48–53, 57–124, 130–142
create_summary_enrichment.py	43	29	28%	22, 44–48, 52–113
database_schema.py	48	33	27%	25, 106–112, 116–178
extract_examples.py	85	12	83%	25, 71, 103–>97, 107–>97, 144, 149–151, 155, 165–166, 171–173
extract_snippets.py	69	5	92%	25, 74, 110–112
scan_commit.py	35	1	93%	60, 69–>71
src/kodit/application/handlers/repository
clone_repository.py	31	19	37%	20, 38–44, 48–79
delete_repository.py	63	49	19%	24, 47–57, 61–95, 102–135, 141–161
sync_repository.py	37	4	84%	21, 60, 84–87
src/kodit/application/services
code_search_application_service.py	128	25	75%	24, 40, 54, 156–171, 178–189, 195–202, 260–282
commit_indexing_application_service.py	40	24	36%	33–37, 45–74, 78–85, 89–90
enrichment_query_service.py	129	40	66%	63, 77–82, 97–>99, 99–>101, 130, 144–156, 162, 170, 203–204, 208–209, 213–214, 220, 228–229, 235, 243–244, 248, 256–257, 279, 287–288, 304, 314–337, 356, 373–>371
indexing_worker_service.py	67	67	0%	3–131
queue_service.py	35	4	85%	54–63
reporting.py	40	5	83%	13, 72, 84, 88, 102
repository_query_service.py	97	72	20%	34–61, 68–80, 92–102, 114–138, 152–186, 190–199, 208, 213, 216, 222–237
repository_sync_service.py	80	11	83%	36, 38, 49, 69–70, 85–>87, 88, 105, 126–130, 139
sync_scheduler.py	40	27	28%	26–30, 34–36, 40–46, 50–65, 69–79
src/kodit/domain
errors.py	1	1	0%	4
protocols.py	107	9	92%	37, 41, 45, 51, 55, 59, 63, 67, 82
value_objects.py	282	11	96%	65, 92–115, 120, 125–128, 338, 645, 649
src/kodit/domain/enrichments
enricher.py	7	1	86%	17
enrichment.py	35	1	97%	41
src/kodit/domain/enrichments/architecture
architecture.py	9	1	89%	20
src/kodit/domain/enrichments/architecture/database_schema
database_schema.py	8	1	88%	17
src/kodit/domain/enrichments/architecture/physical
formatter.py	4	1	75%	11
physical.py	8	1	88%	17
src/kodit/domain/enrichments/history
history.py	9	1	89%	18
src/kodit/domain/enrichments/history/commit_description
commit_description.py	8	1	88%	17
src/kodit/domain/enrichments/usage
cookbook.py	8	1	88%	17
src/kodit/domain/entities
__init__.py	166	17	88%	23, 40, 81, 94, 104–108, 256–258, 262–263, 285–286, 348–349
git.py	121	15	87%	47, 53, 72, 87, 92–96, 141, 145–148, 166, 171, 188
src/kodit/domain/factories
git_repo_factory.py	17	4	68%	39–53, 79
src/kodit/domain/services
cookbook_context_service.py	148	45	65%	105–>109, 111–>115, 116–>120, 121–>124, 153–>159, 159–>184, 163–>167, 170–>161, 174–>161, 178–>181, 184–>201, 190–>201, 195–>198, 202–208, 229–>242, 231–>229, 237, 239–240, 251–258, 262–266, 270–>280, 276–277, 282–286, 291–295, 313–337
embedding_service.py	46	1	95%	92, 125–>124
git_repository_service.py	85	27	66%	70, 91, 112–114, 156–174, 193–194, 198–207, 212, 214–215
physical_architecture_service.py	58	5	86%	132, 134, 136, 138, 140
task_status_query_service.py	11	1	91%	24
src/kodit/domain/tracking
resolution_service.py	33	5	81%	41–44, 56–57
src/kodit/infrastructure/api/client
__init__.py	3	3	0%	3–6
base.py	39	39	0%	3–100
exceptions.py	4	4	0%	4–19
generated_endpoints.py	20	20	0%	9–67
src/kodit/infrastructure/api/middleware
auth.py	13	8	26%	16–19, 27–31
src/kodit/infrastructure/api/v1
dependencies.py	65	18	68%	40–43, 51–54, 76, 112, 124, 136, 146, 158, 170, 182, 194, 204
src/kodit/infrastructure/api/v1/routers
commits.py	117	117	0%	3–482
enrichments.py	54	54	0%	3–188
queue.py	16	16	0%	3–64
repositories.py	111	78	23%	62, 71, 83–97, 112–177, 203–229, 243–>246, 244, 247, 263–270, 298–308, 351–393, 407–416
search.py	16	16	0%	3–77
src/kodit/infrastructure/api/v1/schemas
commit.py	45	45	0%	3–96
queue.py	16	16	0%	3–35
search.py	106	106	0%	3–263
snippet.py	27	27	0%	3–58
src/kodit/infrastructure/bm25
local_bm25_repository.py	76	24	65%	22–23, 46–57, 76–77, 80–81, 88–89, 102–103, 107, 112, 137–>135, 150
vectorchord_bm25_repository.py	89	55	32%	118–120, 124–131, 135–139, 143–148, 152–154, 157–161, 165–202, 206–227, 234–237
src/kodit/infrastructure/cloning/git
git_python_adaptor.py	352	138	58%	19–37, 42–59, 116–118, 145–147, 156–164, 194–196, 226–228, 265, 311–363, 372–401, 440–449, 461–>457, 464–468, 499, 518–522, 569, 583–588, 614–618, 628, 633–634, 649–654, 664–>exit, 682–684, 691–725
working_copy.py	53	3	92%	48–55
src/kodit/infrastructure/database_schema
database_schema_detector.py	129	111	9%	62–80, 84–89, 93–102, 106–116, 120–129, 133–163, 167–268
src/kodit/infrastructure/embedding
embedding_factory.py	39	7	76%	64–65, 74–77, 85–86
local_vector_search_repository.py	35	20	32%	50–70, 75–88, 97
vectorchord_vector_search_repository.py	99	72	21%	100–105, 109–115, 119–120, 124–161, 167–202, 206–240, 249–260, 263–272
src/kodit/infrastructure/embedding/embedding_providers
litellm_embedding_provider.py	40	3	90%	70–78, 103–>107
local_embedding_provider.py	53	2	92%	16–17, 50–>60, 64–>78
src/kodit/infrastructure/enricher
enricher_factory.py	19	11	35%	21, 41–53
litellm_enricher.py	30	13	47%	47–79
local_enricher.py	45	30	28%	35–40, 55–121
utils.py	7	5	22%	20–30
src/kodit/infrastructure/example_extraction
parser.py	81	39	46%	39–>50, 64–70, 78–123, 135–137
src/kodit/infrastructure/git
git_utils.py	12	12	0%	3–32
src/kodit/infrastructure/ignore
ignore_pattern_provider.py	30	30	0%	3–69
src/kodit/infrastructure/mappers
task_mapper.py	12	1	86%	23
src/kodit/infrastructure/physical_architecture/detectors
docker_compose_detector.py	147	31	71%	52, 62–63, 123–>126, 127–>131, 135–>137, 149–162, 166, 195–198, 211–221, 246–>244, 255–>exit, 262–>255, 278, 296–299, 304–305, 321, 334–336
src/kodit/infrastructure/physical_architecture/formatters
narrative_formatter.py	86	5	93%	112–113, 115–116, 145
src/kodit/infrastructure/providers
async_batch_processor.py	20	2	86%	28, 48
litellm_provider.py	76	16	78%	40–>44, 55–87
src/kodit/infrastructure/reporting
db_progress.py	11	4	64%	17–19, 23
log_progress.py	18	9	45%	18–20, 24–40
telemetry_progress.py	9	2	78%	15, 19
src/kodit/infrastructure/slicing
api_doc_extractor.py	79	32	53%	44, 48–49, 57–59, 71, 110, 126, 145–146, 155–187
ast_analyzer.py	100	7	92%	38–39, 54–56, 138, 171
language_analyzer.py	526	159	65%	56, 78, 106, 117–>124, 121–>124, 125, 135–>144, 138–>135, 145–146, 165–168, 179, 186, 191–193, 198, 209–>214, 215, 219, 256, 276–>281, 282, 292–>297, 324, 363–365, 398, 412, 415–>420, 428, 432–443, 447–456, 460–475, 490–>485, 504, 522–527, 530, 534, 538, 562, 565–>570, 571, 578, 591–>588, 600–602, 630, 642–650, 677, 684, 697–702, 706–721, 770, 782–787, 791–806, 850–855, 859–874, 882, 914, 926–931, 935–950
slicer.py	301	49	78%	23, 68, 75–77, 86, 90, 97, 121–>114, 133–134, 140–142, 200, 214–217, 229, 236, 252–253, 271–>286, 281–>277, 322–>332, 324–>332, 336–>335, 342–343, 370–>357, 397–418, 445–>454, 468–470, 474, 489, 515–>514, 528–>526, 534, 548, 567, 585–>581, 588
src/kodit/infrastructure/slicing/formatters
template_formatter.py	44	4	88%	22, 61–63
src/kodit/infrastructure/sqlalchemy
embedding_repository.py	84	2	95%	179, 203, 211–>219
enrichment_association_repository.py	31	1	94%	69
enrichment_v2_repository.py	65	11	75%	153–204
entities.py	205	8	96%	37, 420–426
git_branch_repository.py	36	2	90%	33, 70
git_file_repository.py	26	1	96%	68
git_tag_repository.py	35	7	73%	29, 52, 61–69
query.py	154	19	84%	60, 62, 64, 66, 68, 71–74, 102–>104, 191–196, 235, 245, 265, 284, 340–345, 349–354
repository.py	170	11	91%	50, 70–>78, 95, 99, 136, 187–190, 297, 328, 333
task_repository.py	71	1	97%	46, 110–>exit
unit_of_work.py	30	9	60%	29–>exit, 41–43, 47–49, 53–55
src/kodit/migrations
env.py	30	30	0%	3–85
src/kodit/migrations/versions
4b1a3b2c8fa5_refactor_git_tracking.py	51	51	0%	10–185
04b80f802e0c_foreign_key_review.py	23	23	0%	10–98
7c3bbc2ab32b_add_embeddings_table.py	15	15	0%	10–54
7f15f878c3a1_add_new_git_entities.py	140	140	0%	10–689
9cf0e87de578_add_queue.py	15	15	0%	10–46
9d493502b3ac_add_task_retry_fields.py	15	15	0%	10–34
9e53ea8bb3b0_add_authors.py	33	33	0%	10–102
19f8c7faf8b9_add_generic_enrichment_type.py	52	52	0%	10–259
4073b33f9436_add_file_processing_flag.py	11	11	0%	10–35
4552eb3f23ce_add_summary.py	11	11	0%	10–33
85155663351e_initial.py	23	23	0%	10–97
af4c96f50d5a_remove_branch_tag_fk_constraint.py	35	35	0%	10–93
b9cd1c3fd762_add_task_status.py	21	21	0%	10–76
c3f5137d30f5_index_all_the_things.py	21	21	0%	10–49
f9e5ef5e688f_add_git_commits_number.py	15	15	0%	10–43
src/kodit/utils
dump_config.py	182	182	0%	3–361
dump_openapi.py	25	25	0%	3–39
generate_api_paths.py	40	40	0%	4–137
path_utils.py	38	29	17%	28–57, 63–65, 70–83
TOTAL	9164	3680	56%

Tests	Skipped	Failures	Errors	Time
371	8 💤	0 ❌	0 🔥	53.764s ⏱️