[WEB-4999] feat: implement flexible data export utility with CSV, JSON, and XLSX support#7884
Conversation
… support - Introduced Exporter class for handling various data formats. - Added formatters for CSV, JSON, and XLSX exports. - Created schemas for defining export fields and their transformations. - Implemented IssueExportSchema for exporting issue data with nested attributes. - Enhanced issue export task to utilize the new exporter system for better data handling.
|
You have run out of free Bugbot PR reviews for this billing cycle. This will reset on October 20. To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial. |
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughReplaces bespoke per-format export code with a schema-driven Exporter and formatters (CSV/JSON/XLSX), adds declarative ExportSchema and IssueExportSchema, integrates them into the background export task with per-project exports and failure handling, provides docs and public re-exports, and adds a relation_type field plus reverse mapping to IssueRelation. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant BT as BackgroundTask (export_task)
participant IQ as Issues Queryset
participant ES as IssueExportSchema
participant EX as Exporter
participant FM as Formatter (CSV/JSON/XLSX)
participant S3 as Storage (S3 / presigned URL)
BT->>IQ: prepare base queryset
BT->>ES: serialize_issues(queryset, fields?)
ES-->>BT: records[]
BT->>EX: Exporter(format_type, IssueExportSchema, options)
EX-->>BT: exporter instance
BT->>EX: exporter.export(filename_base, records)
EX->>FM: format(filename_base, records, IssueExportSchema, options)
FM-->>EX: filename.ext, content
EX-->>BT: filename.ext, content
BT->>S3: upload(content) / request presigned URL
S3-->>BT: presigned_url
alt success
BT->>BT: mark ExporterHistory success
else error / invalid format
BT->>BT: mark ExporterHistory failed (log exception)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a flexible data export utility system that refactors the existing issue export functionality to use a schema-based approach with support for multiple output formats.
Key changes:
- Created a new exporters utility module with schema-based field definitions and formatters
- Implemented
IssueExportSchemafor structured issue data exports with nested attributes - Refactored the existing issue export task to use the new exporter system instead of hardcoded formatting functions
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
apps/api/plane/utils/exporters/schemas/base.py |
Defines base export field types and schema metaclass for field declarations |
apps/api/plane/utils/exporters/schemas/issue.py |
Implements issue-specific export schema with field definitions and custom preparers |
apps/api/plane/utils/exporters/formatters.py |
Contains format-specific exporters (CSV, JSON, XLSX) with proper data type handling |
apps/api/plane/utils/exporters/exporter.py |
Main exporter class that coordinates schema and formatter usage |
apps/api/plane/utils/exporters/__init__.py |
Module initialization with exported classes |
apps/api/plane/utils/exporters/schemas/__init__.py |
Schema submodule initialization |
apps/api/plane/utils/exporters/README.md |
Comprehensive documentation for the new export system |
apps/api/plane/bgtasks/export_task.py |
Refactored to use the new exporter system, removing old hardcoded functions |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
apps/api/plane/bgtasks/export_task.py(5 hunks)apps/api/plane/utils/exporters/README.md(1 hunks)apps/api/plane/utils/exporters/__init__.py(1 hunks)apps/api/plane/utils/exporters/exporter.py(1 hunks)apps/api/plane/utils/exporters/formatters.py(1 hunks)apps/api/plane/utils/exporters/schemas/__init__.py(1 hunks)apps/api/plane/utils/exporters/schemas/base.py(1 hunks)apps/api/plane/utils/exporters/schemas/issue.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (5)
apps/api/plane/utils/exporters/schemas/__init__.py (2)
apps/api/plane/utils/exporters/schemas/base.py (9)
BooleanField(107-115)DateField(65-76)DateTimeField(80-91)ExportField(6-49)ExportSchema(170-187)JSONField(137-150)ListField(119-133)NumberField(95-103)StringField(53-61)apps/api/plane/utils/exporters/schemas/issue.py (1)
IssueExportSchema(40-171)
apps/api/plane/utils/exporters/exporter.py (1)
apps/api/plane/utils/exporters/formatters.py (7)
CSVFormatter(55-100)JSONFormatter(103-127)XLSXFormatter(130-179)format(12-30)format(87-100)format(115-127)format(164-179)
apps/api/plane/utils/exporters/schemas/issue.py (2)
apps/api/plane/db/models/asset.py (2)
FileAsset(24-95)EntityTypeContext(29-39)apps/api/plane/utils/exporters/schemas/base.py (8)
DateField(65-76)DateTimeField(80-91)ExportSchema(170-187)JSONField(137-150)ListField(119-133)NumberField(95-103)StringField(53-61)serialize(174-187)
apps/api/plane/utils/exporters/__init__.py (4)
apps/api/plane/utils/exporters/exporter.py (1)
Exporter(6-56)apps/api/plane/utils/exporters/formatters.py (4)
BaseFormatter(9-52)CSVFormatter(55-100)JSONFormatter(103-127)XLSXFormatter(130-179)apps/api/plane/utils/exporters/schemas/base.py (9)
BooleanField(107-115)DateField(65-76)DateTimeField(80-91)ExportField(6-49)ExportSchema(170-187)JSONField(137-150)ListField(119-133)NumberField(95-103)StringField(53-61)apps/api/plane/utils/exporters/schemas/issue.py (1)
IssueExportSchema(40-171)
apps/api/plane/bgtasks/export_task.py (3)
apps/api/plane/db/models/exporter.py (1)
ExporterHistory(20-63)apps/api/plane/utils/exporters/exporter.py (2)
Exporter(6-56)export(32-46)apps/api/plane/utils/exporters/schemas/issue.py (2)
IssueExportSchema(40-171)serialize_issues(146-171)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Build and lint web apps
- GitHub Check: Analyze (javascript)
…xt handling - Updated issue export task to utilize new IssueRelation model for better relationship management. - Refactored Exporter class to accept QuerySets directly, improving performance and flexibility. - Enhanced IssueExportSchema to include parent issues and relations in the export. - Improved documentation for exporting multiple projects and filtering fields during export.
dheeru0198
changed the title
|
Linked to Plane Work Item(s) This comment was auto-generated by Plane |
There was a problem hiding this comment.
Actionable comments posted: 5
♻️ Duplicate comments (2)
apps/api/plane/bgtasks/export_task.py (1)
199-205: Good: per-project exports no longer skipped when emptyYou now generate a file per requested project regardless of issue count, addressing prior feedback.
apps/api/plane/utils/exporters/schemas/issue.py (1)
138-145: Past review comments still apply: cache cycle queries and format dates.The issues flagged in previous reviews remain unaddressed:
- Multiple calls to
i.issue_cycle.last()execute the same query repeatedlyprepare_cycle_start_dateandprepare_cycle_end_datereturn rawdateobjects that bypassDateFieldformatting, causing JSON serialization errorsPlease apply the fixes suggested in the previous review comments.
🧹 Nitpick comments (4)
apps/api/plane/db/models/issue.py (1)
276-289: Use enum values in reverse-mapping; avoid hard-coded stringsHard-coded keys risk drift if enum values change. Build the mapping using IssueRelationChoices values.
Apply this diff:
-IssueRelationChoices._RELATION_PAIRS = ( - ("blocked_by", "blocking"), - ("relates_to", "relates_to"), # symmetric - ("duplicate", "duplicate"), # symmetric - ("start_before", "start_after"), - ("finish_before", "finish_after"), - ("implemented_by", "implements"), -) +IssueRelationChoices._RELATION_PAIRS = ( + (IssueRelationChoices.BLOCKED_BY.value, "blocking"), + (IssueRelationChoices.RELATES_TO.value, "relates_to"), # symmetric + (IssueRelationChoices.DUPLICATE.value, "duplicate"), # symmetric + (IssueRelationChoices.START_BEFORE.value, "start_after"), + (IssueRelationChoices.FINISH_BEFORE.value, "finish_after"), + (IssueRelationChoices.IMPLEMENTED_BY.value, "implements"), +) -# Generate reverse mapping from pairs -IssueRelationChoices._REVERSE_MAPPING = {forward: reverse for forward, reverse in IssueRelationChoices._RELATION_PAIRS} +# Generate reverse mapping from pairs +IssueRelationChoices._REVERSE_MAPPING = { + forward: reverse for forward, reverse in IssueRelationChoices._RELATION_PAIRS +}Additionally, consider constraining IssueRelation.relation_type to known values for data integrity:
# outside selected lines (for IssueRelation.relation_type) relation_type = models.CharField( max_length=20, choices=IssueRelationChoices.choices, verbose_name="Issue Relation Type", default=IssueRelationChoices.BLOCKED_BY, )Please confirm a migration exists to add relation_type and, if needed, backfill accurate types for existing rows (not all should default to "blocked_by").
apps/api/plane/bgtasks/export_task.py (2)
175-179: Use select_related for forward FK 'parent'Prefetching a forward FK is suboptimal; select_related is faster and simpler.
Apply this diff:
- Prefetch( - "parent", - queryset=Issue.objects.select_related("type", "project"), - ), + ).select_related( + "parent", + "parent__type", + "parent__project", )
182-195: Avoid extra DB get on invalid format pathYou already loaded exporter_instance; reuse it to set failure state.
Apply this diff:
- try: + try: exporter = Exporter( format_type=provider, schema_class=IssueExportSchema, options={"list_joiner": ", "}, ) except ValueError as e: # Invalid format type - exporter_instance = ExporterHistory.objects.get(token=token_id) exporter_instance.status = "failed" exporter_instance.reason = str(e) exporter_instance.save(update_fields=["status", "reason"]) returnapps/api/plane/utils/exporters/schemas/base.py (1)
219-244: Consistent defaults when filtering fieldsWhen filtering, you default missing keys to "". Consider using each field’s default for consistency.
Apply this diff:
- filtered_data = {field: obj_data.get(field, "") for field in fields_to_extract if field in obj_data} + filtered_data = {} + for field in fields_to_extract: + if field in obj_data: + filtered_data[field] = obj_data[field]Since obj_data already contains all declared fields with proper defaults, omitting the extra default avoids masking schema defaults.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
apps/api/plane/bgtasks/export_task.py(4 hunks)apps/api/plane/db/models/issue.py(1 hunks)apps/api/plane/utils/exporters/README.md(1 hunks)apps/api/plane/utils/exporters/exporter.py(1 hunks)apps/api/plane/utils/exporters/schemas/base.py(1 hunks)apps/api/plane/utils/exporters/schemas/issue.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (4)
apps/api/plane/bgtasks/export_task.py (5)
apps/api/plane/db/models/exporter.py (1)
ExporterHistory(20-63)apps/api/plane/db/models/issue.py (2)
Issue(104-250)IssueRelation(291-315)apps/api/plane/utils/exception_logger.py (1)
log_exception(9-20)apps/api/plane/utils/exporters/exporter.py (2)
Exporter(8-67)export(34-57)apps/api/plane/utils/exporters/schemas/issue.py (1)
IssueExportSchema(40-173)
apps/api/plane/utils/exporters/schemas/issue.py (3)
apps/api/plane/db/models/asset.py (2)
FileAsset(24-95)EntityTypeContext(29-39)apps/api/plane/utils/exporters/schemas/base.py (8)
DateField(67-78)DateTimeField(82-93)ExportSchema(172-244)JSONField(139-152)ListField(121-135)NumberField(97-105)StringField(55-63)get_context_data(207-216)apps/api/plane/db/models/issue.py (1)
IssueRelationChoices(267-273)
apps/api/plane/utils/exporters/schemas/base.py (1)
apps/api/plane/utils/exporters/schemas/issue.py (1)
get_context_data(169-173)
apps/api/plane/utils/exporters/exporter.py (2)
apps/api/plane/utils/exporters/formatters.py (7)
CSVFormatter(55-100)JSONFormatter(103-127)XLSXFormatter(130-179)format(12-30)format(87-100)format(115-127)format(164-179)apps/api/plane/utils/exporters/schemas/base.py (1)
serialize_queryset(219-244)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Build and lint web apps
- GitHub Check: Analyze (javascript)
🔇 Additional comments (1)
apps/api/plane/utils/exporters/schemas/issue.py (1)
162-163: Remove incorrect critical flag:_REVERSE_MAPPINGis defined onIssueRelationChoices.
…upport - Updated Exporter class to merge fields into options for formatting. - Modified formatters to filter fields based on specified options. - Enhanced ExportSchema to support optional field selection during serialization. - Improved documentation for the serialize method to clarify field filtering capabilities.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
apps/api/plane/utils/exporters/schemas/issue.py (1)
169-177: Serialize cycle dates as strings to prevent JSON serialization errors.These methods return raw
dateobjects which bypass theDateFieldformatting logic. When preparer methods exist, they take precedence over field serialization (seeExportSchema.serialize()in base.py lines 183-197), so the returned date objects will causeTypeError: Object of type date is not JSON serializablein JSON exports.Based on learnings
Apply this diff to properly format the dates:def prepare_cycle_start_date(self, i): cycles_dict = self.context.get("cycles_dict") or {} last_cycle = cycles_dict.get(i.id) - return last_cycle.cycle.start_date if last_cycle else None + if not last_cycle or not last_cycle.cycle: + return "" + return self._format_date(last_cycle.cycle.start_date) def prepare_cycle_end_date(self, i): cycles_dict = self.context.get("cycles_dict") or {} last_cycle = cycles_dict.get(i.id) - return last_cycle.cycle.end_date if last_cycle else None + if not last_cycle or not last_cycle.cycle: + return "" + return self._format_date(last_cycle.cycle.end_date)
🧹 Nitpick comments (2)
apps/api/plane/utils/exporters/schemas/issue.py (2)
122-122: Return empty string for consistency.For consistency with other string fields in the schema, return an empty string instead of
Nonewhen the state is not available.Apply this diff:
def prepare_state_name(self, i): - return i.state.name if i.state else None + return i.state.name if i.state else ""
124-153: Consider prefetch optimization for related queries.Multiple prepare methods (lines 125, 131, 140, 147, 150, 153) trigger additional database queries for related objects. While this is not directly fixable in the schema, the caller should use
prefetch_related()on the queryset to avoid N+1 query problems when exporting many issues.Recommended prefetch list for the export queryset:
queryset = queryset.prefetch_related( 'issue_module__module', 'labels', 'issue_comments', 'issue_link', 'assignees', 'issue_subscribers' )This should be implemented in the calling code (e.g., the export task).
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/api/plane/utils/exporters/schemas/issue.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
apps/api/plane/utils/exporters/schemas/issue.py (4)
apps/api/plane/db/models/cycle.py (1)
CycleIssue(100-123)apps/api/plane/db/models/asset.py (2)
FileAsset(24-95)EntityTypeContext(29-39)apps/api/plane/utils/exporters/schemas/base.py (8)
DateField(55-66)DateTimeField(70-81)ExportSchema(160-234)JSONField(127-140)ListField(109-123)NumberField(85-93)StringField(43-51)get_context_data(202-211)apps/api/plane/db/models/issue.py (1)
IssueRelationChoices(267-273)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Analyze (javascript)
- GitHub Check: Build and lint web apps
🔇 Additional comments (7)
apps/api/plane/utils/exporters/schemas/issue.py (7)
1-16: LGTM!Imports are clean and properly structured.
19-37: LGTM!The function efficiently queries attachments and builds the mapping dictionary with proper use of
annotateanddefaultdict.
40-63: LGTM!The function correctly retrieves the most recent cycle per issue with proper query optimization via
select_related.
69-84: LGTM!Both static helper methods properly handle edge cases and None values.
86-116: LGTM!Field definitions are well-structured with appropriate types and clear labels for exports.
200-206: LGTM!The
get_context_datamethod correctly leverages the helper functions to provide necessary context for efficient serialization.
184-198: No action required:_REVERSE_MAPPINGis defined
IssueRelationChoices._REVERSE_MAPPINGis declared atapps/api/plane/db/models/issue.py:288, so the attribute access is valid.Likely an incorrect or invalid review comment.
Description
Type of Change
Screenshots and Media (if applicable)
Test Scenarios
Tested the export before and after making changes and verified both the files to be identical
References
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Refactor