Skip to content

feat: write failed tuples to stderr#511

Closed
aaguiarz wants to merge 7 commits intomainfrom
codex/implement-fix-for-issue-449
Closed

feat: write failed tuples to stderr#511
aaguiarz wants to merge 7 commits intomainfrom
codex/implement-fix-for-issue-449

Conversation

@aaguiarz
Copy link
Member

@aaguiarz aaguiarz commented Jun 5, 2025

Summary

  • add support for printing failed tuples in the same format as the input file
  • document stderr output in README
  • update changelog

Testing

  • make format
  • make lint
  • make test (fails: docker not found)

#Fixes #449


https://chatgpt.com/codex/tasks/task_e_6842099b1c9c832282621dce3b0fd738

Summary by CodeRabbit

  • New Features

    • Failed tuples from import operations are now output to standard error in the same format as the input file, making it easier to identify and retry failed entries.
  • Documentation

    • Updated instructions to reflect the new behavior for handling failed tuples, including simplified retry steps and examples for redirecting output.
  • Tests

    • Added a new fixture file for basic tuple relationships to support testing.

@aaguiarz aaguiarz requested review from a team as code owners June 5, 2025 21:58
@aaguiarz
Copy link
Member Author

aaguiarz commented Jun 9, 2025

I tested this locally and it works as expected

@aaguiarz aaguiarz requested a review from Copilot June 9, 2025 21:23
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds functionality to emit failed tuples to stderr in the same format as the input, and updates related fixtures, documentation, and the changelog.

  • Writes failed tuples to stderr when any write operations fail
  • Introduces failed_output.go with CSV/YAML/JSON formatting helpers
  • Updates test fixtures, README usage examples, and CHANGELOG

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/fixtures/basic-tuples.yaml Added basic YAML fixture for tuple inputs
tests/fixtures/basic-tuples.csv Added basic CSV fixture for tuple inputs
cmd/tuple/write.go Prints failed tuples to stderr after formatting
cmd/tuple/failed_output.go New utilities for formatting tuples in CSV/YAML/JSON
README.md Documents stderr output of failed tuples
CHANGELOG.md Notes the new stderr feature under “Added”
a.fga.yml Sample config for tests, including tuple definitions
Comments suppressed due to low confidence (1)

cmd/tuple/write.go:208

  • Add a unit test to capture stderr output and verify failed tuples are emitted correctly in each supported format.
fmt.Fprint(os.Stderr, out)

a.fga.yml Outdated
define owner: [user]
define parent: [folder]
define can_view: owner or can_view from parent
define can_write: owner or can_write from parentdi
Copy link

Copilot AI Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in 'parentdi'; it should be 'parent'.

Suggested change
define can_write: owner or can_write from parentdi
define can_write: owner or can_write from parent

Copilot uses AI. Check for mistakes.
@aaguiarz
Copy link
Member Author

@coderabbitai review

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jun 13, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jun 13, 2025

Walkthrough

The changes introduce functionality to output failed tuples to stderr in the same format as the input file during the fga tuple write command. Documentation and changelog are updated to reflect this new behavior, and supporting code for formatting failed tuples in various formats is added.

Changes

File(s) Change Summary
CHANGELOG.md Added entry describing output of failed tuples to stderr in input file format.
README.md Updated retry instructions to use stderr redirection for failed tuples; removed old jq usage.
cmd/tuple/failed_output.go New file: Implements formatting of failed tuples to CSV, YAML, or JSON based on file extension.
cmd/tuple/write.go Enhanced: Outputs failed tuples to stderr in the detected input format after write operation.
tests/fixtures/basic-tuples.yaml New fixture: Example tuple in YAML format for testing.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant CLI
    participant FileSystem

    User->>CLI: Run 'fga tuple write --file tuples.csv 2> failed_tuples.csv > result.json'
    CLI->>FileSystem: Read tuples.csv
    CLI->>CLI: Attempt to write tuples
    CLI->>CLI: Collect failed tuples (if any)
    CLI->>CLI: Format failed tuples in input file format
    CLI->>FileSystem: Write failed tuples to stderr (failed_tuples.csv)
    CLI->>FileSystem: Write results to stdout (result.json)
Loading

Assessment against linked issues

Objective Addressed Explanation
Output failed tuples to stderr in the same format as the input file (#449)
Allow specification of output format for failed tuples (json, csv, yaml) (#449)
Documentation and example usage updated to reflect new failed tuple handling (#449)

Poem

In the warren of code, a new path appears,
Failed tuples now whisper through stderr cheers.
CSV, YAML, or JSON—just name your delight,
Retry with a hop, and all will be right.
The rabbits rejoice, for retrying is clear—
No more sifting with jq, just redirect and cheer!
🐇✨

✨ Finishing Touches
  • 📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
tests/fixtures/basic-tuples.yaml (1)

1-3: YAML style + missing newline

  1. Keys do not need to be quoted and quoting them reduces readability – the parser accepts it but it is unconventional.
  2. The file lacks a trailing newline, flagged by yamllint.
-- "user": "user:anne"
-  "relation": "owner"
-  "object": "group:foo"
+- user: user:anne
+  relation: owner
+  object: group:foo
+                                            # ← newline (editor will usually add)
🧰 Tools
🪛 YAMLlint (1.37.1)

[error] 3-3: no new line character at the end of file

(new-line-at-end-of-file)

cmd/tuple/write.go (1)

195-210: Print failed-tuple block with a newline and cleaner error variable

  1. fmt.Fprint(os.Stderr, out) may merge with subsequent shell prompts or logs.
  2. errFmt2 shadows the outer err and the trailing numeral is noisy.
-       out, errFmt2 := formatTuples(failedTuples, failedFormat)
-       if errFmt2 != nil {
-           return fmt.Errorf("failed to marshal failed tuples: %w", errFmt2)
-       }
-
-       fmt.Fprint(os.Stderr, out)
+       out, ferr := formatTuples(failedTuples, failedFormat)
+       if ferr != nil {
+           return fmt.Errorf("failed to marshal failed tuples: %w", ferr)
+       }
+
+       // always end with a newline so the shell prompt starts on a new line
+       if !strings.HasSuffix(out, "\n") {
+           out += "\n"
+       }
+       fmt.Fprint(os.Stderr, out)
cmd/tuple/failed_output.go (1)

99-118: CSV output may lack trailing newline

Many tools expect the file to end with a newline; gocsv.MarshalBytes omits it when there are zero tuples.
Add one to be safe:

b, err := gocsv.MarshalBytes(dto)
if len(b) > 0 && b[len(b)-1] != '\n' {
    b = append(b, '\n')
}
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 296eca9 and 9dfe245.

⛔ Files ignored due to path filters (1)
  • tests/fixtures/basic-tuples.csv is excluded by !**/*.csv
📒 Files selected for processing (5)
  • CHANGELOG.md (1 hunks)
  • README.md (1 hunks)
  • cmd/tuple/failed_output.go (1 hunks)
  • cmd/tuple/write.go (2 hunks)
  • tests/fixtures/basic-tuples.yaml (1 hunks)
🧰 Additional context used
🪛 YAMLlint (1.37.1)
tests/fixtures/basic-tuples.yaml

[error] 3-3: no new line character at the end of file

(new-line-at-end-of-file)

🪛 LanguageTool
README.md

[typographical] ~788-~788: It appears that a comma is missing.
Context: ...": 1, "total_count": 2 } ``` In some cases you may want to retry failed tuples (fo...

(DURING_THAT_TIME_COMMA)

🔇 Additional comments (3)
cmd/tuple/failed_output.go (1)

88-97: formatFromExtension misses .yml in default branch

Small: the switch already lower-cases the extension. Returning json for an unrecognised extension is fine, but consider an explicit .yml case to avoid surprises.

- case ".yaml", ".yml":
+ case ".yaml", ".yml":
     return yamlFormat
CHANGELOG.md (1)

5-7: Changelog entry LGTM

Clear and concise.

README.md (1)

788-795: Great usability improvement

The updated example clearly shows stderr redirection; docs align with the new behaviour.

🧰 Tools
🪛 LanguageTool

[typographical] ~788-~788: It appears that a comma is missing.
Context: ...": 1, "total_count": 2 } ``` In some cases you may want to retry failed tuples (fo...

(DURING_THAT_TIME_COMMA)

Comment on lines +35 to +56
for _, tupleKey := range tuples {
userParts := strings.SplitN(tupleKey.User, ":", 2)
if len(userParts) != 2 {
continue
}

userType := userParts[0]
userIDRel := userParts[1]
userID := userIDRel
userRel := ""

if strings.Contains(userIDRel, "#") {
parts := strings.SplitN(userIDRel, "#", 2)
userID = parts[0]
userRel = parts[1]
}

objParts := strings.SplitN(tupleKey.Object, ":", 2)
if len(objParts) != 2 {
continue
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Dropping malformed tuples silently can hide the very tuples a user needs to retry

If user or object don’t follow type:id[#relation] the write failed, yet we skip the tuple completely when preparing the CSV.
A user then loses visibility of why the write failed.

Instead, include the raw tuple in the CSV with blank split fields or fall back to JSON formatting and return an explicit error to the caller.

-       if len(userParts) != 2 {
-           continue
-       }
+       if len(userParts) != 2 {
+           // keep the tuple – populate only the fields we can parse
+           result = append(result, tupleCSVDTO{
+               UserType: "UNKNOWN",
+               UserID:   tupleKey.User,
+               Relation: tupleKey.Relation,
+               ObjectType: "UNKNOWN",
+               ObjectID:   tupleKey.Object,
+           })
+           continue
+       }

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents
In cmd/tuple/failed_output.go around lines 35 to 56, the code currently skips
tuples with malformed user or object fields without any indication, causing loss
of visibility into why writes failed. Modify the logic to include these
malformed tuples in the CSV output by either leaving the split fields blank or
falling back to JSON formatting for those entries. Additionally, ensure an
explicit error is returned to the caller to indicate the presence of malformed
tuples, so users can understand and retry appropriately.

@aaguiarz aaguiarz requested a review from rhamzeh June 17, 2025 18:14
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
CHANGELOG.md (1)

18-19: Inconsistent list marker style in CHANGELOG
The new bullet uses - while earlier sections mix * and -. To satisfy markdownlint (MD004) and maintain consistency, switch to a single marker—preferably * for all unordered lists.

Proposed diff:

- Added:
- - Failed tuples are now written to `stderr` in the format of the input file (#449)
+ Added:
+ * Failed tuples are now written to `stderr` in the format of the input file (#449)
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9dfe245 and d73108b.

📒 Files selected for processing (1)
  • CHANGELOG.md (1 hunks)
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
CHANGELOG.md

19-19: Unordered list style
Expected: asterisk; Actual: dash

(MD004, ul-style)

⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Test Release Process
  • GitHub Check: Tests

@dyeam0 dyeam0 requested a review from Siddhant-K-code June 24, 2025 04:41
Copy link
Member

@Siddhant-K-code Siddhant-K-code left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok, but i have couple of thoughts:

  • I am unsure whether we should have this failed tuples behind a configuration option, i.e., there should be an option to disable to by using configuration option in CLI.
  • a following comment regarding continuing the process even if storing to stderr failes.

@rhamzeh
Copy link
Member

rhamzeh commented Jun 26, 2025

TLDR: I will approve this as it's good for now. Normally STDERR is be reserved for application errors (crash, OOM, etc..) not service errors.

But I will create an issue for us to ideate on some improvements.

What I would like to see:

  • Optional, opt-in: Folks can pass in --output-success-file and --failed-success-file to output to the files from the CLI at (without redirection).
  • By default have the CLI output to STDOUT the failed tuples (and optionally the successful yuples) as they occur (not wait till the end). This will help in situations where e.g. the CLI stalls, encounters a bug or exits early for any reason. It will also help the user monitor the progress and don't have to sit wondering until the process is done.

rhamzeh
rhamzeh previously approved these changes Jun 26, 2025
Copy link
Member

@Siddhant-K-code Siddhant-K-code left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for now, will do the followups!!

@rhamzeh rhamzeh dismissed stale reviews from Siddhant-K-code and themself via 76c7eda June 26, 2025 14:43
@rhamzeh rhamzeh enabled auto-merge June 26, 2025 14:43
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
README.md (1)

789-796: Minor wording tweak for readability

Add commas (or an em-dash) around the “for example” parenthetical to avoid the run-on feel and satisfy the linter.

-In some cases you may want to retry failed tuples (for example, due to network issues). Failed tuples are now written to `stderr` in the same format as the input file and can be redirected to another file:
+In some cases, you may want to retry failed tuples—for example, due to network issues. Failed tuples are now written to `stderr` in the same format as the input file and can be redirected to another file:
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 86a01cc and 76c7eda.

📒 Files selected for processing (3)
  • CHANGELOG.md (1 hunks)
  • README.md (1 hunks)
  • cmd/tuple/write.go (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • cmd/tuple/write.go
  • CHANGELOG.md
🧰 Additional context used
🪛 LanguageTool
README.md

[typographical] ~789-~789: It appears that a comma is missing.
Context: ...": 1, "total_count": 2 } ``` In some cases you may want to retry failed tuples (fo...

(DURING_THAT_TIME_COMMA)

⏰ Context from checks skipped due to timeout of 90000ms (1)
  • GitHub Check: Test Release Process

@aaguiarz aaguiarz disabled auto-merge June 27, 2025 13:59
@aaguiarz
Copy link
Member Author

Let's hold this one from merging, we have another idea

@aaguiarz aaguiarz closed this Jul 7, 2025
@Siddhant-K-code Siddhant-K-code deleted the codex/implement-fix-for-issue-449 branch July 8, 2025 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

output failed tuples to stderr

4 participants