Implement uniq builtin command#17
Closed
AlexandreYang wants to merge 1 commit intomainfrom
Closed
Conversation
Co-authored-by: AlexandreYang <49917914+AlexandreYang@users.noreply.github.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Implements the POSIX
uniqcommand as a builtin in the safe shell interpreter. The command filters adjacent matching lines from input, with support for counting, duplicate/unique filtering, case-insensitive comparison, field/character skipping, and group display modes.Motivation
Extends the shell's builtin command set with
uniq, a commonly used text-processing utility needed for deduplication workflows in shell scripts.Testing
interp/builtins/uniq/uniq_test.go): 60+ tests covering all flags, edge cases, error paths, context cancellation, CRLF, binary data, stdin, pipesinterp/builtin_uniq_gnu_compat_test.go): 24 tests verifying byte-for-byte output equivalence with GNU coreutilsinterp/builtin_uniq_pentest_test.go): 26 tests covering integer edge cases, path injection, flag injection, large files, timeout, binary contenttests/scenarios/cmd/uniq/): 27 scenarios across basic, count, repeated, unique, skip, check_chars, ignore_case, zero_terminated, all_repeated, group, stdin, errors, and hardening categoriesgo test ./interp/builtins/uniq/ ./interp/ ./tests/✅Checklist
Summary of changes:
interp/builtins/uniq/uniq.go— Full implementation with flags:-c,-d,-D,-f,-i,-s,-u,-w,-z,--group,-h/--help. Streaming processing (O(1) memory for default and--groupmodes), capped buffering for--all-repeated. All writes error-checked, context cancellation respected in all loops.interp/builtins/uniq/uniq_test.go— Comprehensive unit testsinterp/builtin_uniq_gnu_compat_test.go— GNU output equivalence testsinterp/builtin_uniq_pentest_test.go— Security/hardening testsinterp/runner_exec.go— Register uniq package importtests/import_allowlist_test.go— Add safe symbols:bufio.Scanner,bufio.SplitFunc,io.WriteString,io.Writer,strconv.Itoa,strings.EqualFoldtests/scenarios/cmd/uniq/— 27 YAML scenario filesSHELL_COMMANDS.md— Added uniq to command reference tablePR by Bits
View session in Datadog
Comment @DataDog to request changes