Skip to content

Implement uniq builtin command#17

Closed
AlexandreYang wants to merge 1 commit intomainfrom
dd/implement-uniq-builtin
Closed

Implement uniq builtin command#17
AlexandreYang wants to merge 1 commit intomainfrom
dd/implement-uniq-builtin

Conversation

@AlexandreYang
Copy link
Copy Markdown
Member

What does this PR do?

Implements the POSIX uniq command as a builtin in the safe shell interpreter. The command filters adjacent matching lines from input, with support for counting, duplicate/unique filtering, case-insensitive comparison, field/character skipping, and group display modes.

Motivation

Extends the shell's builtin command set with uniq, a commonly used text-processing utility needed for deduplication workflows in shell scripts.

Testing

  • Go unit tests (interp/builtins/uniq/uniq_test.go): 60+ tests covering all flags, edge cases, error paths, context cancellation, CRLF, binary data, stdin, pipes
  • GNU compat tests (interp/builtin_uniq_gnu_compat_test.go): 24 tests verifying byte-for-byte output equivalence with GNU coreutils
  • Pentest tests (interp/builtin_uniq_pentest_test.go): 26 tests covering integer edge cases, path injection, flag injection, large files, timeout, binary content
  • YAML scenario tests (tests/scenarios/cmd/uniq/): 27 scenarios across basic, count, repeated, unique, skip, check_chars, ignore_case, zero_terminated, all_repeated, group, stdin, errors, and hardening categories
  • All tests pass: go test ./interp/builtins/uniq/ ./interp/ ./tests/

Checklist

  • Tests added/updated
  • Documentation updated (if applicable)

Summary of changes:

  • interp/builtins/uniq/uniq.go — Full implementation with flags: -c, -d, -D, -f, -i, -s, -u, -w, -z, --group, -h/--help. Streaming processing (O(1) memory for default and --group modes), capped buffering for --all-repeated. All writes error-checked, context cancellation respected in all loops.
  • interp/builtins/uniq/uniq_test.go — Comprehensive unit tests
  • interp/builtin_uniq_gnu_compat_test.go — GNU output equivalence tests
  • interp/builtin_uniq_pentest_test.go — Security/hardening tests
  • interp/runner_exec.go — Register uniq package import
  • tests/import_allowlist_test.go — Add safe symbols: bufio.Scanner, bufio.SplitFunc, io.WriteString, io.Writer, strconv.Itoa, strings.EqualFold
  • tests/scenarios/cmd/uniq/ — 27 YAML scenario files
  • SHELL_COMMANDS.md — Added uniq to command reference table

PR by Bits
View session in Datadog

Comment @DataDog to request changes

Co-authored-by: AlexandreYang <49917914+AlexandreYang@users.noreply.github.com>
@datadog-datadog-prod-us1
Copy link
Copy Markdown

datadog-datadog-prod-us1 Bot commented Mar 10, 2026

View session in Datadog

Bits Dev status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant