Implement dd nearing completion by backwaterred · Pull Request #2474 · uutils/coreutils

backwaterred · 2021-07-02T21:28:29Z

Back in the day, I opened #1861 indicating that I would work on dd. I thought it would be a month or so to complete it, but I was off by a small-ish margin ;-). Even though a few features are still missing, and some cleanup work remains, I am opening this pull request to bring dev work to the wider community, and basically stop hoarding the project.

Below is a list of todo items I am aware of.

Windows testing
MacOS testing
RedoxOS testing [edit]

Future work (to be created as separate issues)

Additional testing. CLI testing for bad input. Trigger unit tests when tests performed from project root.
Print stats in response to the INFO signal. I believe this is only available on BSD-style (i.e. MacOS included) systems. I'm not currently running one, and there doesn't appear to be a standard way to do this in Rust.
A robust system-independent iflag=FLAG and oflag=FLAG implementation. Currently flags [cio, nocache, nolinks, text, binary] are not implemented at all, and [direct, directory, dsync, sync, nonblock, noatime, noctty, nofollow] are only expected to work on Linux. Testing is also a little light on this one.
A good Ctrl-c handler. This should try to shut things down nicely at first, then get more aggressive after a second Ctrl-c.
Deduplication of dd_fileout(...) and dd_stdout(...) (ideally there should just be one function). Fixing this will require adjusting the way that fsync() fdatasync() and write_all() are handled. The solution probably involves dynamic dispatch.
Tune Clap to make better use of the lib's features. Ideas: default values, improved help text. Clarity on difference between traditional and modern arg passing (Clap should give us --if foo.file for free, and if=foo.file should continue to work), mention multiplier strings.
Additional documentation for all the things.
Address the newline question: Block & Unblock currently only accept LF as a newline character. Should this behaviour be made system dependent?

I'd like to continue working on this project, and am happy for any suggestions.

- Prototype code. - Adds project boilerplate for integration with coreutils framework.

…ment-dd-bwtr

- Move code closer to real impl - Remove Makefile from repo

- implements basic copy & progress functionality - rough work for conversion tables - adds sanity tests (mt, zeros, ones, deadbeef & rand)

…le) tests for conversions

- Adds support for calling dd fn from cl - Adds basic cl tests from project root - Adds support for multiplier strings (c, w, b, kB, KB, KiB, ... EB, E, EiB.

- Moves all arg parsing & tests to parseargs.rs - Moves conversion tables to conversion_tables.rs - Adds ebcdic_lcase and _ucase tables - Refactors options: This **Breaks Write for Output**

- All conv flags now parsed (full functionality is TODO) - Changes functionality of eg. conv=ebcdic,ucase to match gnudd

- conv=sparse option requires knowledge of File/Stdout to change behaviour. - Unclear how best to impl this. - Possible option: develop 4 versions of dd<X,Y> for each valid pair of { File, Stdin, Stdout }.

- Adds duplicate dd fn :-( for differentiating between File backed and non-File outputs. - Implements cflag=sparse,fsync,fdatasync which were previously blocked. - Adds plumbing for IFlags & OFlags incl parsing. - Partial impl for seek=N and skip=N which were previously blocked.

- Completes impl of skip=N, and seek=N - Parses cbs=N

- Pushes file/stdout specific logic to trait objects. - Extracts read/write helper - Extracts conv/block/unblock helper - Impl block - WIP: Many failing tests!

- Unit tests split into individual folders - Parseargs test module renamed unit_tests to match new structure

- Move test macros to specific modules.

- Adds unit tests - Implements unblock - All tests passing

- Splits read fn into conv=sync and standard (consecutive) versions. - Fixes bug in previous read/fill where short reads would copy to wrong position in output buffer. - Fixes bug in unit tests. Empty source would pass (since no bytes failed to match).

- Adds tests where ibs causes extention - Impl conv=sync. All tests passing.

- Spoiler Alert: Turns out it was just the lcm.

- Adds print fn's - Modifies internal fn's as needed to track read/write state - Modifies status update thread to respect status level - Adds signal handler for SIGUSR1 (print xfer stats)

- Adds tests for count=READS and count=BYTES. - Implements count logic for read count and bytes count limits.

src/uu/dd/Cargo.toml

src/uu/dd/src/datastructures.rs

src/uu/dd/src/dd.rs

- Adds words to cspell exceptions - Converts test macros to use Default trait. - Converts parser to use Default trait. - Adds Windows-friendly test files for block/unblock when nl is present in test/spec file.

miDeb

Some more comments 🙂

src/uu/dd/cspell.json

src/uu/dd/src/dd.rs

src/uu/dd/src/parseargs/unit_tests.rs

- Adds more jargon to the spellchecker jargon file. - Adds more ignores to the per-project ignores. - Fixes clippy warnigns about uneeded clones on non-linux. - Block/Unblock now account for system dependend newlines.

backwaterred · 2021-07-28T15:47:27Z

May I get the pending build actions approved?

- cSpell valiantly vanquished - fmt issues addressed

src/uu/dd/src/dd.rs

…ils-master-2

backwaterred · 2021-08-04T00:56:13Z

I've addressed the block/unblock tests that fail on Windows... by ignoring them. I was working on a solution, but it was just big enough to feel like it should be a separate issue, so I've reverted my changes.

Some questions came up as I was implementing this feature. If there are certain answers to them feel free to link or give them here. If there are no certain answers and a bit of discussion is required, then it would probably be best to have this discussion in the issue itself. Either way, I'll link this comment in the issue when it's created ~~, and link this comment for reference~~. Questions are below:

What set of newline markers should this feature (block & unblock) support?
- Does the project already has a stance on newlines?
Should support be determined by the platform uutils was compiled on (ie. recognize only CR+LF on Windows, only LF on Linux, LF and maybe? CR on MacOS) or take Python's stance of universal newlines (recognize all newlines on all platforms).

sylvestre · 2021-08-04T07:43:16Z

Code coverage and clippy are failing on Windows.
Could you please just disable these macros for this platform and we will land it? :)
this PR is becoming quite big :)

backwaterred · 2021-08-04T19:49:07Z

Agreed :-) Those issues should be addressed in the most recent push.

sylvestre · 2021-08-04T20:48:23Z

Excellent! many thanks :)

ty and others added 30 commits March 16, 2021 14:32

Begins work on dd

af6bf3a

- Prototype code. - Adds project boilerplate for integration with coreutils framework.

Merge branch 'master' of github.com:backwaterred/coreutils into imple…

5271427

…ment-dd-bwtr

Develop prototype code

28e6d0d

- Move code closer to real impl - Remove Makefile from repo

Adds impl & tests

d6086fc

- implements basic copy & progress functionality - rough work for conversion tables - adds sanity tests (mt, zeros, ones, deadbeef & rand)

Implements conversions:- ascii, ebcdic, ibm, lcase, ucase- adds (simp…

0df4575

…le) tests for conversions

Builds out arg parsing. Adds support for if, of, & multiplier strings)

4996308

- Adds support for calling dd fn from cl - Adds basic cl tests from project root - Adds support for multiplier strings (c, w, b, kB, KB, KiB, ... EB, E, EiB.

Starts arg parsing. Cleans & refactors.

5c8c7ef

- Moves all arg parsing & tests to parseargs.rs - Moves conversion tables to conversion_tables.rs - Adds ebcdic_lcase and _ucase tables - Refactors options: This **Breaks Write for Output**

Completes refactor & clean-up. Implements full conv flag parsing.

9e933a3

- All conv flags now parsed (full functionality is TODO) - Changes functionality of eg. conv=ebcdic,ucase to match gnudd

Improves multiplier tests

7f7dd2a

Testt conv arg parsing.

b461f10

Adds plumbing for conv=sparse. Not impl.

5aabdf8

- conv=sparse option requires knowledge of File/Stdout to change behaviour. - Unclear how best to impl this. - Possible option: develop 4 versions of dd<X,Y> for each valid pair of { File, Stdin, Stdout }.

Implements swab. Adds odd & even tests.

beb7abc

Implements noerror (ignores read errors)

3c3af72

Continues impl

d97672d

- Completes impl of skip=N, and seek=N - Parses cbs=N

Refactor. Returns code to buildable

d0b4fe3

- Pushes file/stdout specific logic to trait objects. - Extracts read/write helper - Extracts conv/block/unblock helper - Impl block - WIP: Many failing tests!

Adds additional test resources & unit tests for block.

f2fa1e1

Imlements block. All tests passing.

48374ab

Fix regression in converison logic.

391a175

Project structure refactor.

ead3c8c

- Unit tests split into individual folders - Parseargs test module renamed unit_tests to match new structure

Project structure refactor (cont'd)

7bc151d

- Move test macros to specific modules.

Implements Unblock

787b969

- Adds unit tests - Implements unblock - All tests passing

Impl conv=sync

841faeb

- Adds tests where ibs causes extention - Impl conv=sync. All tests passing.

Adds additional conv=sync test for short reads.

ef6c850

Implements internal read/write buffer optimization

96fd665

- Spoiler Alert: Turns out it was just the lcm.

Minor optimization in calculation of lcm for internal r/w buffer.

a511db5

Implements status=LEVEL

fc110bb

- Adds print fn's - Modifies internal fn's as needed to track read/write state - Modifies status update thread to respect status level - Adds signal handler for SIGUSR1 (print xfer stats)

Implements count=N

8141919

- Adds tests for count=READS and count=BYTES. - Implements count logic for read count and bytes count limits.

Implements status=LEVEL parser.

06dcdc0

miDeb reviewed Jul 22, 2021

View reviewed changes

src/uu/dd/Cargo.toml Outdated Show resolved Hide resolved

src/uu/dd/src/datastructures.rs Show resolved Hide resolved

src/uu/dd/src/datastructures.rs Show resolved Hide resolved

src/uu/dd/src/dd.rs Show resolved Hide resolved

backwaterred added 3 commits July 22, 2021 15:49

Addresses build errors

56c2f50

- Adds words to cspell exceptions - Converts test macros to use Default trait. - Converts parser to use Default trait. - Adds Windows-friendly test files for block/unblock when nl is present in test/spec file.

Addresses build errors

885a875

- Adds words to cspell exceptions - Converts test macros to use Default trait. - Converts parser to use Default trait. - Adds Windows-friendly test files for block/unblock when nl is present in test/spec file.

Merge branch 'master' of github.com:backwaterred/coreutils

6184267

miDeb reviewed Jul 23, 2021

View reviewed changes

backwaterred and others added 6 commits July 23, 2021 14:53

Removes project-specific cspell files.

076ff32

Adds spell checker ignores to remaining project files.

86245ec

Adds additional words to exclude list. Improves documentation in dd.rs.

d1907ed

Attempts to fix remaining checks.

2ab238a

- Adds more jargon to the spellchecker jargon file. - Adds more ignores to the per-project ignores. - Fixes clippy warnigns about uneeded clones on non-linux. - Block/Unblock now account for system dependend newlines.

Merge branch 'uutils:master' into master

b3e7ff9

Merge branch 'master' of github.com:backwaterred/coreutils

0bbd4d4

Build blockers

0cb5cff

- cSpell valiantly vanquished - fmt issues addressed

sylvestre reviewed Jul 31, 2021

View reviewed changes

src/uu/dd/src/dd.rs Outdated Show resolved Hide resolved

sylvestre and others added 5 commits July 31, 2021 10:27

fix a clippy warning

e780fba

Merge branch 'master' of github.com:backwaterred/coreutils

e1fcc85

Fix clippy warnings and build issues. Defer NL to seperate issue.

7ed7c84

Merge branch 'master' of https://github.com/uutils/coreutils into uut…

601c9fc

…ils-master-2

Merge branch 'uutils-master-2'

67059db

backwaterred and others added 2 commits August 4, 2021 12:47

Removes unused macros from windows & macos builds

0864234

Merge branch 'uutils:master' into master

2c41838

sylvestre merged commit 54f4c8f into uutils:master Aug 4, 2021

backwaterred mentioned this pull request Aug 18, 2021

dd: Eliminate code duplication #2572

Closed

This was referenced Dec 31, 2021

implement dd #1861

Closed

implement dd #1313

Closed

Implement dd. #1121

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement dd nearing completion#2474

Implement dd nearing completion#2474
sylvestre merged 84 commits intouutils:masterfrom
backwaterred:master

backwaterred commented Jul 2, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miDeb left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

backwaterred commented Jul 28, 2021

Uh oh!

Uh oh!

backwaterred commented Aug 4, 2021 •

edited

Loading

Uh oh!

sylvestre commented Aug 4, 2021

Uh oh!

backwaterred commented Aug 4, 2021 •

edited

Loading

Uh oh!

sylvestre commented Aug 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

backwaterred commented Jul 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miDeb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

backwaterred commented Jul 28, 2021

Uh oh!

Uh oh!

backwaterred commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvestre commented Aug 4, 2021

Uh oh!

backwaterred commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvestre commented Aug 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

backwaterred commented Jul 2, 2021 •

edited

Loading

backwaterred commented Aug 4, 2021 •

edited

Loading

backwaterred commented Aug 4, 2021 •

edited

Loading