Tar: filesizes >= 8Gb are truncated on extraction due to incorrect size being written

[jozkee]

From https://en.wikipedia.org/wiki/Tar_(computing)

Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final NUL or space character should also be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files.

You can also see this table to understand filesize capabilities of each format:

Create a 8 Gb file and test
On Windows I used fsutil file createnew large_file 8589934592.
On Ubuntu I used dd if=/dev/zero of=8g.img bs=1 count=0 seek=8G.

with BSD Tar and ustar format:

PS C:\repos\runtime\src\libraries\System.Formats.Tar\tests> tar -cvf large_file_bsd.tar --format ustar large_file
a large_filetar.exe: large_file: File size out of range

With GNU Tar and v7 and ustar formats:

david@amd-desktop:~/tar_largefile$ tar -cvf large.tar --format=v7 8g.img
tar: value 8589934592 out of off_t range 0..8589934591
tar: Exiting with failure status due to previous errors
david@amd-desktop:~/tar_largefile$ tar -cvf large.tar --format=ustar 8g.img
tar: value 8589934592 out of off_t range 0..8589934591
tar: Exiting with failure status due to previous errors

Using TarWriter and TarFile I noticed we are writing an incorrect size and that causes that the entry gets truncated when is extracted, either with TarFile.ExtractToDirectory or with another tool.

We should:

• throw if a V7TarEntry or UstarTarEntry entry would exceed the 8bg limit.
• ensure the file can properly roundtrip.

cc @carlossanlop

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

From https://en.wikipedia.org/wiki/Tar_(computing)

Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final NUL or space character should also be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files.

You can also see this table to understand filesize capabilities of each format:

Create a 8 Gb file and test
On Windows I used fsutil file createnew large_file 8589934592.
On Ubuntu I used dd if=/dev/zero of=8g.img bs=1 count=0 seek=8G.

with BSD Tar and ustar format:

PS C:\repos\runtime\src\libraries\System.Formats.Tar\tests> tar -cvf large_file_bsd.tar --format ustar large_file
a large_filetar.exe: large_file: File size out of range

With GNU Tar and v7 and ustar formats:

david@amd-desktop:~/tar_largefile$ tar -cvf large.tar --format=v7 8g.img
tar: value 8589934592 out of off_t range 0..8589934591
tar: Exiting with failure status due to previous errors
david@amd-desktop:~/tar_largefile$ tar -cvf large.tar --format=ustar 8g.img
tar: value 8589934592 out of off_t range 0..8589934591
tar: Exiting with failure status due to previous errors

Using TarWriter and TarFile I noticed we are writing an incorrect size and that causes that the entry gets truncated when is extracted, either with TarFile.ExtractToDirectory or with another tool.

We should:

• throw if a V7TarEntry or UstarTarEntry entry would exceed the 8bg limit.
• ensure the file can properly roundtrip.

cc @carlossanlop

Author:	Jozkee
Assignees:	-
Labels:	area-System.IO
Milestone:	8.0.0

[jozkee]

There's another issue related to large files; the reader parses the resulting octal size to int which makes sizes > 2 Gb overflow and throw System.IO.InvalidDataException : The size field is negative in a tar entry..

runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Read.cs

Lines 379 to 383 in 14cc725

	long size = (int)TarHelpers.ParseOctal<uint>(buffer.Slice(FieldLocations.Size, FieldLengths.Size));
	if (size < 0)
	{
	throw new InvalidDataException(string.Format(SR.TarSizeFieldNegative));
	}

We should address that as well as part of this issue.