Skip to content

fix(storage): use correct offset and size types #8538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
coryan merged 1 commit into from
Mar 15, 2022
Merged

fix(storage): use correct offset and size types #8538

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 5 additions & 8 deletions google/cloud/storage/parallel_upload.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,8 @@ Status ParallelUploadFileShard::Upload() {
left_to_upload_ = 0;
return status;
};
auto const already_uploaded = ostream_.next_expected_byte();
auto const already_uploaded =
static_cast<std::int64_t>(ostream_.next_expected_byte());
if (already_uploaded > left_to_upload_) {
return fail(StatusCode::kInternal, "Corrupted upload state, uploaded " +
std::to_string(already_uploaded) +
Expand All @@ -395,17 +396,13 @@ Status ParallelUploadFileShard::Upload() {
return fail(StatusCode::kNotFound, "cannot open upload file source");
}

static_assert(sizeof(std::ifstream::off_type) >= sizeof(std::uintmax_t),
"files cannot handle uintmax_t for offsets uploads");

// TODO(#...) - this cast should not be necessary.
istream.seekg(static_cast<std::ifstream::off_type>(offset_in_file_));
istream.seekg(offset_in_file_);
if (!istream.good()) {
return fail(StatusCode::kInternal, "file changed size during upload?");
}
while (left_to_upload_ > 0) {
auto const to_copy = static_cast<std::ifstream::off_type>(
std::min<std::uintmax_t>(left_to_upload_, upload_buffer_size_));
auto const to_copy = static_cast<std::streamsize>(
std::min<std::int64_t>(left_to_upload_, upload_buffer_size_));
istream.read(buf.data(), to_copy);
if (!istream.good()) {
return fail(StatusCode::kInternal, "cannot read from file source");
Expand Down
8 changes: 6 additions & 2 deletions google/cloud/storage/parallel_upload.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,8 +365,12 @@ class ParallelUploadFileShard {
std::shared_ptr<ParallelUploadStateImpl> state_;
ObjectWriteStream ostream_;
std::string file_name_;
std::uintmax_t offset_in_file_;
std::uintmax_t left_to_upload_;
// GCS only supports objects up to 5TiB, that fits comfortably in a
// `std::int64_t`, allows for any expected growth on that limit (not that we
// anticipate any), and plays nicer with the types in the standard C++
// library.
std::int64_t offset_in_file_;
std::int64_t left_to_upload_;
std::size_t upload_buffer_size_;
std::string resumable_session_id_;
};
Expand Down