Reduce upload buffer size in GoogleTaskLogs.#16236
Reduce upload buffer size in GoogleTaskLogs.#16236abhishekrb19 merged 2 commits intoapache:masterfrom
Conversation
Use a 1MB upload buffer, rather than the default of 15 MB in the API client. This is mainly because MMs may upload logs in parallel, and typically have small heaps. The default-sized 15 MB buffers add up quickly and can cause a MM to run out of memory.
| ) throws IOException | ||
| { | ||
| storage.get().createFrom(getBlobInfo(bucket, path), mediaContent.getInputStream()); | ||
| if (bufferSize == DEFAULT_BUFFER_SIZE) { |
There was a problem hiding this comment.
We use the term chunkSize in the getObjectOutputStream API down below, and the default value as DEFAULT_WRITE_CHUNK_SIZE. Uniformity by modifying either one would be good.
Another uniformity among both can be in the way the default gets picked: currently one does a null check while another expects the default value.
There was a problem hiding this comment.
"Buffer size" is the term the Google API client uses for this parameter. It looks like it uses "chunk size" for the other parameter you mention. IMO, we should retain the terms the Google client uses, even though they aren't consistent with each other. Better to be consistent with the upstream terms.
As to the way the default gets picked, I'll change the bufferSize to be a @Nullable Integer to match the way chunkSize works.
4 participants
Use a 1MB upload buffer, rather than the default of 15 MB in the API client. This is mainly because MMs may upload logs in parallel, and typically have small heaps. The default-sized 15 MB buffers add up quickly and can cause a MM to run out of memory.