Commit 345c5d6
committed
Add rough memory size tracking with KllItemsSketch
When generating KllSketches, systems may need to have an idea
about how much memory utilization there is for a particular sketch.
For sketches with fixed-width types the answer can be computed
efficiently.
With the KllItemsSketch, this is more difficult because the sketch
can support String-types with variable widths. This commit adds
implementation support to expose the `getTotalItemsNumBytes`
method so that external systems can roughly track the memory
utilization of a particular sketch.
The change accomplishes this by intercepting the code where
a new item is added to the items array, or when a new array is
generated entirely. This will add a slight overhead due to the
sketch now needing to compute the length of inputs. For
fixed-width types the overhead is low. For string this will require
a call to encode the string as UTF-8/16 before adding it to the
array.
For fixed-width types, the calculations have little effective overhead
as the computation is a single array-access lookup + multiplication
with the type width.1 parent 0bff44b commit 345c5d6
File tree
15 files changed
+167
-9
lines changed- src
- main/java/org/apache/datasketches
- common
- kll
- test/java/org/apache/datasketches/kll
15 files changed
+167
-9
lines changedLines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
125 | 131 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
104 | 110 | | |
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
114 | 114 | | |
115 | 115 | | |
116 | 116 | | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
117 | 122 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
103 | 109 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
243 | 249 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
133 | 139 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
127 | 133 | | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
236 | 242 | | |
237 | 243 | | |
238 | 244 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
514 | 514 | | |
515 | 515 | | |
516 | 516 | | |
517 | | - | |
| 517 | + | |
518 | 518 | | |
519 | 519 | | |
520 | 520 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
514 | 514 | | |
515 | 515 | | |
516 | 516 | | |
517 | | - | |
| 517 | + | |
518 | 518 | | |
519 | 519 | | |
520 | 520 | | |
| |||
0 commit comments