Add filter selectivity estimation for auto search strategy by jihoonson · Pull Request #3848 · apache/druid

jihoonson · 2017-01-13T11:50:06Z

Currently, AutoStrategy of search query uses a cost-based planner which needs to compute the exact selectivity of the filter specified in the query. According to the result of JMC profiling, I figured out the bottleneck is calling the expensive union() operation of bitmaps. So, I propose an alternative way to estimate filter selectivity.

In this patch, I simply assumed dimension values are independent to calculate approximate selectivity. As a result, the calculated selectivity may not be exact.
However, I think it's worthwhile because the planning cost of AutoStrategy will be reduced, thereby that strategy can be used in more cases which will be helpful for users. In addition, we can improve the selectivity estimation algorithm if it is needed.

Here are some numbers.

master

concise bitmap

Benchmark                                  (limit)  (numSegments)  (rowsPerSegment)  (schemaAndQuery)  Mode  Cnt       Score      Error  Units
SearchBenchmark.queryMultiQueryableIndex    750000              1            750000           basic.B  avgt   25  463864.923 ± 5935.667  us/op
SearchBenchmark.querySingleQueryableIndex   750000              1            750000           basic.B  avgt   25  471846.438 ± 5391.567  us/op

roaring bitmap

Benchmark                                  (limit)  (numSegments)  (rowsPerSegment)  (schemaAndQuery)  Mode  Cnt       Score       Error  Units
SearchBenchmark.queryMultiQueryableIndex    750000              1            750000           basic.B  avgt   25  803411.999 ± 10526.744  us/op
SearchBenchmark.querySingleQueryableIndex   750000              1            750000           basic.B  avgt   25  807268.072 ± 10779.393  us/op

patch

concise bitmap

Benchmark                                  (limit)  (numSegments)  (rowsPerSegment)  (schemaAndQuery)  Mode  Cnt       Score      Error  Units
SearchBenchmark.queryMultiQueryableIndex    750000              1            750000           basic.B  avgt   25  301427.773 ± 4794.298  us/op
SearchBenchmark.querySingleQueryableIndex   750000              1            750000           basic.B  avgt   25  285926.630 ± 2592.590  us/op

roaring bitmap

Benchmark                                  (limit)  (numSegments)  (rowsPerSegment)  (schemaAndQuery)  Mode  Cnt       Score      Error  Units
SearchBenchmark.queryMultiQueryableIndex    750000              1            750000           basic.B  avgt   25  464101.372 ± 5595.434  us/op
SearchBenchmark.querySingleQueryableIndex   750000              1            750000           basic.B  avgt   25  466196.807 ± 3428.533  us/op

Also, you can see the overhead of selectivity estimation in the following graphs.
These graphs show the results of SearchBenchmark with the basic.C query.
- Concise bitmap

- Roaring bitmap

I think the overhead of selectivity estimation is acceptable. If others agree, I'll change the default search strategy to auto.

This change is

gianm

Interesting finding. Do you have numbers on how much time is taken by looking up bitmaps, vs. computing their size, vs. unioning them? I'm curious how much unioning cost exceeds the cost of just computing all the sizes.

If unioning is really dominant then your approach here sounds good. I'd just like to reduce duplicated code and watch out for behavior with multi-value dimensions.

gianm · 2017-01-19T20:47:59Z

+  boolean supportsBitmapIndex(BitmapIndexSelector selector);
+
+  /**
+   * Estimate selectivity of this filter. The estimated selectivity might be different from the exact value.


Could you expand on what is expected of the accuracy & speed of the "estimate", and what it might be used for? This will help Filter implementors know what they should be doing for this method.

gianm · 2017-01-19T20:53:03Z

+   *
+   * @param selector Object used to retrieve bitmap indexes
+   * @param totalNumRows total number of rows in a segment
+   * @return Selectivity ranging from 0 to 1.


"0 (filter selects no rows) to 1 (filter selects all rows)"

gianm · 2017-01-19T20:53:42Z

+   * Estimate selectivity of this filter. The estimated selectivity might be different from the exact value.
+   *
+   * @param selector Object used to retrieve bitmap indexes
+   * @param totalNumRows total number of rows in a segment


totalNumRows shouldn't be necessary, it should be available through selector.getNumRows()

gianm · 2017-01-19T21:13:31Z

+      for (final String eachVal : values) {
+        matchedNumRows += selector.getBitmapIndex(dimension, eachVal).size();
+      }
+      return (double) matchedNumRows / totalNumRows;


This could be greater than 1 for multi-value dimensions.

I think some of your other implementations will have a similar problem.

I missed multi-value dimensions. Thanks.

gianm · 2017-01-19T21:15:08Z

+  @Override
+  public double estimateSelectivity(BitmapIndexSelector selector, long totalNumRows)
+  {
+    if (extractionFn == null && likeMatcher.getSuffixMatch() == LikeDimFilter.LikeMatcher.SuffixMatch.MATCH_EMPTY) {


There's a lot of copy/pasted code here, is there a way to do better?

Maybe have getBitmapIterator take some higher level args and even handle the fallback vs. optimized case internally. In the fallback case it could return an Iterators.singletonIterator of the one bitmap. Then getBitmapIndex unions them and estimateSelectivity just adds up their sizes.

The helpers in Filters.java, as well as specific filters like BoundFilter and SpatialFilter, maybe others too, could use a similar strategy to reduce duplicated code.

Oh.. I forgot refactoring these copy/pasted codes. Sorry. I'll do soon.

jihoonson · 2017-01-20T00:58:26Z

@gianm, I don't have exact numbers comparing the time taken by bitmap lookup, getting size, and bitmap union, but I think the above SearchQuery benchmark result shows the performance comparison of summing bitmap sizes and bitmap union. When just summing bitmap sizes instead of unioning them, the performance improvement ratios were about 165% and 173% for concise and roaring bitmap types, respectively.

I think unioning bitmaps is the bottleneck quite obviously. Here is a JMC profile result of SearchBenchmark.

As you can see, just unioning bitmaps contributes about 6% to the total execution time of a search query. MinMaxPriorityQueue.poll() takes most of time, and I/O (via mmap) contributes only 20% of the total cost of bitmap unions.

jihoonson · 2017-01-23T09:16:29Z

@gianm, addressed comments. Thank you for your review.

leventov · 2017-01-24T02:26:40Z

      basicQueries.put("B", queryBuilderB);
    }

+    { // basic.C


Maybe make those blocks methods basicX(), and then call them from a single init block?

leventov · 2017-01-24T02:50:23Z

+      numMatchedRows += bitmap.size();
+    }
+
+    // assume multi-value column if columnCapabilities is null


This should probably be a comment before line 280

leventov · 2017-01-24T02:50:47Z

+
+    // assume multi-value column if columnCapabilities is null
+    if (isMultiValueDimension) {
+      final double estimated = numMatchedRows * Filters.computeNonOverlapRatioFromBitmapSamples(bitmapList)


Filters. not needed, we are in the same class

leventov · 2017-01-24T02:54:22Z

+    final int sampleNum = Math.min(SAMPLE_NUM_FOR_SELECTIVITY_ESTIMATION, bitmaps.size());
+    double nonOverlapRatioSum = 0.;
+    for (int i = 0; i < sampleNum; i++) {
+      final ImmutableBitmap b1 = bitmaps.get(random.nextInt(bitmaps.size()));


I suggest to use ThreadLocalRandom.current() in this loop

leventov · 2017-01-24T03:04:37Z

  public ImmutableBitmap getBitmapIndex(BitmapIndexSelector selector)
  {
-    if (extractionFn == null && likeMatcher.getSuffixMatch() == LikeDimFilter.LikeMatcher.SuffixMatch.MATCH_EMPTY) {
+    if (directPrefixMatchable()) {


"directPrefixMatchable" is unclear to me. Maybe "dimensionEqualsPrefix"? Then the comments on lines 57 and 88 could be removed

leventov · 2017-01-24T03:06:26Z

+      @Override
+      public Iterator<ImmutableBitmap> iterator()
+      {
+        return new Iterator<ImmutableBitmap>()


Could you please apply here the same improvement? https://github.com/druid-io/druid/pull/3754/files

leventov · 2017-01-24T03:08:44Z

+    }
+  }
+
+  private static double computeNonOverlapRatioFromBitmapSamples(List<ImmutableBitmap> bitmaps)


I don't understand this method. Could you please add an explanation? Ideally add some tests.

I added java doc.

leventov · 2017-01-24T03:15:23Z

+  )
+  {
+    long numMatchedRows = 0;
+    final List<ImmutableBitmap> bitmapList = Lists.newArrayList(bitmaps);


We try to avoid materializing all bitmaps in memory at the same time. See #3642 (comment). Maybe you need to pass around IntList + a lazy int -> ImmutableBitmap function, and use it in computeNonOverlapRatioFromBitmapSamples().

cc @gianm

Fixed. Thanks.

jihoonson · 2017-01-24T11:35:24Z

@leventov, thanks for your review. I addressed comments.

leventov · 2017-01-24T23:23:34Z

+  {
+    return Filters.bitmapsFromIndexes(getBitmapIndexIterator(boundDimFilter, bitmapIndex), bitmapIndex);
+  }
+


I suggest to use specialized IntIterator. E. g. we have 100s of thousands of values in some dimensions, don't want to do boxing on this scale.

Changed. Thanks.

leventov · 2017-01-24T23:24:30Z

-      public Iterator<ImmutableBitmap> iterator()
+      public Iterator<Integer> iterator()
      {
-        return new Iterator<ImmutableBitmap>()


Replaceable with IntIterators.fromTo()

leventov · 2017-01-24T23:30:01Z

-  static double estimatePredicateSelectivity(
+  private static double estimateSelectivityOfBitmapList(
+      BitmapIndex bitmapIndex,
+      Iterable<Integer> bitmapIndexeIterable,


Iterable<ImmutableBitmap> was used only to facilitate laziness. List of integers shouldn't be lazy, so IntList could be used, and if so, ImmutableList.copyOf() is not needed in the body of this method.

Changed to IntList.

leventov · 2017-01-24T23:34:43Z

+    long numMatchedRows = 0;
+    for (Integer index : bitmapIndexes) {
+      final ImmutableBitmap bitmap = bitmapIndex.getBitmap(index);
+      numMatchedRows += bitmap.size();


Unfortunately Concise format doesn't store the size, so size() is not cheap. Roaring storing sizes of the blocks, but computing the total size of Roaring bitmap also requires some iteration and touching a lot of memory.

This problem also complicates implementation of #3878.

I think we should store bitmap sizes on segment format level and add ways to read bitmap size without creating bitmap itself.

Right, size() is not cheap. So, you mean frequent call of size() is expensive? I'll caching it.

leventov · 2017-01-24T23:43:49Z

    if (isMultiValueDimension) {
-      final double estimated = numMatchedRows * Filters.computeNonOverlapRatioFromBitmapSamples(bitmapList)
-                               / totalNumRows;
+      final double estimated = numMatchedRows * computeNonOverlapRatioFromFirstNBitmapSamples(


It seems to me that the math model of this expression and computeNonOverlapRatioFromRandomBitmapSamples() is wrong. Imagine there are 10 bitmaps each of 0.1 selectivity, and they all overlap 100%. computeNonOverlapRatioFromRandomBitmapSamples() will return 0.5, estimated will be 10 * 0.1 * 0.5 = 0.5, while it should be 0.1.

Oops. I'm such an idiot. It's my bad. I'll change this method to calculate the average non-overlap ratio of the right hand side bitmap of the union operation like below.
(The below is just an idea, and not optimized)

while (...) { overlapSize = size(unionedBitmap) + size(rightBitmap) - size(union(unionedBitmap, rightBitmap)) unionedBitmap = union(unionedBitmap, rightBitmap) nonOverlapRatioOfRightBitmap = (size(rightBitmap) - overlapSize) / rightBitmap.size() nonOverlapRatioSum += nonOverlapRatioOfRightBitmap } return nonOverlapRatioSum / sampleNum;

And, estimateSelectivityOfBitmapList() calling this method will be changed like below.

double numMatchedRows = bitmapIndex.getBitmap(bitmapIndexes.get(0)).size(); for (int i = 1; i < bitmapIndexes.size(); i++) { final ImmutableBitmap bitmap = bitmapIndex.getBitmap(bitmapIndexes.get(i)); numMatchedRows += bitmap.size() * nonOverlapRatio; } return numMatchedRows / totalNumRows

I'll do add some tests.

leventov · 2017-01-24T23:45:30Z

-      final ImmutableBitmap b1 = bitmaps.get(random.nextInt(bitmaps.size()));
-      final ImmutableBitmap b2 = bitmaps.get(random.nextInt(bitmaps.size()));
+      final ImmutableBitmap b1 = bitmapIndex.getBitmap(
+          bitmapIndexes.get(ThreadLocalRandom.current().nextInt(bitmapIndexes.size())));


I suggest to extract ThreadLocalRandom.current() as a local variable outside the loop

If expression is wrapped after brace in Druid's style guide the pairing expression should be on another line, i. e. ); should be on a separate line

Thanks. I'm still not familiar with Druid's style, and making mistakes a lot. I'll be more careful.

leventov · 2017-01-24T23:47:01Z

+      final ImmutableBitmap b1 = bitmapIndex.getBitmap(
+          bitmapIndexes.get(ThreadLocalRandom.current().nextInt(bitmapIndexes.size())));
+      final ImmutableBitmap b2 = bitmapIndex.getBitmap(
+          bitmapIndexes.get(ThreadLocalRandom.current().nextInt(bitmapIndexes.size())));


leventov · 2017-01-24T23:55:42Z

@@ -328,43 +457,84 @@ private static Iterable<ImmutableBitmap> makePredicateQualifyingBitmapIterable(
      public Iterator<ImmutableBitmap> iterator()
      {


Replaceable with Iterators.transform()

leventov · 2017-01-25T00:03:06Z


+  @Override
+  public double estimateSelectivity(ColumnSelector columnSelector, BitmapIndexSelector indexSelector)
+  {


This assumption could be "checked" approximately the same way as in computeNonOverlapRatioFromFirstNBitmapSamples()

Would you tell me more detailed idea?

Ah,I got your point. But this is a very basic assumption for estimating selectivity in databases and there are many more improvements (https://scholar.google.co.kr/scholar?ion=1&espv=2&um=1&ie=UTF-8&lr&q=related:A7vzlp4zVKU7uM:scholar.google.com/). I think we need to improve selectivity estimation later when druid is ready for providing more statistics like histogram, but it looks good for now. What do you think?

My concern is that for somebody this assumption could be wrong, and some selectivity computed incorrectly, and the overall select query chooses suboptimal strategy and performs worse.

Ultimately in this PR, I want to ensure that there won't be performance regression

Yes, this assumption might be wrong, maybe in many real applications. However, in that cases, users can simply use other search strategies like UseIndexStrategy or CursorOnlyStrategy. This is why we still use UseIndexStrategy as the default strategy.

Let's ensure there won't be regression if users don't touch any configurations, just update druid binary to the version which includes this patch. "No regression if user has to manually change some config" is a regression, IMO

@leventov I believe that is the case already: this code only gets activated if the search strategy is "auto", which is not the default strategy. It also doesn't exist in the current release, so regression is impossible.

- Fix wrong non-overlap ratio computation and added unit tests. - Change Iterable<Integer> to IntIterable - Remove unnecessary Iterable<Integer>

jihoonson · 2017-01-25T09:27:16Z

@leventov thanks for your detailed review. I addressed comments.

jihoonson · 2017-01-25T13:23:39Z

The Travis failure is due to Kafka connection timeout and looks not related to this issue.

leventov · 2017-01-26T00:53:08Z

-      final ImmutableBitmap bitmap = bitmapIndex.getBitmap(index);
-      numMatchedRows += bitmap.size();
+    double numMatchedRows = bitmapIndexes.size() > 0 ? bitmapIndex.getBitmap(bitmapIndexes.get(0)).size() : 0;
+    final double nonOverlapRatio = isMultiValueDimension && bitmapIndexes.size() > 1


Could you please split this into if-else blocks? Chains of ternary operators in one expression are unreadable

leventov · 2017-01-26T01:03:02Z

   * @see #estimatePredicateSelectivity(ColumnSelector, String, BitmapIndexSelector, Predicate)
   */
-  private static double computeNonOverlapRatioFromRandomBitmapSamples(
+  static double computeNonOverlapRatioFromRandomBitmapSamples(


Please add @VisibleForTesting in such cases

I don't believe this model as well. It computes mean of values, which are position-dependent (discount in time). Consider you have two bitmaps, one covers 10% of bits, another 100%. If you select them in one order you will get 0.0 from this method, in another order - 0.9.

Another problem with this estimation, is that is computes pairwise unions, i. e. generates a lot of garbage.

I suggest to not try compute selectivity with multi-value dimensions in this PR.

Later, I propose the following approach to be implemented:

Compute a sample union of N randomly selected bitmaps (not one-by-one, union of all of them together)

Based on 1) size of the sample union 2) sizes of all unified bitmaps 3) total number of rows 4) number and sizes of bitmaps not included in the sample union, compute the expected size of union of all bitmaps. A non-trivial math model is required to do this, and I don't expect it could be done without help or at least review by somebody who remembers statistics course from high school really well.

Reuse the computed sample union later in Filter/UseIndexesStrategy, when the union of all bitmaps is needed (and if needed).

Also for efficiency of the union size estimation, some parameters could be pre-computed when segment is created and stored in the segment format. For example, the sum of sizes of all bitmaps (which is bigger than the number of rows, for multi-value dimensions). This is related to #3882.

First in your example, non-overlap ratios will be 0.0 and 0.9. They cannot be negatives because bSize - overlapSize = unionedSize - preUnionedSize >= 0 always holds which means bSize >= overlapSize.

I agree on that there will be more efficient and good models for this. There have been a lot of studies on this issue, and we need to investigate them and choose one. However, I'm not sure about completely disabling this feature for multi-value dimensions. Users can just use other strategies until we improve this sufficiently. Isn't it?

On the other hand, it would be ok to disable this feature because it works well under very strict conditions. I'm open to this.

I vote for disabling this feature now not because the model yields very imprecise results (it's ok as long as there is no regression compared to the state prior to this PR), but because it generates a lot of garbage. The default bitmap algorithm is concise. Pairwise union of 100 bitmaps creates 100 distinct bitmaps, which immediately become garbage. Union of 100 bitmaps at once creates only one resulting bitmap.

Agree. I disabled it.

leventov · 2017-01-26T01:40:38Z

+      final BitmapIndex bitmapIndex
+  )
+  {
+    return IntIteratorUtils.toIntList(getBitmapIndexIterator(boundDimFilter, bitmapIndex).iterator());


Something like RangeIntList(from, to) could be implemented and used here:

class RangeIntList extends AbstractIntList { final int start, end; final int size RangeIntList(int start, int end) { this.start = start; this.end = end; this.size = end - start; } @Override public int getInt(int index) { Preconditions.checkElementIndex(index, size); return start + index; } @Override public int size() { return size; } }

Thanks. I added a new util class IntListUtils for this class, but am not sure about its location..

…r-selectivity-estimation

- Split a long ternary operation into if-else blocks - Add IntListUtils.fromTo()

jihoonson · 2017-01-26T04:18:12Z

I'm investigating test failures.

…r-selectivity-estimation

jihoonson · 2017-01-27T00:22:11Z

Fixed test failures.

leventov · 2017-01-27T05:09:12Z

+
+  public static IntList fromTo(int from, int to)
+  {
+    return new RangeIntList(from, to);


Please add a check that from <= to

I also would like to do, but BoundFilter.getStartEndIndexes() sometimes returns a smaller to than from and it makes some tests failed. I added a comment.

leventov · 2017-01-27T19:13:35Z


  public static IntList fromTo(int from, int to)
  {
+    // TODO: check `from` is always smaller than or equal to `to`.


I don't think it's a good idea to skip the check because of a single use case. If to could be less than from on BoundFilter side, we can add a check on that side and return IntLists.EMPTY_LIST.

Also, in Druid we try to avoid adding TODOs, add issues instead, even for relatively small things. And too small things should probably be fixed in-place. See https://github.com/metamx/java-util/pull/50/files#r77869324

Right. I added the check and fixed test failure.

jihoonson · 2017-01-30T23:41:39Z

@leventov, thank you for your detailed review.
@gianm, any comments?

gianm · 2017-02-03T22:07:19Z

Taking another look now, @jihoonson

leventov · 2017-02-06T04:33:13Z

      }
    }

+    endIndex = startIndex > endIndex ? startIndex : endIndex;


Math.max(startIndex, endIndex)

gianm · 2017-02-06T19:15:01Z

LGTM

leventov · 2018-10-31T12:43:53Z

+    dimFilters.add(new SelectorDimFilter(dimName, "3", null));
+    dimFilters.add(new BoundDimFilter(dimName, "100", "10000", true, true, true, null, null));
+    dimFilters.add(new InDimFilter(dimName, dimUniformFilterVals, null));
+    dimFilters.add(new InDimFilter(dimName, dimUniformFilterVals, null));


@jihoonson did you intentionally add three identical lines in a row?

It looks a mistake. Thanks for finding.

Add filter selectivity estimation for auto search strategy

f02fc66

gianm requested changes Jan 19, 2017

View reviewed changes

leventov self-assigned this Jan 19, 2017

Addressed comments

587ecfe

leventov requested changes Jan 24, 2017

View reviewed changes

gianm mentioned this pull request Jan 24, 2017

Filters on high cardinality dimensions should sometimes use dim index bitset + full scan instead of unioning bitsets of dim values #3878

Closed

Lazy bitmap materialization for bitmap sampling and java docs

7dc4e4f

leventov requested changes Jan 25, 2017

View reviewed changes

leventov mentioned this pull request Jan 25, 2017

Store bitmap sizes along with bitmaps in the segment format #3882

Closed

Addressed comments.

c121c62

- Fix wrong non-overlap ratio computation and added unit tests. - Change Iterable<Integer> to IntIterable - Remove unnecessary Iterable<Integer>

leventov reviewed Jan 26, 2017

View reviewed changes

jihoonson added 2 commits January 26, 2017 11:09

Merge branch 'master' of https://github.com/druid-io/druid into filte…

cdca521

…r-selectivity-estimation

Addressed comments

010fe7b

- Split a long ternary operation into if-else blocks - Add IntListUtils.fromTo()

jihoonson added 3 commits January 26, 2017 14:34

Merge branch 'master' of https://github.com/druid-io/druid into filte…

e790f0c

…r-selectivity-estimation

Fix test failure and add a test for RangeIntList

606abff

fix code style

71eb14f

leventov requested changes Jan 27, 2017

View reviewed changes

Diabled selectivity estimation for multi-valued dimensions

fcd6f07

leventov reviewed Jan 27, 2017

View reviewed changes

Address comment

92da826

leventov approved these changes Jan 28, 2017

View reviewed changes

leventov reviewed Feb 6, 2017

View reviewed changes

leventov removed their assignment Feb 6, 2017

gianm approved these changes Feb 6, 2017

View reviewed changes

gianm merged commit ddd8c9e into apache:master Feb 6, 2017

gianm added this to the 0.10.0 milestone Feb 6, 2017

gianm added the Feature label Feb 6, 2017

gianm mentioned this pull request Feb 6, 2017

Simplify LikeFilter implementation of getBitmapIndex, estimateSelectivity. #3910

Merged

leventov reviewed Oct 31, 2018

View reviewed changes

leventov mentioned this pull request Nov 8, 2018

Find duplicate lines with checkstyle; enable some duplicate inspections in IntelliJ #6558

Merged

		@@ -328,43 +457,84 @@ private static Iterable<ImmutableBitmap> makePredicateQualifyingBitmapIterable(
		public Iterator<ImmutableBitmap> iterator()
		{

Conversation

jihoonson commented Jan 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Jan 20, 2017

Uh oh!

jihoonson commented Jan 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leventov Jan 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Jan 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jihoonson commented Jan 13, 2017 •

edited

Loading

leventov Jan 24, 2017 •

edited

Loading

jihoonson Jan 25, 2017 •

edited

Loading