Add CliIndexer process type and initial task runner implementation by jon-wei · Pull Request #8107 · apache/druid

jon-wei · 2019-07-18T22:20:13Z

First part of implementation for #7900

Description

This PR adds the initial foundation for the work in #7900:

A new Indexer process type is added, running on port 8091 like MiddleManagers.
This process type is like MM but uses a new ThreadingTaskRunner, which runs Tasks using a thread pool instead of forking processes
A new interface, AppenderatorsManager, which creates and manages Appenderators for tasks, has been added. The AppenderatorsManager for the new Indexer process uses a shared query execution pool and per-datasource segment walkers/imelines for the Appenderators it creates.
The TaskReportWriter has been adjusted to support managing multiple tasks
The new Indexer process serves requests on a single port, but divides the HTTP connection pool into 3 parts: one pool of connections for chat handler requests, one pool of connections for non chat handler requests, and 2 reserved connections for lookup listening. Each of the first two pools is sized to druid.server.http.numThreads. The distinct connection pooling is handled through Jetty QoS filters.
Lookups are loaded once for the entire Indexer process

No additional memory management for the tasks is present in this PR, they will use whatever has been configured in their task specs with no additional external control.

The new process is not currently exposed in the docs, as this is an intermediate PR and the process is not ready for use yet.

This PR has:

been self-reviewed.
- using the concurrency checklist
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths.
been tested in a test Druid cluster.

clintropolis

Had a first pass, also pulled the branch and fired one of these things up, and (after setting druid.service) it worked for me! 🤘

clintropolis · 2019-07-19T01:16:05Z


    synchronized (lock) {
      try {
+        // When stopping, the task status should not be communicated to the overlord, so the listener and exec


thanks for adding comment 👍

clintropolis · 2019-07-19T01:23:32Z

+  @Override
+  public void removeAppenderatorForTask(String taskId)
+  {
+


can you add a // nothing to remove or something so this doesn't look so sad?

Added a comment

clintropolis · 2019-07-19T07:08:28Z

-    binder.bind(DruidNode.class).annotatedWith(RemoteChatHandler.class).to(Key.get(DruidNode.class, Self.class));
+    if (useSeparatePort) {
+      // bind a modified DruidNode that will be used by the Jetty server installed below
+      binder.bind(DruidNode.class)


This ends up being unfortunate in that it requires that druid.service be set since the 'default' value is not available on the binding at the point that this runs, however I'm not sure how to better fix it 😢

Would it be possible to internally use a QoS filter with jetty to leave some room for chat handler traffic instead relying on a separate server with a separate pool? I don't think this would require bubbling up to the user configs, just something we compute based on the top level configs (though it might require some minimum).

I think it would simplify things both conceptually and for the implementation, and would make this no longer an issue I think.

I redid this to use QoS filters, the configured number of connections are doubled (one set for chat handler, the other for non-chat handler requests), with 2 reserved connections for lookups

clintropolis · 2019-07-19T07:08:58Z

+ */
+@Command(
+    name = "indexer",
+    description = "Runs an Indexer. Description TBD."


Don't forget to add a description

Added description

clintropolis · 2019-07-19T07:12:59Z

+                .addBinding("local")
+                .to(LocalTaskActionClientFactory.class)
+                .in(LazySingleton.class);
+            // all of these bindings are so that we can run the peon in local mode


This comment isn't quite accurate, this should probably refer to indexer. Additionally, there are lot of bindings getting wired up shared between this and peon, with this entire block looking like a straight copy and paste for one, is it possible to restructure stuff so this can be shared between them?

Moved some of the shared binding blocks into methods on CliPeon/CliMiddleManager for now

clintropolis · 2019-07-19T08:13:37Z

+{
+  private Appenderator realtimeAppenderator;
+
+  public TestAppenderatorsManager()


Does this constructor need to exist?

Removed unnecessary constructor

clintropolis · 2019-07-19T08:21:40Z


-      toolbox.getDataSegmentServerAnnouncer().unannounce();
-      toolbox.getDruidNodeAnnouncer().unannounce(discoveryDruidNode);
+      if (appenderatorsManager.shouldTaskMakeNodeAnnouncements()) {


Should we just make the AppenderatorsManager handle task announcements entirely for tasks, and instead here just call methods on it signaling to do so or not? I guess sort of awkward with the way things are currently since the announcers are fetched from the toolbox

I had done that initially, but decided to just use a boolean since the CliIndexer implementation would've done nothing for those methods

clintropolis · 2019-07-19T08:23:56Z

+ */
+public class UnifiedIndexerAppenderatorsManager implements AppenderatorsManager
+{
+  private static final Logger log = new Logger(UnifiedIndexerAppenderatorsManager.class);


this log isn't used, also I think we are supposed to make static variables all uppercase.

Removed unused logger

clintropolis · 2019-07-19T08:24:33Z

+  private static final Logger log = new Logger(UnifiedIndexerAppenderatorsManager.class);
+
+  private final ConcurrentHashMap<String, SinkQuerySegmentWalker> datasourceSegmentWalkers = new ConcurrentHashMap<>();
+  private final ConcurrentHashMap<String, Appenderator> taskAppenderatorMap = new ConcurrentHashMap<>();


Stuff is only put and removed from here, but never read?

It's something that would've been expanded in follow on PRs, but I removed it for now

clintropolis · 2019-07-19T08:25:31Z

+@ManageLifecycle
+public class UnifiedIndexerLifecycleHandler
+{
+  private static final EmittingLogger log = new EmittingLogger(UnifiedIndexerLifecycleHandler.class);


I think static variables are supposed to be all caps

Changed to all caps

jihoonson · 2019-07-22T06:58:30Z

Can't be null here since taskWorkItem.shutdown is already checked above. I guess this should be checked first before checking taskWorkItem.shutdown?

Nice catch, I fixed this in ForkingTaskRunner as well

clintropolis · 2019-07-23T06:49:35Z

+import java.util.concurrent.Executor;
+
+/**
+ * Base class for {@link ForkingTaskRunner} and {@link ThreadingTaskRunner} which support task restoration.


clintropolis · 2019-07-23T07:01:33Z

+  }
+
+  @Override
+  public Collection<TaskRunnerWorkItem> getRunningTasks()


Any reason getRunningTasks, getPendingTasks, and getRunnerTaskState are not abstract instead of default explodey? BaseRestorableTaskRunner doesn't have any abstract other than calling the class that currently, seems like

@Override public abstract Collection<TaskRunnerWorkItem> getRunningTasks(); @Override public abstract Collection<TaskRunnerWorkItem> getPendingTasks(); @Nullable @Override public abstract RunnerTaskState getRunnerTaskState(String taskId);

would be appropriate.

Good point, fixed

clintropolis · 2019-07-23T07:03:24Z

+ *   Only one shutdown per-task can be running at a given time, and there is one control thread per task,
+ *   thus the pool has 2 * worker slots.
+ *
+ * Note that separate task logs are not supported, all task log entries will be written to the Indexer process log


It does seem like it would be very tricky to retain having individual task logs that get pushed to deep storage, involving some sort of log4j wizardry unless I'm missing something clever.

clintropolis · 2019-07-23T07:07:51Z

+import java.util.concurrent.ExecutorService;
+
+/**
+ * This interface defines entities that create and manage potentially multiple Appenderators.


nit: could you add javadoc links to things like Appenderator and other types? I find them handy for easily navigating to things in intellij when I'm reading about how stuff works

clintropolis · 2019-07-23T07:09:29Z

+
+import java.util.concurrent.ExecutorService;
+
+public class PeonAppenderatorsManager implements AppenderatorsManager


Could you add javadoc for this guy?

Added javadocs (also added javadoc links in AppenderatorsManager)

clintropolis · 2019-07-23T08:27:21Z

-
  public static final int DEFAULT_GZIP_INFLATE_BUFFER_SIZE = 4096;

+  public ServerConfig(


Should this be done as @JsonCreator to get rid of the need for the empty constructor or is there something in this that doesn't come from json?

The empty constructor is still used by some tests, I added a comment about how the new non-empty constructor is used.

I'm surprised that serde works fine with two constructors and without anyone marked @JsonCreator ... how does ObjectMapper decide which constructor to use for deserialization ?

I'm not sure on the exact details of what ObjectMapper does to deserialize with those constructors, but I added a new ServerConfigSerdeTest that checks this behavior

clintropolis · 2019-07-23T08:31:35Z

        binder,
        ListenerResource.BASE_PATH + "/" + LookupCoordinatorManager.LOOKUP_LISTEN_ANNOUNCE_KEY,
-        2 // 1 for "normal" operation and 1 for "emergency" or other
+        LOOKUP_LISTENER_QOS_MAX_REQUESTS // 1 for "normal" operation and 1 for "emergency" or other


Heh neat, is this documented anywhere I wonder?

It wasn't, I added a brief mention of it to the lookup docs

clintropolis

lgtm 👍

clintropolis · 2019-07-24T00:38:04Z

 The query serving processes have an internal API for managing lookups on the process and those are used by the Coordinator.
 The Coordinator periodically checks if any of the processes need to load/drop lookups and updates them appropriately.

+Please note that only 2 simultaneous lookup configuration propagation requests can be concurrently handled by a single query serving process, to prevent lookup handling from consuming too many server HTTP connections.


optional: I think this is a very dense sentence, it might be friendlier to break up at the comma, maybe:

Please note that only 2 simultaneous lookup configuration propagation requests can be concurrently handled by a single query serving process. This is done to prevent lookup handling from consuming too many server HTTP connections.

Thanks for adding it to the docs though 👍

clintropolis · 2019-07-24T00:39:40Z

+    }
+  }
+
+  // Save running tasks to a file, so they can potentially be restored on next startup. Suppresses exceptions that


convert to javadocs?

clintropolis · 2019-07-24T06:18:29Z

I did some more testing with this on my laptop with a setup of 1 of each broker, router, coordinator, overlord, and 2 indexer and historicals

Doing small scale some kafka indexing testing to make sure realtime queries and handoff were functioning

Overall things are working nicely. I did run into an issue when trying to stop an indexer node (SIGTERM), I believe the issue lies with the order of lifecycle shutdown, in that the tasks are gracefully stopped after jetty is stopped. This causes the lifecycle stop on the indexer to hang during graceful task stop, because the task is waiting to hear from the overlord a message it will never be able to hear without a running jetty.

The supervisor on the overlord is then forever stuck in a loop, performing an action it can never complete because the indexer has stopped listening.

2019-07-24T01:58:21,268 INFO [KafkaSupervisor-wikipedia] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - {id='wikipedia', generationTime=2019-07-24T01:58:21.268Z, payload=KafkaSupervisorReportPayload{dataSource='wikipedia', topic='wikipedia', partitions=1, replicas=2, durationSeconds=600, active=[{id='index_kafka_wikipedia_ed020815fc3c3f4_bebmfiod', startTime=2019-07-24T01:55:50.422Z, remainingSeconds=449}, {id='index_kafka_wikipedia_ed020815fc3c3f4_cfcibngo', startTime=2019-07-24T01:55:50.525Z, remainingSeconds=449}], publishing=[], suspended=false, healthy=true, state=RUNNING, detailedState=RUNNING, recentErrors=[]}}
2019-07-24T01:58:36,268 INFO [IndexTaskClient-wikipedia-0] org.apache.druid.indexing.common.IndexTaskClient - submitRequest failed for [http://localhost:8092/druid/worker/v1/chat/index_kafka_wikipedia_ed020815fc3c3f4_bebmfiod/offsets/current], with message [Connection refused (Connection refused)]

The indexer eventually gives up after 5 minute timeout and ungracefully stops, but the supervisor/overlord appears to remain stuck until either the indexer comes back on the same host/port or the overlord is restarted. This also jams up what the web ui displays as the task status, where the task of the stuck indexer remains in the 'running' state until the same condition of the indexer returning or the overlord is restarting is met.

This issue aside, I'm still +1 on this if you'd rather fix this in a follow-up PR, since this is currently an undocumented feature anyway.

jon-wei · 2019-07-24T23:10:51Z

@clintropolis

I observed the Jetty server shutdown before task shutdown in a Peon too, that may be worth looking into as a separate issue.

The hanging you observed was due to SeekableStreamIndexTask not doing any shutdown actions if druid.indexer.task.restoreTasksOnRestart=false, so the indexer just waited for the graceful shutdown timeout before interrupting the task thread.

I've updated the patch to have the stopGracefully() methods on the tasks interrupt their run method

himanshug · 2019-07-25T22:57:58Z

        }
    );

+    DruidNodeDiscovery druidNodeDiscoveryIndexer = druidNodeDiscoveryProvider.getForNodeType(NodeType.INDEXER);


Instead of repeated code you can change line 441 to

DruidNodeDiscovery druidNodeDiscovery = druidNodeDiscoveryProvider.getForService(WorkerNodeService.DISCOVERY_SERVICE_KEY);

I changed this to use the suggested code.

It looks like WorkerNodeService.DISCOVERY_SERVICE_KEY was incorrectly mapped to NodeType.PEON in DruidNodeDiscoveryProvider previously, I also changed that to the MM and indexer types.

himanshug · 2019-07-26T00:47:42Z

+  // Synchronizes start/stop of this object.
+  private final Object startStopLock = new Object();
+
+  private volatile boolean started = false;


there is a LifecycleLock utility class to manage lifecycle instead of startStopLock and started flag, that can be used here.

Adjusted this to use LifecycleLock

himanshug · 2019-07-26T00:53:12Z

+   * It is used by UnifiedIndexerAppenderatorsManager which allows queries on data associated with multiple
+   * Appenderators.
+   */
+  AppenderatorImpl(


can other constructor use this constructor instead of repeating everything ?

Adjusted the constructors so one calls the other, the sink timeline handling in the constructor I added was also changed to accommodate that

himanshug · 2019-07-26T17:48:06Z

+
+/**
+ * Base class for {@link ForkingTaskRunner} and {@link ThreadingTaskRunner} which support task rest
+ * oration.


nit: can we keep "restoration" as single word

Whoops, fixed

himanshug · 2019-07-26T17:53:11Z

+  public abstract Collection<TaskRunnerWorkItem> getPendingTasks();
+
+  /**
+   * The concrete implementation should override this, as the implemention depends on the specifics of the


nit: comments on these 3 methods have no value as those methods are already declared abstract. if we do, the comment should be about what is expected from implementors or no comment.

Removed the comments

himanshug · 2019-07-26T17:59:18Z

+  }
+
+  @Override
+  public void write(String taskId, Map<String, TaskReport> reports)


it is weird that instance of this class will retain and manage a map of taskId -> file (because it inherits TaskReportFileWriter) but those would be ignored all time.
TaskReportWriter probably can be trimmed down a lot more, declared abstract/interface and have separate MultiFileTaskReportFileWriter wherever TaskReportWriter is currently used.

I redid these classes, there's now a TaskReportFileWriter interface with three implementations:

SingleFileTaskReportFileWriter

MultipleFileTaskReportFileWriter

NoopTestTaskReportFileWriter

himanshug · 2019-07-26T22:07:57Z

+ *   Only one shutdown per-task can be running at a given time, and there is one control thread per task,
+ *   thus the pool has 2 * worker slots.
+ *
+ * Note that separate task logs are not supported, all task log entries will be written to the Indexer process log


is this a choice we are making intentionally to keep log in same place or no separation is done because we haven't figured it out yet ?
if we do end up keeping all logs in same place then it would be great if taskId is included in the log line printed by specific log, so as to be able to get some sense of a task by just grepping for its taskId.

himanshug · 2019-07-26T22:18:25Z

+                                    task.getId(),
+                                    TaskStatus.running(task.getId())
+                                );
+                                taskStatus = taskStatusFuture.get();


not sure why we are running task in a separate thread while this thread is waiting for the task to finish anyway ?

if this is done for making calls to notifyXXX() then they could be done right before calling task.run(toolbox) if task was run in this thread itself ?

It was partly for making the notify calls after the run started, and partly because I wanted to have a similar structure to ForkingTaskRunner where the control context and the task code were separated by thread/process, but it's not really necessary to do that.

I changed the task setup and run() to execute using a thread in taskExecutor, the control thread pool is only used to run externally triggered task shutdown now (and there is only 1 control thread per task slot now)

himanshug · 2019-07-26T22:22:25Z

+                          );
+
+                          try {
+                            final Closer closer = Closer.create();


couldn't understand why we have this Closer object, nothing appears to be accumulated in it for closing.

Thanks, removed this, it was a relic from ForkingTaskRunner

himanshug · 2019-07-26T22:26:46Z

+
+                          try {
+                            final Closer closer = Closer.create();
+                            try {


do we really need two try blocks ... line 164 and 166 , could they be folded in to one ?

Collapsed into a single try

himanshug · 2019-07-26T22:33:35Z

+
+                if (status == null) {
+                  if (taskInfo.thread != null) {
+                    taskInfo.thread.interrupt();


is it possible that the interrupt status of the thread is not cleared by this task if it wasn't blocked waiting on something. in that case same thread with interrupt status set would pick a new task ?

also I see that you have also updated some of the tasks to capture their own threads on run and interrupt them inside stopGracefully(..) .... one of these appear redundant, no ?

For the first point, I added a call to Thread.interrupted() at the very end of the outermost finally in the task callable.

For the second point, the interrupt here is meant as a "last resort" if the graceful shutdown didn't complete for some reason within the configured timeout, the interrupt within the tasks should normally end the shutdown before this is called

I might be wrong but it appears to me that whole business with the thread is unnecessary and you can achieve same by calling ThreadingTaskRunnerWorkItem.shutdownFuture.cance(true) and you wouldn't have to deal manually with interrupts. That would also remove task from executor if it hasn't started running already.

sidenote: I see that you are trying to simulate force stop of a task which is impossible to do in this context. If task is not cooperative, nothing is gonna work unfortunately. in the long run, this taskRunner requires adding another method to Task interface to tell task to force stop itself.

also, since we are interrupting the thread from outside, doing same in the task implementations is not achieving anything extra and should probably be removed, that just introduces confusion with no benefit.... or maybe I am still missing something :)

himanshug · 2019-07-26T22:35:27Z

+      return null;
+    } else {
+      if (workItem.getThread() == null) {
+        return RunnerTaskState.PENDING;


can't it mean that task is finished as we set thread to null when task finishes ?

I forget if I had the thread set to null after task finishes in earlier revisions, but currently the thread doesn't get nulled out after task completion

himanshug · 2019-07-29T04:18:43Z

+                              taskStatus = task.run(toolbox);
+                            }
+                            catch (Throwable t) {
+                              LOGGER.info(t, "Exception caught while running the task.");


Thanks, fixed

himanshug · 2019-07-29T04:19:10Z

+                            return taskStatus;
+                          }
+                          catch (Throwable t) {
+                            LOGGER.info(t, "Exception caught during execution");


himanshug · 2019-07-29T04:34:30Z

+
+                if (status == null) {
+                  if (taskInfo.thread != null) {
+                    taskInfo.thread.interrupt();


I might be wrong but it appears to me that whole business with the thread is unnecessary and you can achieve same by calling ThreadingTaskRunnerWorkItem.shutdownFuture.cance(true) and you wouldn't have to deal manually with interrupts. That would also remove task from executor if it hasn't started running already.

sidenote: I see that you are trying to simulate force stop of a task which is impossible to do in this context. If task is not cooperative, nothing is gonna work unfortunately. in the long run, this taskRunner requires adding another method to Task interface to tell task to force stop itself.

jon-wei · 2019-07-29T19:07:48Z

@himanshug

I changed the interrupt to use future.cancel(true), and added a comment about how it's not possible to truly force a hard shutdown there (my intent in putting it there was for hypothetical cases where a task's stopGracefully takes too long or has a bug and doesn't result in a shutdown, but the task is still interruptible)

…move thread from workitem

himanshug · 2019-07-29T20:50:27Z

+                  // Note that we can't truly force a hard termination of the task thread externally.
+                  // In the future we may want to add a forceful shutdown method to the Task interface.
+                  if (taskInfo.shutdownFuture != null) {
+                    taskInfo.shutdownFuture.cancel(true);


we need to cancel status future and not shutdownFuture.
also Future.get(..) throws TimeoutException instead of null, so this will not be called if task didn't finish within timeout
also, we can remove keeping the thread as "state"

anyways, I thought it would be easier to write code :) , can you merge jon-wei#4 which will add a commit making above changes ?

Ah, my bad, cancelled the wrong future.

Thanks for the patch!

ThreadingTaskRunner: cancel task run future not shutdownFuture and remove thread from workitem

jon-wei · 2019-07-30T00:20:55Z

@clintropolis @jihoonson @himanshug Thanks a lot for the review!

Add CliIndexer process type and initial task runner implementation

43cb678

jon-wei added Feature Area - Batch Ingestion Design Review Area - Streaming Ingestion labels Jul 18, 2019

jon-wei mentioned this pull request Jul 18, 2019

Develop a new Indexer process for running ingestion tasks #7900

Open

Fix HttpRemoteTaskRunnerTest

54c2424

clintropolis requested changes Jul 19, 2019

View reviewed changes

jon-wei added 2 commits July 19, 2019 13:28

Remove batch sanity check on PeonAppenderatorsManager

cd9f9d4

Fix paralle index tests

780a600

jihoonson reviewed Jul 22, 2019

View reviewed changes

PR comments

7fd2e4b

jon-wei force-pushed the big_indexer branch from 86963db to 7fd2e4b Compare July 22, 2019 23:19

jon-wei added 7 commits July 22, 2019 16:22

Merge remote-tracking branch 'upstream/master' into big_indexer

8835b0b

Adjust Jersey resource logging

c5f7a7d

Additional cleanup

05af951

Fix SystemSchemaTest

f4cbf14

Add comment to LocalDataSegmentPusherTest absolute path test

d3e18f9

More PR comments

f7b182f

Use Server annotated with RemoteChatHandler

37ef2aa

clintropolis reviewed Jul 23, 2019

View reviewed changes

jon-wei added 3 commits July 23, 2019 15:10

More PR comments

a748e26

Merge remote-tracking branch 'upstream/master' into big_indexer

8525429

Checkstyle

a3cc12f

clintropolis approved these changes Jul 24, 2019

View reviewed changes

jon-wei added 3 commits July 24, 2019 11:05

PR comments

564bbce

Merge remote-tracking branch 'upstream/master' into big_indexer

f1e6f4b

Add task shutdown to stopGracefully

6de3d1f

jon-wei added 3 commits July 24, 2019 19:58

Merge remote-tracking branch 'upstream/master' into big_indexer

8e9ce54

Small cleanup

c68bee6

Compile fix

ae63057

clintropolis approved these changes Jul 25, 2019

View reviewed changes

himanshug reviewed Jul 25, 2019

View reviewed changes

himanshug reviewed Jul 26, 2019

View reviewed changes

Address PR comments

0d17365

himanshug reviewed Jul 26, 2019

View reviewed changes

jon-wei added 4 commits July 26, 2019 16:51

Adjust TaskReportFileWriter and fix nits

3be1596

Remove unnecessary closer

bec8f65

More PR comments

01c280f

Minor adjustments

dd8f4ff

himanshug reviewed Jul 29, 2019

View reviewed changes

PR comments

e3deef6

ThreadingTaskRunner: cancel task run future not shutdownFuture and re…

6cf986a

…move thread from workitem

himanshug reviewed Jul 29, 2019

View reviewed changes

Merge pull request #4 from himanshug/pr_8107

7609e3b

ThreadingTaskRunner: cancel task run future not shutdownFuture and remove thread from workitem

himanshug approved these changes Jul 29, 2019

View reviewed changes

himanshug merged commit 640b7af into apache:master Jul 30, 2019

clintropolis added this to the 0.16.0 milestone Aug 8, 2019

clintropolis mentioned this pull request Aug 21, 2019

0.16.0-incubating release notes #8369

Closed


		import java.util.concurrent.ExecutorService;

		public class PeonAppenderatorsManager implements AppenderatorsManager


		public static final int DEFAULT_GZIP_INFLATE_BUFFER_SIZE = 4096;

		public ServerConfig(

Conversation

jon-wei commented Jul 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jon-wei Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jon-wei commented Jul 18, 2019 •

edited

Loading

jon-wei Jul 23, 2019 •

edited

Loading