implementing profiler ELT callbacks for AMD64 Linux by sergign60 · Pull Request #7719 · dotnet/coreclr

sergign60 · 2016-10-19T10:05:05Z

This PR implements profiler ELT callbacks for AMD64 Linux. It's tested with historical debugger.
The main differences from appropriate Windows implementation:

The arguments to profile callbacks are passed via rdi & rsi registers (DONE)
It should be taken into account that rax & rdx and xmm0 & xmm1 are integer and float return registers (DONE)
Two integer registers can be used for passing arguments of structure value (DONE)
The struct _PROFILE_PLATFORM_SPECIFIC_DATA should be expanded for storing arguments registers: xmm0-xmm7 & rdi, rsi, rdx, rcx, r8, r9 ( @sivarv: it is correct to assume that on am64 (both unix and windows) the argument registers are intact (or untouched) at the point of generating Enter call back) (DONE)
The temporary registers r10 & r11 are used for temporary computation - for storing addresses of callbacks and inside them (DONE)
There is the problem to pass original values of rdi & rsi into profiling callback. This PR proposes to use r12&r13 for it
@sivarv: I don't think this (regSet.AddMaskVars(REG_R10 | REG_R11 | REG_R12 | REG_R13);) is sufficient to have R10-R13 saved in prolog.
Note then genProfilingEnterCallback() is called after OS prolog and zero init of frame. By the time we are generating Enter callback, we have already generated the method prolog that has saved all required callee saved regs.
@BruceForstall: Can you please update Documentation/botr/clr-abi.md, specifically the "# Profiler Hooks" section, to be correct, and include information about, the Linux AMD64 ABI? (DONE)

sergign60 · 2016-10-19T10:16:36Z

CC @kvochko @Dmitri-Botcharnikov

BruceForstall · 2016-10-19T17:34:55Z

There's a big "TODO" in codegenxarch.cpp, genReturn(), that it looks like you haven't addressed yet:

    // TODO-AMD64-Unix: If the profiler hook is implemented on *nix, make sure for 2 register returned structs
    //                  the RAX and RDX needs to be kept alive. Make the necessary changes in lowerxarch.cpp
    //                  in the handling of the GT_RETURN statement.
    //                  Such structs containing GC pointers need to be handled by calling gcInfo.gcMarkRegSetNpt
    //                  for the return registers containing GC refs.

BruceForstall · 2016-10-19T17:35:04Z

cc @dotnet/jit-contrib

sergign60 · 2016-10-20T09:26:57Z

@BruceForstall thanks for your comment!

Is it not enough to add push rax; push rdx; ... ; pop rdx; pop rax; into ProfileLeaveNaked for this TODO?

BruceForstall · 2016-10-20T17:00:54Z

The issue isn't the values in registers getting preserved, it is proper reporting of the GC information for those registers.

If you look at Compiler::impFixupStructReturnType(), you can see how multi-reg returns of 8-15 byte structs are massaged:

        // In case of multi-reg struct return, we force IR to be one of the following:
        // GT_RETURN(lclvar) or GT_RETURN(call).  If op is anything other than a
        // lclvar or call, it is assigned to a temp to create: temp = op and GT_RETURN(tmp).

In genReturn(), I believe that under #ifdef FEATURE_UNIX_AMD64_STRUCT_PASSING, you would need to check if the return type is struct, and then, if so, check each eight-byte to see if it is a GC type, using something like varTypeIsGC(GetEightByteType(structDesc,...)). Then, call the gcMarkRegPtrVal() / gcMarkRegSetNpt() functions like is currently done for "normal" single-register returns.

sergign60 · 2016-11-09T13:27:28Z

@BruceForstall I've placed the following code in the CodeGen::genReturn function

#ifdef FEATURE_UNIX_AMD64_STRUCT_PASSING 
       if (varTypeIsStruct(compiler->info.compRetType))
       {
            SYSTEMV_AMD64_CORINFO_STRUCT_REG_PASSING_DESCRIPTOR structDesc;
            CORINFO_CLASS_HANDLE retClsHnd = compiler->info.compMethodInfo->args.retTypeClass;
            assert(retClsHnd != NO_CLASS_HANDLE);
            compiler->eeGetSystemVAmd64PassStructInRegisterDescriptor(retClsHnd, &structDesc);
            for (unsigned int i = 0; i < structDesc.eightByteCount; i++)
            {
                if (varTypeIsGC(compiler->GetEightByteType(structDesc, i)))
                {
                    gcInfo.gcMarkRegPtrVal(REG_INTRET, compiler->GetEightByteType(structDesc, i));
                }
            }
            genProfilingLeaveCallback();
            for (unsigned int i = structDesc.eightByteCount; i > 0; i--)
            {
                  if (varTypeIsGC(compiler->GetEightByteType(structDesc, i - 1)))
                  {
                      gcInfo.gcMarkRegSetNpt(REG_INTRET);
                  }
            }
            return;
        }
#endif // FEATURE_UNIX_AMD64_STRUCT_PASSING

but I got the following assertion

Regset after BB07 gcr=00000001 {rax}, byr=00000000 {}, regVars=00000000 {}
genCodeForBBList assert: nonVarPtrRegs 1 RBM_NONE 0

Assert failure(PID 5015 [0x00001397], Thread: 5015 [0x1397]): Assertion failed 'nonVarPtrRegs == RBM_NONE' in 'System.RuntimeTypeHandle:GetMetadataImport(ref):struct' (IL size 18)

    File: .. coreclr/src/jit/codegenlinear.cpp Line: 423

Could you give me a hint to perform it by the right way? Thanks in advance

sergign60 · 2016-11-09T17:21:33Z

@BruceForstall as I see the current version of coreclr generate the following code for a method that returns struct value

G_M58419_IG08:
       488B45C0             mov      rax, gword ptr [rbp-40H]
       488B55C8             mov      rdx, qword ptr [rbp-38H]
       48BF501F3DCA357F0000 mov      rdi, 0x7F35CA3D1F50
       488D7510             lea      rsi, [rbp+10H]
       49BB005A8B43367F0000 mov      r11, 0x7F36438B5A00
       41FF13               call     qword ptr [r11]CORINFO_HELP_PROF_FCN_LEAVE
       90                   nop      

G_M58419_IG09:
       488D65F8             lea      rsp, [rbp-08H]
       415D                 pop      r13
       5D                   pop      rbp
       C3                   ret

Could you give some sample template of generated code for this case ?

sivarv · 2016-11-09T18:38:47Z

@sergign60

On unix structs of size > 8 and <=16 gets returned in two return registers RAX/XMM0 and RDX/XMM1. Consider a struct with one or two gc-ref type fields being returned. The gc-ref could be in RAX or RDX. Whichever register contains gc-ref that needs to be explicitly reported.

In the code that you have posted above you are iterating through each of the 8-bytes of the struct and always marking REG_INTRET (which is RAX) as gcptr or non-ptr. That could be the reason why you are hitting an assert. Here is the right way to do it if method is returning multi-reg return type struct

#ifdef FEATURE_UNIX_AMD64_STRUCT_PASSING
if (compiler->compMethodReturnsMultiRegRetType())
{
    ReturnTypeDesc retTypeDesc;
    retTypeDesc.InitializeStructReturnType(compiler, varDsc->lvVerTypeInfo.GetClassHandle());

   gcInfo.gcMarkRegPtrVal(retTypeDesc.GetABIReturnReg(0), retTypeDesc.GetReturnRegType(0))
   gcInfo.gcMarkRegPtrVal(retTypeDesc.GetABIReturnReg(1), retTypeDesc.GetReturnRegType(1))

   genProfilingLeaveCallback();

   gcInfo.gcMarkRegSetNpt(retTypeDesc.GetABIReturnReg(0))
   gcInfo.gcMarkRegSetNpt(retTypeDesc.GetABIReturnReg(1))
}
#endif

Apart from the generated code, what you need to verify is that gcinfo is correctly reported for return regs containing gc-refs around the call to profile leave callback.

sergign60 · 2016-11-10T11:09:45Z

@sivarv many thanks for your hint

sivarv · 2016-11-15T00:12:55Z

+                unsigned __int8 offset1 = 0;
+                var_types       type0   = TYP_UNKNOWN;
+                var_types       type1   = TYP_UNKNOWN;
+                compiler->GetStructTypeOffset(structDesc, &type0, &type1, &offset0, &offset1);


I see this code got duplicated. How about abstracting this in Compiler::GetStructTypeOffset() overload whose first argument is a CORINFO_CLASS_HANDLE and has out params to return offset and type of eightbytes of the struct?

sivarv · 2016-11-15T00:18:39Z

+    genEmitHelperCall(CORINFO_HELP_PROF_FCN_ENTER, 0, EA_UNKNOWN, REG_ARG_2);
+#else
+    genEmitHelperCall(CORINFO_HELP_PROF_FCN_ENTER, 0, EA_UNKNOWN, REG_R11);
+#endif


Is there a specific reason why we want to use REG_ARG2 on amd64 windows and REG_R11 on Unix?
Please note that without specifying a callTargetReg, genEmitHelperCall() would use REG_RAX.

sivarv · 2016-11-15T00:19:39Z

-#if FEATURE_VARARG
-        if (compiler->info.compIsVarArgs && varTypeIsFloating(loadType))
+#if defined(UNIX_AMD64_ABI)
+#ifdef FEATURE_UNIX_AMD64_STRUCT_PASSING


I think FEATURE_UNIX_AMD64_STRUCT_PASSING #ifdef alone is sufficient here.

sivarv · 2016-11-15T00:20:07Z

-            regNumber argReg    = varDsc->lvArgReg;
-            getEmitter()->emitIns_S_R(ins_Store(storeType), emitTypeSize(storeType), argReg, varNum, 0);
+#if defined(UNIX_AMD64_ABI)
+#ifdef FEATURE_UNIX_AMD64_STRUCT_PASSING


I think UNIX_AMD64_ABI is not required here.

sivarv · 2016-11-15T00:21:43Z

    genEmitHelperCall(helper, 0, EA_UNKNOWN, REG_ARG_2);
+#else
+    genEmitHelperCall(helper, 0, EA_UNKNOWN, REG_R11);
+#endif


Any specific reason why we don't want to use default call target reg (rax)?

@sivarv I guess that it would be better to use r10 & r11 for intermediate computations on Linux

sivarv · 2016-11-15T00:22:26Z

    if (compiler->compCurBB == compiler->genReturnBB)
    {
+#if defined(_TARGET_AMD64_)
+#ifdef FEATURE_UNIX_AMD64_STRUCT_PASSING


This #ifdef is redundant since FEATURE_UNIX_AMD64_STRUCT_PASSING is defined only on amd64.

I think what you mean is that the first #if is redundant, as it is implied by the #ifdef

In reply to: 87924333 [](ancestors = 87924333)

sivarv · 2016-11-15T00:24:20Z

+
+            unsigned regCount = retTypeDesc.GetReturnRegCount();
+
+            if (varTypeIsGC(compiler->info.compRetType))


if (varTypeIsGC(compiler->info.compRetType)) [](start = 1, length = 55)

This condition would always be false if the return type is a struct (i.e. TYP_STRUCT) returned in two regs.
This if-check is not required and should be deleted. This comment also applies to if-check on line 1176.

@sivarv I tried it but I have got the assertion in codegenlinear.cpp

System.RuntimeTypeHandle:GetMetadataImport(ref):struct: Regset after BB07 gcr=00000001 {rax}, byr=00000000 {}, regVars=00000000 {} Assert failure(PID 27642 [0x00006bfa], Thread: 27642 [0x6bfa]): Assertion failed 'nonVarPtrRegs == RBM_NONE' in 'System.RuntimeTypeHandle:GetMetadataImport(ref):struct' (IL size 18) File: /home/signatov/TEST/historical_debugging/coreclr/src/jit/codegenlinear.cpp Line: 422 Image: /home/signatov/TEST/historical_debugging/coreclr/bin/Product/Linux.x64.Debug/corerun

In my test case this assertion arises only on the forth method where the code with gcMarkRegPtrVal was inserted. Three methods with this code

System.RuntimeTypeHandle:GetIntroducedMethods(ref):struct IntroducedMethodEnumerator:GetEnumerator():struct:this System.RuntimeMethodHandle:GetUtf8Name(long):struct

are compiled successfully before this assertion. May be you have some ideas?

@sivarv I inserted debug prints and see the following

=== System.RuntimeTypeHandle:GetMetadataImport(ref):struct: regCount 2 00000000 00000000 in GCInfo::gcMarkRegPtrVal type 0xe TYP_LONG 0xa TYP_REF 0xe TYP_BYREF 0xf in GCInfo::gcMarkRegSetGCref GC regs: 00000000 {} => 00000001 {rax} Byref regs: (unchanged) 00000000 {} 00000001 00000000 in GCInfo::gcMarkRegPtrVal type 0xa TYP_LONG 0xa TYP_REF 0xe TYP_BYREF 0xf in GCInfo::gcMarkRegSetNpt GC regs: (unchanged) 00000001 {rax} Byref regs: (unchanged) 00000000 {} 00000001 00000000 --------->>>>>>>> in GCInfo::gcMarkRegSetNpt GC regs: (unchanged) 00000001 {rax} Byref regs: (unchanged) 00000000 {} in GCInfo::gcMarkRegSetNpt GC regs: (unchanged) 00000001 {rax} Byref regs: (unchanged) 00000000 {} System.RuntimeTypeHandle:GetMetadataImport(ref):struct: Regset after BB07 gcr=00000001 {rax}, byr=00000000 {}, regVars=00000000 {} Assert failure(PID 12181 [0x00002f95], Thread: 12181 [0x2f95]): Assertion failed 'nonVarPtrRegs == RBM_NONE' in 'System.RuntimeTypeHandle:GetMetadataImport(ref):struct' (IL size 18) File: /home/signatov/TEST/historical_debugging/coreclr/src/jit/codegenlinear.cpp Line: 422 Image: /home/signatov/TEST/historical_debugging/coreclr/bin/Product/Linux.x64.Debug/corerun

In the previous successfully compiled methods arg type is TYP_LONG.

As I see the argument of gcMarkRegSetNpt is regMaskTP not var_types, so I should replace

gcInfo.gcMarkRegSetNpt(var_types);

by

gcInfo.gcMarkRegSetNpt(genRegMask(var_types));

Am I right?

CarolEidt · 2016-11-15T00:31:35Z

    LclVarDsc* varDsc;

    // Since the method needs to make a profiler callback, it should have out-going arg space allocated.
    noway_assert(compiler->lvaOutgoingArgSpaceVar != BAD_VAR_NUM);


Shouldn't this first noway_assert also be under the following #if? I believe it's only defined under FEATURE_FIXED_OUT_ARGS which is only true for amd64.

sivarv · 2016-11-15T00:31:45Z

+                    gcInfo.gcMarkRegSetNpt(retTypeDesc.GetABIReturnReg(i));
+                }
+            }
+            return;


return; [](start = 1, length = 18)

returns in the middle of the method are very hard to notice and could lead to accidental bugs. Instead I would suggest this to restructure this logic as follows. Note that the below logic requires no #ifdefs.

e.g.

ReturnTypeDesc retTypeDesc; if (compiler->compMethodReturnsMultiRegRetType()) { CORINFO_CLASS_HANDLE retClsHnd = compiler->info.compMethodInfo->args.retTypeClass; retTypeDesc.InitializeStructReturnType(compiler, retClsHnd); } if (varTypeIsGC(compiler->info.compRetType)) { gcInfo.gcMarkRegPtrVal(REG_INTRET, compiler->info.compRetType); } else if (compiler->compMethodReturnsMultiRegRetType()) { for (unsigned i = 0; i < retTypeDesc.GetReturnRegCount(); ++i) { gcInfo.gcMarkRegPtrVal(retTypeDesc.GetABIReturnReg(i), retTypeDesc.GetReturnRegType(i)); } } genProfilingLeaveCallback(); if (varTypeIsGC(compiler->info.compRetType)) { gcInfo.gcMarkRegSetNpt(REG_INTRET); } else if (compiler->compMethodReturnsMultiRegRetType()) { for (unsigned i = 0; i < retTypeDesc.GetReturnRegCount(); ++i) { gcInfo.gcMarkRegSetNpt(retTypeDesc.GetABIReturnReg(i)); } }

@sivarv I think that it is the good idea, thanks

CarolEidt · 2016-11-15T00:38:14Z

-
+#if defined(_TARGET_AMD64_)
    // Since the method needs to make a profiler callback, it should have out-going arg space allocated.
    noway_assert(compiler->lvaOutgoingArgSpaceVar != BAD_VAR_NUM);


Here too I think that this needs to be only under !UNIX_AMD64_ABI

@CarolEidt sorry, but I didn't quite catch you. This code is for both Unix && Windows on AMD64.

CarolEidt · 2016-11-15T00:40:18Z

+//           - integer argument registers (rcx, rdx, r8, r9)
+//           - floating point argument registers (xmm1-3)
+//           - volatile integer registers (r10, r11)
+//           - volatile floating point registers (xmm4-5)


You should add here something like:

upper halves of ymm registers on AVX (which are volatile)

@CarolEidt ok, I'll insert it

CarolEidt · 2016-11-15T00:44:23Z

+  push_argument_register  rcx
+  push_argument_register  rdx
+  push_nonvol_reg         r10
+  push_nonvol_reg         rax


How are these pushed registers represented in GC info? Or does the GC "know" about these?

ELT profiler helpers are considered NO GC helpers. That is GC won't happen within the helper call.

Great - thanks for clarifying that.

sivarv · 2016-11-15T00:58:21Z

CC : @noahfalk - to review profiler helpers.

noahfalk · 2016-11-16T03:19:18Z

+  //       Upon entry :
+  //           rdi = clientInfo
+  //           rsi = profiledRsp
+


On windows these arguments are passed in rcx and rdx. Does Linux use a different 64bit ABI in general or is there a reason for the difference?

Linux has different ABI for x64. It has 6 argument registers - RDI, RSI, RDX, RCX, R8 and R9

@noahfalk According to the Unix ABI, the first 6 integer or pointer arguments to a function are passed in registers. The first is placed in rdi, the second in rsi, the third in rdx, and then rcx, r8 and r9.

thanks, makes sense then

noahfalk · 2016-11-16T03:33:46Z

+  push_nonvol_reg         r10
+  push_nonvol_reg         rax
+
+  lea                     rax, [rsp + 0x10]    // caller rsp


I rarely deal with assembly, but it doesn't seem like this would calculate the caller's rsp correctly?

At this point in the function we have already pushed r15, r14, r13, r12, rbp, rbx, rsi, rdi, r10, and rax. Did those pushes generate code that doesn't update rsp?

Right, this is not correct. It was this way in the Windows version of this function when only RAX was pushed on the stack.
@sergign60 what is the reason for pushing all the callee saved registers plus RSI, RDI and R10 when Windows version doesn't do that (I guess that's why the function name is ProfileEnterNaked)

@noahfalk @janvorli I removed all redundant register savings, added only rdx saving in Leave because of struct return ABI

noahfalk · 2016-11-16T03:42:02Z

+
+
+  // setup ProfilePlatformSpecificData structure
+  xor                     r11, r11 // nullify r11


Any reason not to use r8 so that we retain consistency with the windows implementation?

@noahfalk I guess that it would be better to use r10 & r11 for intermediate computations on Linux

makes sense now that I understand the linux ABI difference

noahfalk · 2016-11-16T03:42:52Z

+//       Upon entry :
+//           rdi = clientInfo
+//           rsi = profiledRsp
+


Same questions as above on ProfileEnter

janvorli · 2016-11-16T09:59:20Z

+//    } PROFILE_PLATFORM_SPECIFIC_DATA, *PPROFILE_PLATFORM_SPECIFIC_DATA;
+//
+.equ SIZEOF_PROFILE_PLATFORM_SPECIFIC_DATA, 0x8*11 + 0x4*2    // includes fudge to make FP_SPILL right
+.equ SIZEOF_OUTGOING_ARGUMENT_HOMES, 0x8*6


This is not right. AMD64 ABI on Linux doesn't home arguments. On Windows, there are always 4 stack slots reserved for storing the 4 argument registers before the return address. So you can just remove this constant and also the OFFSETOF_PLATFORM_SPECIFIC_DATA below, since they are both zero.

@janvorli thanks for this note.

sergign60 · 2017-03-26T14:54:13Z

@janvorli
Sorry I was ill last week.

sergign60 · 2017-04-02T18:15:05Z

@janvorli Now I get the another set of failed tests, but the reason is not this profiling callbacks implementation.

For example,

coreclr/Tests/JIT/Regression/VS-ia64-JIT/V1.2-M01/b10827/b10827

fails (segmentation fault) with

COMPlus_JitELTHookEnabled=1
COMPlus_GCStress=0xC
COMPlus_TailcallStress=1

both with the profiler's callback calls and when I commented their calls.

sergign60 · 2017-04-06T12:32:55Z

https://github.com/dotnet/coreclr/issues/10765

sergign60 · 2017-04-10T18:41:16Z

#10857

sergign60 · 2017-04-10T18:42:20Z

@janvorli I've got the results on Windows. Please look at [AMD64 Windows ELT] Some CoreCLR tests fail with ELT hooks enabled and GCStress & TailcallStress 10857

hseok-oh · 2017-04-17T16:34:58Z

@dotnet-bot test Ubuntu arm Cross Release Build

danmoseley · 2017-04-22T05:25:06Z

@janvorli please see @sergign60 note above.

rartemev · 2017-06-19T23:58:11Z

 //
 void CodeGen::genProfilingLeaveCallback(unsigned helper /*= CORINFO_HELP_PROF_FCN_LEAVE*/)
 {
+


I'd rather remove such new empty lines

@rartemev fixed

rartemev · 2017-06-20T00:03:53Z

+//        UINT64      flt1;
+//        UINT64      flt2;
+//        UINT64      flt3;
+// #if defined(UNIX_AMD64_ABI)


Is it source file used only for Unix? If so I think it would be good to remove this #ifdef bracket

rartemev · 2017-06-20T00:26:46Z

+  mov                     [rsp + 0x88], rdx     //                -- struct rdx field
+  mov                     [rsp + 0x90], rcx     //                -- struct rcx field
+  mov                     [rsp + 0x98], r8      //                -- struct r8 field
+  mov                     [rsp + 0xa0], r9      //                -- struct r9 field


Do we really need to save those scratch registers (rdi-r9)?

@rartemev these registers are not scratch registers, they are argument registers.

dnfclas added the cla-already-signed label Oct 19, 2016

sergign60 force-pushed the profelt branch from 4bd5d20 to eddd486 Compare October 20, 2016 08:53

sergign60 force-pushed the profelt branch from ac7c6dd to 52e146e Compare November 14, 2016 10:19

sivarv reviewed Nov 15, 2016

View reviewed changes

CarolEidt reviewed Nov 15, 2016

View reviewed changes

sivarv reviewed Nov 15, 2016

View reviewed changes

CarolEidt reviewed Nov 15, 2016

View reviewed changes

noahfalk reviewed Nov 16, 2016

View reviewed changes

janvorli reviewed Nov 16, 2016

View reviewed changes

sergign60 force-pushed the profelt branch 2 times, most recently from 95b6b51 to b4c7322 Compare April 2, 2017 14:47

sergign60 force-pushed the profelt branch 3 times, most recently from ad9fa9a to 65eb9e6 Compare April 4, 2017 14:42

sergign60 force-pushed the profelt branch 4 times, most recently from 24e48d9 to e180fe6 Compare April 14, 2017 09:18

sergign60 force-pushed the profelt branch from e180fe6 to 4b53beb Compare April 19, 2017 11:39

sergign60 force-pushed the profelt branch from 4b53beb to 279af1a Compare May 10, 2017 11:51

BruceForstall self-requested a review June 2, 2017 22:21

rartemev reviewed Jun 19, 2017

View reviewed changes

rartemev reviewed Jun 20, 2017

View reviewed changes

sergign60 force-pushed the profelt branch 3 times, most recently from 4d3a75a to 9200f7a Compare June 27, 2017 07:55

sergign60 closed this Jul 3, 2017

sergign60 force-pushed the profelt branch from 9200f7a to 01088c7 Compare July 3, 2017 10:19

sergign60 mentioned this pull request Jul 3, 2017

implementing profiler ELT callbacks for AMD64 Linux #12603

Merged

kvochko mentioned this pull request Jan 31, 2020

[Win 7, Linux/x64] Tests fail with COMPlus_JitELTHookEnabled dotnet/runtime#7228

Closed


		unsigned regCount = retTypeDesc.GetReturnRegCount();

		if (varTypeIsGC(compiler->info.compRetType))



		// setup ProfilePlatformSpecificData structure
		xor r11, r11 // nullify r11

Conversation

sergign60 commented Oct 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergign60 commented Oct 19, 2016

Uh oh!

BruceForstall commented Oct 19, 2016

Uh oh!

BruceForstall commented Oct 19, 2016

Uh oh!

sergign60 commented Oct 20, 2016

Uh oh!

BruceForstall commented Oct 20, 2016

Uh oh!

sergign60 commented Nov 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergign60 commented Nov 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sivarv commented Nov 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergign60 commented Nov 10, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergign60 Nov 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergign60 Nov 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sivarv Nov 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sivarv commented Nov 15, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

sergign60 commented Oct 19, 2016 •

edited

Loading

sergign60 commented Nov 9, 2016 •

edited

Loading

sergign60 commented Nov 9, 2016 •

edited

Loading

sivarv commented Nov 9, 2016 •

edited

Loading

sergign60 Nov 15, 2016 •

edited

Loading

sergign60 Nov 15, 2016 •

edited

Loading

sivarv Nov 15, 2016 •

edited

Loading

sergign60 Nov 16, 2016 •

edited

Loading

sergign60 Nov 16, 2016 •

edited

Loading

sergign60 Nov 16, 2016 •

edited

Loading

sergign60 Nov 16, 2016 •

edited

Loading

sergign60 commented Apr 2, 2017 •

edited

Loading