Add benchmarks for (de)serialization using ReferenceHandling.Preserve by jozkee · Pull Request #1130 · dotnet/performance

jozkee · 2020-01-16T04:07:52Z

jozkee · 2020-01-16T04:08:12Z

Deserialize using Preserve

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18363
Intel Core i7-6700 CPU 3.40GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-alpha.1.20071.3
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.7004, CoreFX 5.0.20.7004), X64 RyuJIT
  Job-KCALPP : .NET Core 5.0.0 (CoreCLR 5.0.20.7004, CoreFX 5.0.20.7004), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  IterationTime=250.0000 ms  MaxIterationCount=20  
MinIterationCount=15  WarmupCount=1

Type	Method	IsDataPreserved	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
ReadPreservedReferences<IndexViewModel>	DeserializePreserved	False	35,684.7 ns	656.71 ns	614.28 ns	35,496.6 ns	34,849.3 ns	37,185.0 ns	0.505	0.01	5.3993	-	-	22616 B
ReadPreservedReferences<Location>	DeserializePreserved	False	1,495.5 ns	29.87 ns	30.68 ns	1,491.6 ns	1,454.4 ns	1,569.9 ns	0.021	0.00	0.1254	-	-	528 B
ReadPreservedReferences<LoginViewModel>	DeserializePreserved	False	636.1 ns	12.20 ns	11.41 ns	641.3 ns	613.3 ns	649.2 ns	0.009	0.00	0.0562	-	-	248 B
ReadPreservedReferences<MyEventsListerViewModel>	DeserializePreserved	False	392,185.8 ns	3,625.03 ns	3,213.50 ns	391,890.1 ns	386,771.4 ns	398,360.4 ns	5.554	0.14	18.5471	4.6368	-	78824 B
ReadPreservedReferences<SimpleListOfInt>	DeserializePreserved	False	526.4 ns	8.61 ns	8.06 ns	529.3 ns	515.6 ns	537.5 ns	0.007	0.00	0.0354	-	-	152 B
ReadPreservedReferences<SimpleStructWithProperties>	DeserializePreserved	False	499.9 ns	4.17 ns	3.90 ns	499.0 ns	494.0 ns	506.8 ns	0.007	0.00	0.0333	-	-	144 B
ReadPreservedReferences<IndexViewModel>	NewtonsoftDeserializePreserved	False	70,627.7 ns	1,488.18 ns	1,461.60 ns	70,171.9 ns	69,098.9 ns	73,879.6 ns	1.000	0.00	7.5083	0.8343	-	32064 B
ReadPreservedReferences<Location>	NewtonsoftDeserializePreserved	False	2,783.0 ns	47.81 ns	42.38 ns	2,776.2 ns	2,724.4 ns	2,850.7 ns	0.039	0.00	0.7295	-	-	3056 B
ReadPreservedReferences<LoginViewModel>	NewtonsoftDeserializePreserved	False	1,203.3 ns	17.32 ns	15.36 ns	1,203.4 ns	1,183.2 ns	1,234.1 ns	0.017	0.00	0.6612	-	-	2776 B
ReadPreservedReferences<MyEventsListerViewModel>	NewtonsoftDeserializePreserved	False	762,897.1 ns	13,303.24 ns	11,792.97 ns	767,457.8 ns	740,424.1 ns	775,222.6 ns	10.803	0.29	36.1446	3.0120	-	160432 B
ReadPreservedReferences<SimpleListOfInt>	NewtonsoftDeserializePreserved	False	793.7 ns	13.20 ns	12.35 ns	790.3 ns	777.5 ns	811.0 ns	0.011	0.00	0.6492	-	-	2728 B
ReadPreservedReferences<SimpleStructWithProperties>	NewtonsoftDeserializePreserved	False	1,350.5 ns	26.75 ns	25.02 ns	1,344.4 ns	1,324.3 ns	1,394.4 ns	0.019	0.00	0.6684	-	-	2800 B

ReadPreservedReferences<IndexViewModel>	DeserializePreserved	True	11,509.0 ns	285.43 ns	253.02 ns	11,441.9 ns	11,200.8 ns	12,183.3 ns	0.79	0.02	1.3068	-	-	5488 B
ReadPreservedReferences<Location>	DeserializePreserved	True	1,615.1 ns	31.88 ns	29.83 ns	1,605.5 ns	1,579.9 ns	1,689.7 ns	0.11	0.00	0.1629	-	-	688 B
ReadPreservedReferences<LoginViewModel>	DeserializePreserved	True	796.1 ns	12.70 ns	10.61 ns	795.5 ns	783.0 ns	813.5 ns	0.05	0.00	0.0974	-	-	408 B
ReadPreservedReferences<MyEventsListerViewModel>	DeserializePreserved	True	41,854.7 ns	722.36 ns	675.69 ns	41,802.1 ns	41,148.1 ns	43,290.8 ns	2.87	0.06	4.1295	-	-	17360 B
ReadPreservedReferences<SimpleListOfInt>	DeserializePreserved	True	945.6 ns	13.57 ns	12.69 ns	943.4 ns	928.3 ns	974.8 ns	0.06	0.00	0.0779	-	-	336 B
ReadPreservedReferences<SimpleStructWithProperties>	DeserializePreserved	True	630.1 ns	11.65 ns	10.90 ns	626.3 ns	618.9 ns	653.7 ns	0.04	0.00	0.0702	-	-	304 B
ReadPreservedReferences<IndexViewModel>	NewtonsoftDeserializePreserved	True	14,592.4 ns	288.88 ns	321.09 ns	14,460.3 ns	14,148.0 ns	15,207.1 ns	1.00	0.00	1.7694	-	-	7496 B
ReadPreservedReferences<Location>	NewtonsoftDeserializePreserved	True	3,692.8 ns	373.32 ns	399.45 ns	3,586.3 ns	3,275.0 ns	4,746.7 ns	0.25	0.03	0.8563	-	-	3640 B
ReadPreservedReferences<LoginViewModel>	NewtonsoftDeserializePreserved	True	1,679.7 ns	37.63 ns	41.82 ns	1,676.1 ns	1,622.6 ns	1,759.4 ns	0.12	0.00	0.7971	-	-	3360 B
ReadPreservedReferences<MyEventsListerViewModel>	NewtonsoftDeserializePreserved	True	47,163.6 ns	923.00 ns	863.37 ns	47,011.4 ns	45,569.2 ns	49,059.5 ns	3.23	0.12	3.5481	-	-	15544 B
ReadPreservedReferences<SimpleListOfInt>	NewtonsoftDeserializePreserved	True	1,407.4 ns	19.63 ns	16.39 ns	1,403.6 ns	1,384.4 ns	1,439.0 ns	0.10	0.00	0.8345	-	-	3504 B
ReadPreservedReferences<SimpleStructWithProperties>	NewtonsoftDeserializePreserved	True	2,015.3 ns	187.94 ns	201.09 ns	1,974.5 ns	1,777.8 ns	2,582.6 ns	0.14	0.01	0.8028	-	-	3384 B

IsDataPreserved indicates if the payload contains preserve semantics when parsed. You can compare the results of using IsDataPreserved=false versus the ReadJson.cs benchmarks to see the penalty of just enabling preserve without actually preserving anything.

Deserialize NOT using preserve

Type	Method	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated
ReadJson<IndexViewModel>	DeserializeFromString	35,494.4 ns	668.36 ns	625.18 ns	35,414.8 ns	34,778.2 ns	37,084.6 ns	5.3612	-	-	22536 B
ReadJson<Location>	DeserializeFromString	1,501.9 ns	33.58 ns	35.93 ns	1,496.3 ns	1,456.2 ns	1,581.9 ns	0.1055	-	-	448 B
ReadJson<LoginViewModel>	DeserializeFromString	604.5 ns	13.79 ns	15.33 ns	600.4 ns	582.4 ns	645.3 ns	0.0383	-	-	168 B
ReadJson<MyEventsListerViewModel>	DeserializeFromString	396,155.0 ns	7,944.45 ns	8,500.47 ns	392,571.5 ns	389,705.8 ns	419,558.4 ns	18.2927	4.5732	-	78744 B
ReadJson<SimpleListOfInt>	DeserializeFromString	505.3 ns	6.09 ns	5.70 ns	504.0 ns	497.8 ns	514.4 ns	0.0160	-	-	72 B
ReadJson<SimpleStructWithProperties>	DeserializeFromString	449.0 ns	8.80 ns	9.78 ns	443.3 ns	438.9 ns	469.1 ns	0.0143	-	-	64 B

jozkee · 2020-01-16T04:09:15Z

Serialize using preserve

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18363
Intel Core i7-6700 CPU 3.40GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-alpha.1.20071.3
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.7004, CoreFX 5.0.20.7004), X64 RyuJIT
  Job-CRDAEN : .NET Core 5.0.0 (CoreCLR 5.0.20.7004, CoreFX 5.0.20.7004), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  IterationTime=250.0000 ms  MaxIterationCount=20  
MinIterationCount=15  WarmupCount=1

Type	Method	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
WritePreservedReferences<IndexViewModel>	SerializePreserved	7,057.3 ns	133.75 ns	131.36 ns	6,988.4 ns	6,917.1 ns	7,331.5 ns	0.63	0.02	0.8057	-	-	3448 B
WritePreservedReferences<Location>	SerializePreserved	1,383.6 ns	27.35 ns	31.49 ns	1,371.7 ns	1,346.5 ns	1,461.3 ns	0.12	0.00	0.1932	-	-	816 B
WritePreservedReferences<LoginViewModel>	SerializePreserved	665.8 ns	18.70 ns	21.53 ns	666.0 ns	640.0 ns	724.2 ns	0.06	0.00	0.1352	-	-	576 B
WritePreservedReferences<MyEventsListerViewModel>	SerializePreserved	28,524.1 ns	993.85 ns	1,063.41 ns	28,655.7 ns	27,112.9 ns	30,535.3 ns	2.55	0.09	1.9774	-	-	8616 B
WritePreservedReferences<SimpleListOfInt>	SerializePreserved	584.3 ns	10.16 ns	9.01 ns	585.0 ns	572.5 ns	603.1 ns	0.05	0.00	0.1166	-	-	496 B
WritePreservedReferences<SimpleStructWithProperties>	SerializePreserved	418.8 ns	8.13 ns	8.70 ns	416.8 ns	406.3 ns	440.3 ns	0.04	0.00	0.0803	-	-	336 B
WritePreservedReferences<IndexViewModel>	NewtonsoftSerializePreserved	11,232.0 ns	224.20 ns	220.19 ns	11,154.6 ns	10,940.4 ns	11,773.6 ns	1.00	0.00	2.4209	-	-	8976 B
WritePreservedReferences<Location>	NewtonsoftSerializePreserved	2,119.4 ns	45.03 ns	51.85 ns	2,099.8 ns	2,047.6 ns	2,222.8 ns	0.19	0.01	0.5372	-	-	2280 B
WritePreservedReferences<LoginViewModel>	NewtonsoftSerializePreserved	1,174.1 ns	19.44 ns	17.23 ns	1,170.2 ns	1,153.1 ns	1,200.8 ns	0.10	0.00	0.4868	-	-	2048 B
WritePreservedReferences<MyEventsListerViewModel>	NewtonsoftSerializePreserved	40,623.3 ns	2,225.47 ns	2,381.22 ns	39,639.5 ns	38,268.5 ns	46,017.9 ns	3.62	0.26	4.8513	0.1565	-	19512 B
WritePreservedReferences<SimpleListOfInt>	NewtonsoftSerializePreserved	1,159.8 ns	21.88 ns	20.47 ns	1,153.2 ns	1,138.7 ns	1,212.3 ns	0.10	0.00	0.5124	-	-	2168 B
WritePreservedReferences<SimpleStructWithProperties>	NewtonsoftSerializePreserved	1,058.1 ns	51.29 ns	59.07 ns	1,059.7 ns	995.1 ns	1,193.0 ns	0.09	0.01	0.4643	-	-	1960 B

Serialize NOT using preserve

Type	Method	Mean	Error	StdDev	Median	Min	Max	Gen 0	Gen 1	Gen 2	Allocated
WriteJson<IndexViewModel>	SerializeToString	26,632.6 ns	269.80 ns	225.30 ns	26,552.3 ns	26,284.1 ns	27,117.4 ns	6.0870	-	-	25568 B
WriteJson<Location>	SerializeToString	1,193.5 ns	16.05 ns	15.01 ns	1,192.1 ns	1,176.6 ns	1,221.6 ns	0.1352	-	-	584 B
WriteJson<LoginViewModel>	SerializeToString	519.8 ns	38.58 ns	42.88 ns	500.9 ns	483.8 ns	632.7 ns	0.0798	-	-	344 B
WriteJson<MyEventsListerViewModel>	SerializeToString	582,594.2 ns	11,021.54 ns	10,309.56 ns	582,388.3 ns	565,335.8 ns	598,398.0 ns	46.2963	46.2963	46.2963	276408 B
WriteJson<SimpleListOfInt>	SerializeToString	356.5 ns	9.17 ns	9.42 ns	357.8 ns	339.9 ns	376.4 ns	0.0555	-	-	240 B
WriteJson<SimpleStructWithProperties>	SerializeToString	345.0 ns	7.02 ns	8.09 ns	343.1 ns	333.0 ns	361.4 ns	0.0611	-	-	256 B

Here you can see that some types show less time when preserve is on, such as IndexViewModel and MyEventsListerViewModel, this is because they contain way too many duplicate objects so in those cases preserve improves performance; the cases where preserve decreases performance are Location, SimpleListOfInt, and SimpleStructWithProperties due those types does not contain any duplicate references.

adamsitnik

LGTM, thank you!

When should we expect the new API to get merged into dotnet/runtime? I am not sure I should merge the PR now or wait.

…e repo

adamsitnik

@jozkee the CI fails with following error:

[2020/01/21 22:27:02][INFO] corefx\System.Text.Json\Serializer\ReadPreservedReferences.cs(50,39): error CS0117: 'Categories' does not contain a definition for 'CoreFX' [C:\h\w\A9B50997\p\src\benchmarks\micro\MicroBenchmarks.csproj]
[2020/01/21 22:27:02][INFO] corefx\System.Text.Json\Serializer\WritePreservedReferences.cs(36,39): error CS0117: 'Categories' does not contain a definition for 'CoreFX' [C:\h\w\A9B50997\p\src\benchmarks\micro\MicroBenchmarks.csproj]
[2020/01/21 22:27:02][INFO]     0 Warning(s)
[2020/01/21 22:27:02][INFO]     2 Error(s)

Because I've changed the folder structure in the meantime in #1132 to use the new naming after repos got merged.

Basically everything that was called CoreFX is now called Libraries (this includes folder name and category name). Could you please fetch the latest changes and sync your branch?

adamsitnik · 2020-01-22T09:06:10Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadPreservedReferences.cs

+        }
+
+
+        [BenchmarkCategory(Categories.CoreFX, Categories.JSON)]


The name has recently changed to Libraries

Suggested change

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]

[BenchmarkCategory(Categories.Libraries, Categories.JSON)]

adamsitnik · 2020-01-22T09:06:30Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/WritePreservedReferences.cs

+            _settings = new JsonSerializerSettings { PreserveReferencesHandling = PreserveReferencesHandling.All };
+        }
+
+        [BenchmarkCategory(Categories.CoreFX, Categories.JSON)]


Suggested change

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]

[BenchmarkCategory(Categories.Libraries, Categories.JSON)]

adamsitnik · 2020-01-22T09:06:55Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/WritePreservedReferences.cs

+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETCOREAPP3_0 && !NETCOREAPP3_1


Suggested change

#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETCOREAPP3_0 && !NETCOREAPP3_1

#if NETCOREAPP5_0

adamsitnik · 2020-01-22T09:07:20Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadPreservedReferences.cs

+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETCOREAPP3_0 && !NETCOREAPP3_1


I think that as for now we can simplify it to #if NETCOREAPP5_0 and later when we have more 5.0-only files extend the project file with a new rule for 5.0

performance/src/benchmarks/micro/MicroBenchmarks.csproj

Line 73 in ca80d8e

<ItemGroup Condition=" '$(TargetFrameworkIdentifier)' == '.NETFramework' Or ('$(TargetFrameworkIdentifier)' == '.NETCoreApp' And '$(_TargetFrameworkVersionWithoutV)' < '3.0')">

Suggested change

#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETCOREAPP3_0 && !NETCOREAPP3_1

#if NETCOREAPP5_0

adamsitnik · 2020-01-22T09:16:08Z

src/benchmarks/micro/corefx/System.Text.Json/Serializer/ReadPreservedReferences.cs

+            }
+        }
+
+


nit: empty line

Suggested change

and exclude benchmarks for versions < 5.0

…efeerenceHandlingBenchmarks

jozkee · 2020-01-23T00:34:30Z

src/benchmarks/micro/MicroBenchmarks.csproj

    <Compile Remove="runtime\PacketTracer\**\*.cs" />
  </ItemGroup>

+  <ItemGroup Condition=" '$(TargetFrameworkIdentifier)' == '.NETFramework' Or ('$(TargetFrameworkIdentifier)' == '.NETCoreApp' And '$(_TargetFrameworkVersionWithoutV)' &lt; '5.0')">


@adamsitnik I added the exclude condition to avoid using the directive which would need extra changes in future netcoreapp releases.

jozkee · 2020-01-23T00:38:15Z

src/benchmarks/micro/MicroBenchmarks.csproj

+    <Compile Remove="libraries\System.Text.Json\Serializer\WritePreservedReferences.cs" />
+  </ItemGroup>
+
+  <!-- TODO: Remove package dependency for future -->


People that pull my changes will be broken and will need to install the latest SDK build; to avoid that, the 5.0 benchmarks will use the NuGet package version of System.Text.Json.

In general for tests targeting the current master branch we have the expectation that you need to always have an updated SDK matching it. It's fine to assume they'll have the right 5.0 bits.

I see, then I will proceed to remove this. Thanks.

In general for tests targeting the current master branch we have the expectation that you need to always have an updated SDK matching it.

Interesting. I didn't know that. Sounds good :)

Does the benchmarks_ci.py script install the latest SDK for the developer?
https://github.com/dotnet/performance/blob/ca80d8e2886b583d0a69635740b188248d3d6fdd/docs/benchmarkdotnet.md#using-python-script

ahsonkhan · 2020-01-23T01:37:12Z

src/benchmarks/micro/libraries/System.Text.Json/Serializer/ReadPreservedReferences.cs

+
+        [BenchmarkCategory(Categories.ThirdParty, Categories.JSON)]
+        [Benchmark(Baseline = true)]
+        public T NewtonsoftDeserializePreserved() => JsonConvert.DeserializeObject<T>(_serialized, _settings);


Comparing to Newtonsoft is useful for the first time analysis and comparison. However, I don't think we should check this in. We don't want fluctuations in that library between perf runs show up as regressions which are not actionable for us (and whoever is reviewing the regression would have to filter those out, since filing issues for it doesn't make sense).

If a benchmark belongs to ThirdParty category, we don't run it in our lab. So it can stay as is and be used for future manual comparisons.

ahsonkhan · 2020-01-23T01:38:52Z

src/benchmarks/micro/libraries/System.Text.Json/Serializer/WritePreservedReferences.cs

+        public string SerializePreserved() => JsonSerializer.Serialize(_value, _options);
+
+        [BenchmarkCategory(Categories.ThirdParty, Categories.JSON)]
+        [Benchmark(Baseline = true)]


This Baseline isn't doing what we would want it to do since this benchmark is generic and instead of comparing
SerializePreserved vs NewtonsoftSerializePreserved, we are comparing one model vs another, which isn't helpful.

Even though, I think we should remove the Newtonsoft test, it brings up an interesting question on how would we get the baseline comparison we want (even on local runs), if we wanted to? @adamsitnik - any suggestions?

Is very likely that what is causing such behavior is that I am running the benchmarks with --join, it is not refreshing the baseline when joining, should it?

Here's my command:
dotnet run -c Release -f netcoreapp5.0 --filter *WritePreservedReferences* --join

ahsonkhan

Otherwise, looks good.

Add benchmarks for (de)serialization using ReferenceHandling.Preserve

6884e2f

billwert approved these changes Jan 16, 2020

View reviewed changes

adamsitnik approved these changes Jan 16, 2020

View reviewed changes

jozkee added 2 commits January 21, 2020 13:49

Set ReferenceHandling properly now that the API is merged into runtim…

9c886e2

…e repo

Code should only run for netcoreapp5.0

33c3f56

adamsitnik requested changes Jan 22, 2020

View reviewed changes

jozkee added 3 commits January 22, 2020 16:11

Add PackageReference to System.Text.Json for 5.0

af391ae

and exclude benchmarks for versions < 5.0

Merge branch 'master' of https://github.com/Jozkee/performance into R…

2cea21b

…efeerenceHandlingBenchmarks

Fix references in csproj after merge

0a73517

jozkee commented Jan 23, 2020

View reviewed changes

jozkee added 2 commits January 22, 2020 16:56

Remove unnecessary nuget reference.

7e4199a

remove unnecessary changes

4feb6a6

jozkee force-pushed the RefeerenceHandlingBenchmarks branch from cf5306a to 4feb6a6 Compare January 23, 2020 01:12

ahsonkhan reviewed Jan 23, 2020

View reviewed changes

ahsonkhan approved these changes Jan 23, 2020

View reviewed changes

adamsitnik approved these changes Jan 23, 2020

View reviewed changes

adamsitnik merged commit e2b8f69 into dotnet:master Jan 23, 2020

jozkee deleted the RefeerenceHandlingBenchmarks branch January 24, 2020 01:08

	[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
	[BenchmarkCategory(Categories.Libraries, Categories.JSON)]

	#if !NETFRAMEWORK && !NETCOREAPP2_1 && !NETCOREAPP2_2 && !NETCOREAPP3_0 && !NETCOREAPP3_1
	#if NETCOREAPP5_0

Conversation

jozkee commented Jan 16, 2020

Uh oh!

jozkee commented Jan 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jozkee commented Jan 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahsonkhan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jozkee commented Jan 16, 2020 •

edited

Loading

jozkee commented Jan 16, 2020 •

edited

Loading