Slow parsing

[galmok]

I am (ab)using the commandlineparse (2.1.1-beta) in that I want to offer to supply the arguments from a file instead of the command line. This makes it easier for me to handle my test programs. But it also allows for many more arguments as there isn't a command line limit anymore.

I have supplied approx 9000 values for a string[] option and this causes CommandLineParser to use a very long time (still waiting on it to complete). This is the thread stack (from Process Hacker):

0, ntoskrnl.exe!PoStartNextPowerIrp+0x1833
1, ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d
2, ntoskrnl.exe!KeWaitForSingleObject+0x19f
3, ntoskrnl.exe!PoStartNextPowerIrp+0xbd0
4, ntoskrnl.exe!PoStartNextPowerIrp+0x186d
5, ntoskrnl.exe!PoStartNextPowerIrp+0x1ae7
6, System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeModule, Int32, Int32, System.RuntimeType, Boolean, System.Collections.IList, Boolean) + 0x2c <-- mscorlib.ni.dll+0x54e10c
7, System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimePropertyInfo, System.RuntimeType) + 0x72 <-- mscorlib.ni.dll+0x4b7102
8, CommandLine.Core.ReflectionExtensions+<>c__0`1[[System.__Canon, mscorlib]].<GetSpecifications>b__0_1(System.Reflection.PropertyInfo) + 0x2e <-- 0x7fe933a40fe
9, System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0xdf <-- System.Core.ni.dll+0x2d5d7f
10, System.Linq.Enumerable+WhereEnumerableIterator`1[[System.__Canon, mscorlib]].MoveNext() + 0x33 <-- System.Core.ni.dll+0x2d7193
11, System.Linq.Lookup`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Create[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>) + 0x12d <-- System.Core.ni.dll+0x2d96cd
12, System.Linq.GroupedEnumerable`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].GetEnumerator() + 0x57 <-- System.Core.ni.dll+0x2d9547
13, System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0x53 <-- System.Core.ni.dll+0x2d5cf3
14, System.Linq.Enumerable+<OfTypeIterator>d__92`1[[System.__Canon, mscorlib]].MoveNext() + 0xfb <-- System.Core.ni.dll+0x2f71fb
15, System.Linq.Enumerable.SingleOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>) + 0x7f <-- System.Core.ni.dll+0x2d824f
16, CommandLine.Core.NameLookup.HavingSeparator(System.String, System.Collections.Generic.IEnumerable`1<CommandLine.Core.OptionSpecification>, System.StringComparer) + 0x7c <-- 0x7fe933a91dc
17, CommandLine.Core.Tokenizer+<>c__DisplayClass3_0.<ExplodeOptionList>b__0(CommandLine.Core.Token, Int32) + 0x47 <-- 0x7fe933a9027
18, System.Linq.Enumerable+<SelectIterator>d__5`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0x180 <-- System.Core.ni.dll+0x2dfb90
19, System.Linq.Enumerable+<SkipWhileIterator>d__32`1[[System.__Canon, mscorlib]].MoveNext() + 0x162 <-- System.Core.ni.dll+0x5e9c72
20, System.Linq.Enumerable.FirstOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>) + 0x73 <-- System.Core.ni.dll+0x2f6013
21, CommandLine.Core.Tokenizer+<>c__DisplayClass3_0.<ExplodeOptionList>b__3(CommandLine.Core.Token, Int32) + 0x79 <-- 0x7fe933a8ef9
22, System.Linq.Enumerable+<SelectIterator>d__5`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0x180 <-- System.Core.ni.dll+0x2dfb90
23, System.Linq.Enumerable+<SelectManyIterator>d__16`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0xf0 <-- System.Core.ni.dll+0x2d4300
24, System.Linq.Buffer`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>) + 0x107 <-- System.Core.ni.dll+0x2d3857
25, System.Linq.Enumerable.ToArray[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>) + 0x6a <-- System.Core.ni.dll+0x2d754a
26, CSharpx.EnumerableExtensions.Memorize[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>) + 0x57 <-- 0x7fe933a7587
27, CommandLine.Core.TokenPartitioner.Partition(System.Collections.Generic.IEnumerable`1<CommandLine.Core.Token>, System.Func`2<System.String,CSharpx.Maybe`1<CommandLine.Core.TypeDescriptor>>) + 0x32 <-- 0x7fe933a86f2
28, CommandLine.Core.InstanceBuilder+<>c__DisplayClass0_0`1[[System.__Canon, mscorlib]].<Build>b__7() + 0xd3 <-- 0x7fe933a6363
29, CommandLine.Parser.ParseArguments[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.String>) + 0x1b8 <-- 0x7fe933a1578

I would guess it is the reflection making it all run slow.

I'll find another way to pass this many arguments.

[ericnewton76]

I would suggest an easier way to pass a crap-ton of arguments would probably be to JSON serialize (using Newtonsoft.Json library) your options types.

…

On Thu, Nov 9, 2017 at 6:50 AM, galmok ***@***.***> wrote: I am (ab)using the commandlineparse (2.1.1-beta) in that I want to offer to supply the arguments from a file instead of the command line. This makes it easier for me to handle my test programs. But it also allows for many more arguments as there isn't a command line limit anymore. I have supplied approx 9000 values for a string[] option and this causes CommandLineParser to use a very long time (still waiting on it to complete). This is the thread stack (from Process Hacker): 0, ntoskrnl.exe!PoStartNextPowerIrp+0x1833 1, ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d 2, ntoskrnl.exe!KeWaitForSingleObject+0x19f 3, ntoskrnl.exe!PoStartNextPowerIrp+0xbd0 4, ntoskrnl.exe!PoStartNextPowerIrp+0x186d 5, ntoskrnl.exe!PoStartNextPowerIrp+0x1ae7 6, System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimeModule, Int32, Int32, System.RuntimeType, Boolean, System.Collections.IList, Boolean) + 0x2c <-- mscorlib.ni.dll+0x54e10c 7, System.Reflection.CustomAttribute.GetCustomAttributes(System.Reflection.RuntimePropertyInfo, System.RuntimeType) + 0x72 <-- mscorlib.ni.dll+0x4b7102 8, CommandLine.Core.ReflectionExtensions+<>c__0`1[[System.__Canon, mscorlib]].<GetSpecifications>b__0_1(System.Reflection.PropertyInfo) + 0x2e <-- 0x7fe933a40fe 9, System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0xdf <-- System.Core.ni.dll+0x2d5d7f 10, System.Linq.Enumerable+WhereEnumerableIterator`1[[System.__Canon, mscorlib]].MoveNext() + 0x33 <-- System.Core.ni.dll+0x2d7193 11, System.Linq.Lookup`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Create[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>) + 0x12d <-- System.Core.ni.dll+0x2d96cd 12, System.Linq.GroupedEnumerable`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].GetEnumerator() + 0x57 <-- System.Core.ni.dll+0x2d9547 13, System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0x53 <-- System.Core.ni.dll+0x2d5cf3 14, System.Linq.Enumerable+<OfTypeIterator>d__92`1[[System.__Canon, mscorlib]].MoveNext() + 0xfb <-- System.Core.ni.dll+0x2f71fb 15, System.Linq.Enumerable.SingleOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>) + 0x7f <-- System.Core.ni.dll+0x2d824f 16, CommandLine.Core.NameLookup.HavingSeparator(System.String, System.Collections.Generic.IEnumerable`1<CommandLine.Core.OptionSpecification>, System.StringComparer) + 0x7c <-- 0x7fe933a91dc 17, CommandLine.Core.Tokenizer+<>c__DisplayClass3_0.<ExplodeOptionList>b__0(CommandLine.Core.Token, Int32) + 0x47 <-- 0x7fe933a9027 18, System.Linq.Enumerable+<SelectIterator>d__5`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0x180 <-- System.Core.ni.dll+0x2dfb90 19, System.Linq.Enumerable+<SkipWhileIterator>d__32`1[[System.__Canon, mscorlib]].MoveNext() + 0x162 <-- System.Core.ni.dll+0x5e9c72 20, System.Linq.Enumerable.FirstOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>) + 0x73 <-- System.Core.ni.dll+0x2f6013 21, CommandLine.Core.Tokenizer+<>c__DisplayClass3_0.<ExplodeOptionList>b__3(CommandLine.Core.Token, Int32) + 0x79 <-- 0x7fe933a8ef9 22, System.Linq.Enumerable+<SelectIterator>d__5`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0x180 <-- System.Core.ni.dll+0x2dfb90 23, System.Linq.Enumerable+<SelectManyIterator>d__16`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() + 0xf0 <-- System.Core.ni.dll+0x2d4300 24, System.Linq.Buffer`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>) + 0x107 <-- System.Core.ni.dll+0x2d3857 25, System.Linq.Enumerable.ToArray[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>) + 0x6a <-- System.Core.ni.dll+0x2d754a 26, CSharpx.EnumerableExtensions.Memorize[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>) + 0x57 <-- 0x7fe933a7587 27, CommandLine.Core.TokenPartitioner.Partition(System.Collections.Generic.IEnumerable`1<CommandLine.Core.Token>, System.Func`2<System.String,CSharpx.Maybe`1<CommandLine.Core.TypeDescriptor>>) + 0x32 <-- 0x7fe933a86f2 28, CommandLine.Core.InstanceBuilder+<>c__DisplayClass0_0`1[[System.__Canon, mscorlib]].<Build>b__7() + 0xd3 <-- 0x7fe933a6363 29, CommandLine.Parser.ParseArguments[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.String>) + 0x1b8 <-- 0x7fe933a1578 I would guess it is the reflection making it all run slow. I'll find another way to pass this many arguments. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#186>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACG_zNznJPKVQkVz6AMTE81CJzxGl4bgks5s0ubygaJpZM4QX2Qx> .

-- E.Newton Senior Software Craftsman Cell: 407-497-1429 EMail: ericnewton76@gmail.com

[galmok]

Yeah, I know... I just hoped I could use this shortcut, but that was not a viable path.

[ericnewton76]

Were you going to use the Json option I mentioned above? We (I) might peruse the Json source to see how they do it so quickly... 😄

Reality is that its an edge case for having to parse any more than 10 items.

[nemec]

I agree that this is somewhat of an edge case, but there is one situation that I think would benefit from having better performance: a chat bot. Imagine, for example, a Slack bot that let you call it and customize its behavior with various options. Depending on how widely it's used, you could run into performance and thread-safety issues.

When Giacomo rewrote version 2 he modeled the internals after functional programming techniques (F#) which lets you do some things easily, but it's also takes a massive performance hit. Not bad enough when you run it once or twice, but in a tight loop it definitely has issues. I think if we put this code in a profiler we could find a lot of hot spots that are easy to optimize. Reflection on custom attributes may be one of those, considering attributes don't (usually) change at runtime.

[galmok]

I did not use the json option, but implemented some coffee that could approximate the result of the large bunch of argument. I do like the command line parser and I hope it eventually is able to parse options somewhat faster. ;-)