spec run appears stuck at “Initializing new graph for mastadon_openapi”

[rahilmehta-dev]

Hi, I’m not sure if this is a bug or just expected behavior, but when I run AutoRestTest on the Mastodon spec it seems to get stuck for more than 3 hours at this stage:

BEGINNING AUTO-REST-TEST...
CREATING SEMANTIC OPERATION DEPENDECY GRAPH...
Initializing new graph for mastadon_openapi.

It doesn’t produce any further output or errors, even after a long time.

Spec: mastadon_openapi.json

I see simillar output for seaweedfs.yaml

[tylerstennett]

Hey Rahil, thanks for the issue report! Taking a look now, but off the top of my head, I remember that mastadon is pretty huge which could be the reason, since the graph construction runtime is pretty costly. I'll take a look at some ways to improve speed for you though.

[tylerstennett]

It's strange that seaweedfs is taking so long, but for mastadon, it's certainly due to the specification size (hundreds of operations). However, I see a way to improve embedding performance which should make construction a lot faster. Let me add this.

[tylerstennett]

Okay, I've merged commit 524493f into main. It should hopefully speed up the graph construction process. On testing, seaweedfs took less than a second (though I don't think this was actually too impacted by the change — I believe it always ran fast for me).

Notably, when testing with mastadon_openapi.json, I found that the specification parser actually took a significant amount of time, and not the graph constructor. I've added some prints for logging when this starts and ends, but since Prance has no callback hooks (to my knowledge) that I can easily modify, I can't give progress updates on the specification parsing itself. I can't really speed up this process.

I did more digging into why mastadon_openapi.json took so long for the specification parser, and I found that it contained circular references (e.g., Status references itself) which is recursively parsed. Since this reference is used many times, it is parsed recursively thousands of times, which leads to significant slowdowns. Have you configured the recursion_limit parameter in your configuration file @rahilmehta-dev, to mitigate these circular references? I suspect the default value of 50 is far too big for this specification.

[tylerstennett]

I reduced the recursion_limit parameter in the specification parser configs for AutoRestTest, and mastadon_openapi.json ran pretty quickly for me (<5 minutes).

I understand the default values were misleading, so I modified them. Please use the main branch's recent updates, and let me know if you encounter any more difficulties!

[rahilmehta-dev]

I reduced the recursion_limit parameter in the specification parser configs for AutoRestTest, and mastadon_openapi.json ran pretty quickly for me (<5 minutes).

Is there any documentation that explains how testers should configure this parameter?

[tylerstennett]

Is there any documentation that explains how testers should configure this parameter?

Yes, I realize this is probably confusing, and I happened to just push a commit addressing this as you commented. The original README.md has an explanation, but I just fleshed it out. The recursion_limit parameter is for recursive references, which essentially limits the amount of times the same reference can be resolved again in the specification. This should typically be assigned by analyzing the tool semantics; for example, if an operation has a messaging system where a Comment -> Reply chain is permitted to a limit of 3 times, then a limit of 3 should be assigned. This is just a convenience feature for testers. However, for most cases (especially benchmarking), a limit of 1 is okay. I've set the default to 1, and modified the README.md and configurations.toml comments accordingly.