Improve the algorithm used to identify the endpoints to be connected

[shravandoda]

The algorithm used right now for identifying the endpoints to be connected traverses the whole graph to find the ones to connect. This method is really slow. One idea is to use python dictionaries for storing the nodes to be connected while they are being created.

[Mec-iS]

we are working with the perspective of storing hundreds of millions of nodes, that is why we use Redis. Graph traversal has to be done using built-in Redis data structures and redis-graph querying system

[shravandoda]

@Mec-iS the issue is not with how edges are constructed between 2 nodes. It's with how the nodes to be connected are identified in the agent. Sorry for improper language. Currently the agent uses 4 nested loops(2 of which traverse the whole graph) to identify the nodes to be connected.

[Mec-iS]

in any case we cannot keep a dictionary in memory with nodes data. Redis is already an in-memory datastore, anything about the graph should be queried from the graph.

[shravandoda]

The way program works right now is that it uses a function to create nodes and then connect them. How about we use a dictionary as a local variable in the function? This way after all the concerned endpoints have been connected, the dictionary will be destroyed and node data will only be present in the graph. Using 4 nested loops is going to be very inefficient especially as the number of nodes increase.

[Mec-iS]

@sandeepsajan0 ^

[Mec-iS]

Using 4 nested loops is going to be very inefficient especially as the number of nodes increase.

yeah that's true. but any query or operation on the graph has to be done in Redis, we cannot allow to have the graph/nodes replicated by Python memory at any time. So I would try to refactor that code by leveraging Redis commands. The first version of the agent will be strictly coupled to Redis, working on it requires an understanding about what the datastore can do.