Implement a Redis backend storage

[chadell]

This PRs addresses issue #57 partially, abstracting the store for DiffSyncModel objects, and extending to using Redis backend besides the local memory.

This feature it's specially important when working on a distributed data pipeline when different workers would load the data from adapters and finally the data must be shared to create the diff. It leverages on #58 proposal.

The implementation approach has moved the previous _data into the LocalStore implementation, creating abstract methods to interact with the storage, so we were able to create another implementation using Redis.

TODO

• Extend Redis test coverage and refine cornercase
• Fix basic testing
• Update Docs (to be done when final approach is agreed)

[chadell]

@glennmatthews I found an issue with this PR, specifically on the handling of children. Today we run the add_child and remove_child directly in the DiffSyncModel, and this works well when we run it in memory, but when using another store, such as Redis, this changes in the memory object are not update to the backend store.

For instance, from example 01:

...
                self.add(device)
                site.add_child(device)
...

In this example, site is no longer updated in the store. We would need to to a self.update(site) after this change, but I'm concerned on the backwards compatibility implications this may have, because if we forgot the update the code only works well for local memory.

...
                self.add(device)
                site.add_child(device)
                self.update(site)
...

I have tested it and if we do the self.update() after the add_child works well for both cases. So, my proposal would be to at least update the examples, to use the pattern with the update.

Looking forward to your thoughts

[glennmatthews]

1. Is there a known minimum version of redis that's required?
2. Can we make this into an optional/extras dependency, i.e. so that pip install diffsync doesn't pull in redis (and only supports local storage as a result), but pip install diffsync[redis] does?

[chadell]

good idea

[glennmatthews]

Perhaps?

Suggested change

	if url and host and port:
	if url and (host or port):

[chadell]

if we do like this, then I have to remove the default port from the kwargs, that I thought it would be convenient. But, it could be also confusing... not sure

[glennmatthews]

OK, perhaps just if url and host then?

[glennmatthews]

I don't think I like having a private but abstract method. If this is something subclasses are responsible for implementing, it shouldn't be private, IMHO.

[chadell]

yes, I remember a previous conversation about this public/private question :)

[glennmatthews]

It occurs to me we should probably be taking advantage of structlog better on this error message. Perhaps something like:

Suggested change

	self._log.error(f"Unable to remove child {child_id} of {modelname} {uid} - not found!")
	self._log.error(f"Unable to remove child element as it was not found!", child_type=child_type, child_id=child_id, parent_type=modelname, parent_id=uid)

[glennmatthews]

Don't we need some logic here to handle the case where redis isn't installed?