Evolve the TCK

[marcellanz]

After a short Gitter discussion with @pvlugter and @sleipnir, I open this issue for discussion (attached is the gitter exchanges for context).

In short, the issue is to evolve the TCK so that a language support library implementation can be verified.

I think the topic is too broad to capture it in one issue and I propose to discuss it here and then create actionable issues out of it. I list here suggested topics and existing related issues.

Suggested Objectives

1. Evolve the TCK for all state models defined by the Cloudstate Protocol.
2. Enhance or adapt the TCK Shopping Cart Example to include other state models as explanatory as possible. This might not be exhaustive in terms of TCK completeness, but still have a use case that is easy to understand, and also can be an example how to use the state models. I'm not sure if this is that important for the TCK.
3. Define synthetic entities and interactions so that the Cloudstate Protocol can be verified by the TCK. These entities have to be provided then by each support library so that the TCK can run against them.
4. Let support libraries execute the TCK within their repositories, preferable without the need of a JVM to be installed or a SBT dependency => a Docker based approach might be feasible.
5. Maybe, existing tests of the proxy can be used to guide the synthetic tests defined.
6. Perhaps for the future and a low priority for now: the TCK currently runs as "Akka + <name_of_language>'"; how would the TCK test a reference implementation of the proxy.

Related Issues

• Make the TCK easily executed by different language supports #216 – Make the TCK easily executed by different language supports
• Language support module release cycles #289 – Language support module release cycles

---

Related discussion (Gitter, May 11th)

@marcellanz : I think it may be time to extend the TCK for other state models like CRDTs, Stateless, KV, Eventing, <name_any> for Cloudstate. I come to a point where I'd like to verify an implementation and can't. The TCK so far was a great enabler getting started with EventSourcing.

There is tck/src/ with the Shopping Cart example. How should we model the TCK around other state models? While the proxy itself has many synthetic test cases (AFAIKS) the TCK shopping cart is a nice and illustrative example, should we find such an example for other types too, or perhaps expand on the shopping cart?
With a few language support libraries in different states existing, and new coming, we also might better have the TCK before they are complete.

@sleipnir: I think we should do a lot of things in several steps (parallel or not) at TCK. I have illustrated this need several times and I agree with you. So my thinking is divided like this?

1. The TCK must be less than a selected docker image and accept the necessary parameters to run the tests easily, without depending on the entire Cloudstate project.
2. We must expand the TCK and the shopping cart example, there is a reference somewhere in the list that can be used (HotItems) for example to test "forwards and effects" that are not currently supported in the TCK.
3. We must include (with shoppingcart or another) a test for statelessness (a simple request/response would be enough in this case).
4. A basic example for CRDT should be included at first and then expanded to more complex examples. I also think it would be a good way to test the various types of CRDT independently, that way we could have a compatibility matrix supporting in supported idioms more granular.

@pvlugter: Yes, please open an issue, if it doesn't exist already, with everything we need for the TCK. And we have #216 for supporting the TCK in the language supports.
I'm personally not sure whether having the TCK based on examples, like shopping cart, or just artificial tests is better. Possibly a combination in the end? Simple synthetic test cases to get full coverage, plus examples that demonstrate the main use cases.

[marcellanz]

Discussion Notes:

CRDT
Similar to the ShoppingCartClient that the TCK is using, there is a io.cloudstate.samples.CrdtsClient. Also similar to the shopping cart example, there is a crdt-example.proto that defines the corresponding io.cloudstate.samples.CrdtExample service.

The CrdtExample Service covers

• GCounter
• PNCounter
• GSet
• ORSet

with missing

• Flag
• LWWRegister
• ORMap
• Vote

How to implement
Implement the io.cloudstate.samples.CrdtExample and execute its service interface similar to the shopping cart TCK. This service can be the synthetic use case for the CRDT TCK. The synthetic methods used and possibly expanded can be similar or even the same as the language supports unit tests. There is a difference to eventsourcing as CRDTs can have streamed commands. The io.cloudstate.samples.CrdtExample service has already two methods from samples-java-chat where streamed commands are used.

Stateless
TBD

KV
TBD

[marcellanz]

@sleipnir this can be perhaps done similarly for the stateless model? wdyt?

[marcellanz]

I bootstrapped myself a bit and started with the extension of the existing TCK implementing synthetic tests for the CRDT support.

I currently use the existing io.cloudstate.samples.CrdtExample service from the crdt-example.proto and wired its service client into io.cloudstate.tck.CloudStateTCK so that it gets setup and called similar to existing event sourcing tests. I'm not yet sure how to test replication of CRDTs this way.

WIP branch Cloudstate:
https://github.com/marcellanz/cloudstate/tree/feature/tck_crdt_support

[marcellanz]

Testing CRDT state replication within the TCK was surprisingly easy to do, at least to trigger an init message with a state applied by the proxy:

• Increment GCounter by X
• Get GCounter, but crash the entity intentionally on reception on the user function side.
• Get GCounter, 2nd attempt => proxy delivers an init message + initial state
• Return the GCounter and verify count == X

I'm sure there is perhaps a better way to force the proxy to replicate a CRDTs state, but this way, a full circle of an crashed and re-initialized user functions state is involved, which might be a use-case of its own.

sequence: tck_crdt_support_001.json – Verify GCounter

[sleipnir]

Beautiful work Em segunda-feira, 18 de maio de 2020, Marcel Lanz <notifications@github.com> escreveu:

…

Testing CRDT state replication within the TCK was surprisingly easy to do, at least to trigger an init message with a state applied by the proxy: - Increment GCounter by X - Get GCounter, but crash the entity intentionally <https://gist.github.com/marcellanz/f9dfc651941e08666260ab21517f4137#file-tck_crdt_support_001-json-L145> on reception on the user function side. - Get GCounter, 2nd attempt => proxy delivers an init message + initial state <https://gist.github.com/marcellanz/f9dfc651941e08666260ab21517f4137#file-tck_crdt_support_001-json-L229> - Return the GCounter and verify count == X I'm sure there is perhaps a better way to force the proxy to replicate a CRDTs state, but this way, a full circle of an crashed and re-initialized user functions state is involved, which might be a use-case of its own. sequence: tck_crdt_support_001.json – Verify GCounter <https://gist.github.com/marcellanz/f9dfc651941e08666260ab21517f4137> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#316 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACTTZXBJP4L3SNXPTM7RDLRSGSJZANCNFSM4M7DLGGQ> .

-- Adriano P. Santos "O homem erudito é um descobridor de fatos que já existem - mas o homem sábio é um criador de valores que não existem e que ele faz existir." Albert Einstein

[pvlugter]

👍 great work

[marcellanz]

I'm coming to the point where crdt-example.proto limits its use to TCK-verify a CRDT implementation. Also because crdt-examples.proto lacks a few CRDT types and I'd like to verify streamed commands for the CRDT types.

I came to the conclusion that the CRDT TCK for sure has a completely synthetic part to capture all CRDT types. A useful synthetic service would not serve any other purpose than operate on CRDT types' API for verification. Anything with a practical use-case like the shopping cart would be supplementary.

There are uses for crdt-examples.proto right now for:

• https://github.com/cloudstateio/cloudstate/tree/v0.5.1/samples/js-crdts
• https://github.com/cloudstateio/cloudstate/blob/v0.5.1/proxy/proxy-tests/src/test/scala/io/cloudstate/proxy/stress/CrdtStressSpec.scala
• https://github.com/cloudstateio/cloudstate/blob/v0.5.1/samples/akka-client/src/main/scala/io/cloudstate/samples/CrdtsClient.scala

Two a bit older and CrdtStressSpec.scala heavily used.

Options

• Evolve crdt-examples.proto and have nothing existing changed and not breaking anything of its current use.
• Add a new Service to be used for synthetic CRDT TCK.

I think I'd probably go for the latter option and having the crdt-examples.proto use consolidated in the future or even leave it as it is.

@pvlugter @viktorklang @jroper wdyt?

[viktorklang]

I vote for creating a new service as well. The original thought of the shopping-cart being both a tck validator and an example was a good idea initially but as features grow I feel the need for dedicated tck protos increases. -- Cheers, √

[pvlugter]

Agree on adding a new service for synthetic tests.

[marcellanz]

Thanks for voting :)

I have a few points for the service design and the TCKs behaviour for the synthetic tests. Let me know wdyt, as the last point is a bit of work:

New service
is under protocols/tck/tck_crdt.proto. I think protocols/example is for examples, even if the evensourcing TCK shoppingcart service is currently under example.

1:1 mapping of TCK-Service <> CRDT API
There is no implicit protocol or story of things like mutating domain data and then error on that. Each CRDT type will have gRPC service method that is in line with their API. There will be most of the time a 1:1 mapping of what the TCK calls on the support library. This also simplifies the implementation on the user support library.

Errors
There is room "to error", but the support library has to do it explicitly by doing so in part as we can't enforce invalid data to be sent most of the time. An example is to decrement a GCounter, we can't enforce that as, GCounter state and delta are by design uint64 and, depending on the language, the implementation would have to type-cast values to an incompatible type.

If we really like to decrement a GCounter, we would have a dedicated TCK rpc method where we instruct the support library to decrement the GCounter. And as said above, this would lead to an unusual typecast to call GCounter.Increment(-1) if that is even possible with the libraries types API.
(JS and Python are exceptions of that, the two most common languages currently used in the field^3.4.4, so its perhaps still a good idea to do so.)

Force errors / Replicate state
To test error handling, we could send a flag that the user support should error. This way we could test the operation would error and with it, that the user support would propagate this error properly. Also, having an entity being crashed right now, is the preferred way to let the proxy send an init message with an initial state and lets us test applying state to a CRDT.
(I currently don't know how to have the proxy sending delta-messages, @jroper do you have an idea how to do that?)

Test Coverage
The TCK will cover the complete set of unit tests like we have already for user support in JS and Go. It makes little sense only to cover a few smoke-like tests I think.

This applies especially for synthetic tests and might be different for a TCK use-case like the shoppingcart that we have right now for eventsourcing. There, the user support library proves to implement that usecase and can be used for demonstration purposes. This can be useful as one can replace the JS based shoppingcart implementation with any of the other ones implemented for the TCK, and the shopping cart demo would work.

[sleipnir]

Hello everyone!
I think having a TCK with specific services to cover tests with all types is a great idea. But I think that support libraries should make some individual effort in implementing language-specific coverage tests. In the spring support, for example, the shopping cart is tested through unit tests, so that I know that the change to pass the TCK is highly likely before I even put it in the TCK test.

[marcellanz]

I agree; every language support implementation should have their own unit tests covering, the TCK does not replace that. I'm sure, many language specific tests can't be done by the TCK and it was not my intent to propose that.

[sleipnir]

Yes @marcellanz I understood that it was not your intention. Is that I believe that a mix of the two approaches (TCK, Coverage tests) should be desired. I say this because in your GCounter negative example I see this being tested in the support libraries because it is a very specific case and it is very related to the unfortunate path of a request. What I mean is that in addition to a good TCK we have to have a good specification on the expected behaviors, this must be defined clearly and publicly, so that the implementers have the ability to validate the idiosyncrasies of each language against the expected behaviors that were defined by the protocol.