Proposal: in-query retract and assert

[rnewman]

A common pattern I've encountered is:

let link_title = ip.get_entid(&kw!(:link/title)).unwrap();

let results = ip.q_once(r#"
    [:find ?e ?t2
     :where [?e :link/title ?t1]
            [?e :link/title ?t2]
            [(unpermute ?t1 ?t2)]]
"#, None).into_rel_result()?;

if !results.is_empty() {
    let mut builder = TermBuilder::new();
    for row in results.into_iter() {
        let mut r = row.into_iter();
        let e = r.next().and_then(|e| e.into_known_entid()).expect("entity");
        let obsolete = r.next().expect("value");
        builder.retract(e, link_title, obsolete)?;
    }
    ip.transact_builder(builder)?;
}

That is: query to build up retractions and/or assertions, then transact those.

Datomic follows this model easily: it's loosely typed, data is local and in memory, etc. etc.

It's slightly less comfortable in Rust because of the rigid type system: we can't easily make builder ergonomic, and the caller needs to do some ritual (into_rel_result, consuming the results, etc.) to make this happen.

It's likely to be even more uncomfortable, and perhaps a little slow, from other languages.

We might instead choose to borrow from SPARQL UPDATE and extend our syntax to do this inline:

PREFIX foaf:  <http://xmlns.com/foaf/0.1/>

WITH <http://example/addresses>
DELETE { ?person foaf:givenName 'Bill' }
INSERT { ?person foaf:givenName 'William' }
WHERE
  { ?person foaf:givenName 'Bill'
  } 

which for Mentat might look exactly like a :find query with different operands:

[:retract (?person :person/name "Bill")
 :assert (?person :person/name "William")
 :where [?person :person/name "Bill"]]

This could be evaluated in one of two ways:

• As a special kind of projection that does the ritual consume/build/transact work for you. This is expensive — values will be pulled into memory, collected, then serialized for SQL and transacted — but it's relatively little work.
• By translation into SQL. In part this will encode some of the work the transactor already does. More development work, but more production-grade.

[ncalexan]

I have nothing against this kind of specialized syntax in principle, but I feel that any such specialization is premature right now. We don't have any production customers; until we do, let's not try to anticipate too much.

[ncalexan]

Having written some non-trivial query-transact stanzas myself now, I can see what you're driving at. I have a few thoughts:

1. Prefer a generic :transact to :assert and :retract. Using new keywords fits the short-hand for find specs, but it doesn't scale well to :db/retract{Attribute,Entity}. We might want to allow boolean true/false for :db/add/:db/retract, because otherwise it's difficult to make the operation depend on a query result. (This might be worth doing for regular entities too, since creating booleans is easier than keywords.)

2. It's not clear to me how the find spec bindings that specify scalar, tuple, collection, or relation fit into this world. If we allow collections or relations, it's not clear to me that the resulting :transact will be a non-overlapping set of transaction entities. Right now, transact pushes the burden of not overlapping entities onto the consumer. That might not be the right thing here.

3. I'm very skeptical about translating :transact stanzas directly into SQL. There's a reason we didn't pursue that approach for the initial transactor implementation -- Datomic's tempid and upsert mechanism does not map easily onto SQL! I'm doubly concerned that we'd grow two transactor implementations with subtly different bugs. I think a robust implementation of query that performs the ritual sacrifices before handing over to a robust implementation of transact is a good idea; I'm very skeptical that trying to handle non-trivial transactions directly as SQL is a good idea.

[ncalexan]

My particular use case follows the Sync 1.5 logins schema that @rnewman sketched at https://gist.github.com/rnewman/4a1442f88d471956b62b9c88a6adfa58. To update a login identified by UUID, I need to update two related entities, but there's no unique identity relationship to leverage. That means manually writing the equivalent of

[:transact
 [:db/add ?c :credential/password ?password]
 [:db/add ?f :form/usernameField ?username-field]
 :in
 ?uuid ?password ?username-field
 :where
 [?login :sync.login/uuid ?uuid]
 [?login :sync.login/credential ?c]
 [?login :sync.login/form ?f]]

[rnewman]

Prefer a generic :transact to :assert and :retract

Mmm, perhaps. My thinking was that it's clearer that retractions happen before assertions.

I'm very skeptical about translating :transact stanzas directly into SQL.…

I think that would have to happen in two ways:

• By populating a temporary table with the output of the query, and then feeding that through the transactor.
• By encoding some transactor knowledge in a kind of template. That is: we know from the :transact clause whether some tempids will need to be resolved; if not (and I suspect that 'not' is the most common), then no upserting logic needs to occur, which might allow us to do all the work in SQL for the common case.

[ncalexan]

A couple more thoughts here:

Prefer a generic :transact to :assert and :retract. Using new keywords fits the short-hand for find specs, but it doesn't scale well to :db/retract{Attribute,Entity}. We might want to allow boolean true/false for :db/add/:db/retract, because otherwise it's difficult to make the operation depend on a query result. (This might be worth doing for regular entities too, since creating booleans is easier than keywords.)

I pumped the breaks on #650, and this can actually replace :db/retractEntity, if we're willing to give up recursion; retracting an entity is essentially

[:transact
 [:db/retract ?e ?a ?v]
 :in
 ?e
 :where
 [?e ?a ?v]]

We might want to say "narrow" the relation using a findspec, say

[:transact
 [:db/retract ?e ?a ?v]
 :using
 ?e ?a ?v
 :in
 ?e
 :where
 [?e ?a ?v]]

It's not clear to me how the find spec bindings that specify scalar, tuple, collection, or relation fit into this world. If we allow collections or relations, it's not clear to me that the resulting :transact will be a non-overlapping set of transaction entities. Right now, transact pushes the burden of not overlapping entities onto the consumer. That might not be the right thing here.

#685 changed a lot of the semantics around overlapping entities, so we're better positioned to handle overlapping and doing the right thing in the face of repeated [e a] pairs now. That might help implementation here.