Cannot create complex circular relationships between tables and datatypes

The title here is a bit misleading, but the basic idea is that complex mutually-dependent tables and datatypes are not currently possible.

An example:

Foo
    name Text
Bar
    name Text
    parent Parent

This schema will work fine if Parent is declared before the splice occurs. However, we run into a problem if we want the following:

data Parent = ParentFoo FooId | ParentBar BarId
    deriving (Show, Read, Eq)

This datatype depends on the datatypes introduced by mkPersist, but mkPersist also depends on those datatypes. The natural (for an experienced programmer, anyway) approach is to include the declaration of Parent in the splice:

$(concat <$> sequence [ [d| data Parent = ParentFoo $(conT $ mkName "FooId")
                                        | ParentBar $(conT $ mkName "BarId")
                        |]
                      , derivePersistField "Parent"
                      , mkPersist ps $(persistFileWith lowerCaseSettings modelFile)
                      ])

This approach still has a problem though! Because persistFileWith returns the SqlType unadorned in the FieldDef, the compiler attempts to splice in the Parent datatype in the inner splice, which naturally fails.

This two-stage problem has been seen before, as is evidenced by the code in getSqlType in Database.Persist.TH, which has to work around the issue.

Better, I think, would be to return a lifted Exp, changing the type of the EntityDef spliced in by persistFileWith (and other functions that produce EntityDefs) from EntityDef SqlType to EntityDef SqlTypeExp, as this will allow the splicing of the SqlType to be delayed until the outer splice, where the new datatype is visible and able to be compiled mutually with the rest of the database schema.

I've uploaded a proof-of-concept. It's a bit hacky since it introduces a new type which gets lifted into SqlTypeExp so that SqlTypeExp gets lifted into SqlType on the second splice, but it illustrates the idea and, much more importantly, works.

[gregwebs]

This has been a long-standing issue with persistent, thanks for tackling it! What instances are being used from th-orhpans?

The Lift Exp instance, to allow passing an Exp out from the first splice into the second without needing to actually turn it into real AST.

Probably, the code should be cleaned up by making SqlTypeExp lift into itself rather than straight into SqlType, and running a preprocessing pass in mkPersist that extracts the SqlType. There's a few associated cleanups that could be done inside getSqlType as a consequence.

This would of course risk breaking any code that relies on persistWith & co to actually produce an SqlType, but I'm not sure how much code there is that relies on this, and it would be possible to export the preprocessing pass, which would work fine as long as it wasn't used before the mkPersist splice.

[gregwebs]

can you create a pull request? Then Travis will run the tests automatically

[gregwebs]

I think I made the changes you wanted here around the SqlType on the persistent2 branch.

I have not tested out your more advanced use case yet. Some support already exists for Sum types, but I would definitely like to improve it.

[gregwebs]

This still does not work.

SumTypeTest.hs:76:153:
    Not in scope: type constructor or class `Parent'

SumTypeTest.hs:76:153:
    GHC stage restriction:
      `Parent' is used in a top-level splice or annotation,
      and must be imported, not defined locally
    In an expression type signature: Data.Proxy.Proxy Parent
    In the first argument of `sqlType', namely
      `Data.Proxy.Proxy :: Data.Proxy.Proxy Parent'
    In the fourth argument of `FieldDef', namely
      `sqlType (Data.Proxy.Proxy :: Data.Proxy.Proxy Parent)'

Where's this SumTypeTest.hs? This doesn't appear to be one in the repos.

[gregwebs]

I just pasted your code onto the end of that file which exists under the perstent-test folder. I am not going to commit that though since it doesn't work.

I am on the peristent2 branch now, getting ready to merge it to master.

Ah, ok. I'm not sure why that would happen without looking at the new code.

[snoyberg]

Closing out old issues, please reopen if still relevant