Missing related data when inserting data into new item

[johanskoldekrans]

I fetch the data from the database into a new declared object and I dont get the related data like I did in version 2.0.*.

Example:

public class SomeItem
{
     public int Id {get; set;}
     public List<OtherTable1> OtherTable1 {get;set}
}

var data = ctx.SomeTable.AsNoTracking()

.Include(x => x.OtherTable1)
.ThenInclude(x => x.OtherTable2)
.ThenInclude(x => x.SomeOtherTable)

.Include(x => x.OtherTable1)
.ThenInclude(x => x.OtherTable2)
.ThenInclude(x => x.SomeTypeTable)

.Select(c => new SomeItem {
    Id = c.Id,
    OtherTabledata = OtherTable1.ToList() <--This is missing the data from OtherTable2
}).ToList();

In my real life example OtherTable1 has a lot of Include/ThenInclude/ThenIncludes.

should I be doing something different in 2.1 or is this a bug/known issue?

No exceptions... except null exceptions...

Further technical details

EF Core version: 2.1
.Net Core version: 2.1
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Windows 2016
IDE: Visual Studio 2017 15.7.2

[ajcvickers]

@smitpatel @maumar Can you look into this as a possible regression from 2.0?
/cc @divega

[smitpatel]

Correlated subquery optimization causes the include annotations going through navigation being optimized to be ignored hence the navigation did not get populated.
Work-around is to remove ToList to have 2.0 behavior.

[johanskoldekrans]

@smitpatel Sorry the workaround didnt help. Made it into an ICollection but same result. Looking at the SQL Profiler it still seems to be optimized away. The data isn't read from the database.

[johanskoldekrans]

@smitpatel Sorry, just to be clear. I removed the .ToList() and changed the property in the class to an ICollection.

[smitpatel]

@johanskoldekrans - Thanks for pointing out. I verified that there is no work-around yet.

[maumar]

if SomeTable entity doesn't have too many properties, you can do something like this as a workaround:

var data = ctx.SomeTable.AsNoTracking()
.Include(x => x.OtherTable1)
.ThenInclude(x => x.OtherTable2)
.ThenInclude(x => x.SomeOtherTable)
.ToList()
.Select(c => new SomeItem {
    Id = c.Id,
    OtherTabledata = c.OtherTable1.ToList()
}).ToList();

[johanskoldekrans]

@maumar Yes I know. But it is a gazillion times slower and I would never do that even if my life depended on it. :-)

[johanskoldekrans]

I measured before and it was about 35 times slower..

[maumar]

There are multiple problems here. For cases when the root entity is not in the final projection (e.g. because Select/SelectMany is used on navigation), the recommended pattern should be to apply include after the projection, like its done here: #11053

However, if the Include is applied directly on projected collection (i.e. without the select/selectmany like in #11053), AsQueryable call is needed in order to compose include on top of it. That in turn blocks the correlated collection optimization (#12195), because AsQueryable is a result opearator - so we get N+1 queries in that case.

Moreover, if the included property is a collection, we generate invalid query plan because correlated collection and include don't work well together (both modifying and cloning the QM - that should potentially be addressed by the include unification - #10736

[maumar]

@johanskoldekrans the performance hit you are seeing is unexpected. Are you seeing 35x hit when comparing the workaround I suggested with the 2.0.2 query?

Strange thing is that 2.0.2 was essentially performing the workaround query under the covers. When the query was projecting a "naked" collection navigation we would convert that into include, which then merged with includes that you specified explicitly and thats why it all worked. So in theory, the performance between 2.0.2 and 2.2 using the include workaround should be very similar.

In 2.2 we delegated this into the correlated collection optimization pipeline and that currently doesn't play well with additional collection includes.

Could you provide a repro project (entity classes query as well as approximate size of the data - i.e. how many entities of each type) so that we can see whats going on?

[johanskoldekrans]

@maumar I have rewritten all of the affected viewcomponents. I removed the original includes and add them later on using ForEach on the List after I have executed the main query.

The tables arent that huge at all, maybe a couple of hundred thousand rows and good indices on them.

I think the performance hit I got was due to a mix of problems with 15.7.2 of Visual Studio and the Resharper extension I have. After upgrading to 15.7.3 everything is a lot smoother both debugging 2.1 as well as the performance of the queries. I could revert the code but I don't think that ef was the culprit.

I will try the pattern you suggested in #11053 and let you know.

But the 20-30 ms to 3,3 s hit a got was probably due to something else. :-)