Skip to content

Add Notification Eager Load Support#281

Merged
Gummibeer merged 15 commits intofenos:developfrom
gaussian1:fix-280
Jul 19, 2017
Merged

Add Notification Eager Load Support#281
Gummibeer merged 15 commits intofenos:developfrom
gaussian1:fix-280

Conversation

@gaussian1
Copy link

@gaussian1 gaussian1 commented Jun 24, 2017

The Notification Parser made a direct call to the NotificationCategory::findOrFail function resulting in a n+1 query error, even if the notifications were already eager loaded.

This PR does the following:

  1. Fixes the N+1 Problem In the Notification Parser (Fixes Eager Loading Notification Categories #280)
  2. Adds a configuration parameter to manage Notification eager loading

Implementation Details:

:octocat: [Bug Fix] Fixes the N+1 Problem In the Notification Parser: (Fixes #280)
In src/Notifynder/Parsers/NotificationParser.php: NotificationCategory::findOrFail function call was replaced by a model relation call:

public function parse($notification, $categoryId)
{
  $category = $notification->category;
  if (is_null($category)) {
    throw (new ModelNotFoundException)->setModel(
      NotificationCategory::class, $notification->category_id
    );
  //...
}

❤️ [UX] Adds An eager loading configuration parameter called 'eager_load':
Each time a user wants to use eager loading he has to write something like this:

$notifications = $user->getNotificationRelation()
                      ->with('category')
                      ->get()

It is better to have an 'eager_load' configuration parameter to specify whether users want eager loading or not in their projects. The parameter can take any of these values:

  • true: Always eager load relations (from, to, category)
  • false: (default) Don't eager load any relations
  • array: Specify which relations to eager load

The getNotificationRelation was divided into two methods:

  1. A lazy loading relation: getLazyNotificationRelation to get the relations without eager loading.
public function getLazyNotificationRelation()
{
  return $this->notifications();
}
  1. A configuration based relation: getNotificationRelation to eager load the relations according to the configuration.
public function getNotificationRelation($eagerLoad = null)
{
  $with = [];
  if (is_null($eagerLoad)) {
    $eagerLoad = notifynder_config('eager_load', false);
  }

  if ($eagerLoad === true) {
    $with = ['category', 'from', 'to']; // all relations
  } elseif (is_array($eagerLoad)) {
    $with = $eagerLoad;
  }

  return $this->getLazyNotificationRelation()->with($with);
}

Call Modifications:
Modifications were made to ignore the configuration parameter and use lazy loading for the following functions in src/Notifynder/Traits/NotifableBasic.php

  1. updateSingleReadStatus
  2. readAllNotifications
  3. unreadAllNotifications
  4. countUnreadNotifications

Electronically Yours,
Looking forward to your feedback!


EDIT: This next section discusses other eager loading options instead of using a configuration parameter, these options didn't make it to the Actual PR.. you can ignore it, or keep reading if you are interested:

  • Always Eager Load (Bad for queries that just do counts and updates)
/* src/Notifynder/Traits/Notifable */
public function getNotificationRelation()
{
  return $this->notifications()->with('category');
}
/* src/Notifynder/Traits/NotifableLaravel53 */
public function getNotificationRelation()
{
  return $this->notifynderNotifications()->with('category');
}
  • Don't make any changes: the programmer will have to eager load manually
$notifications = $user->getNotificationRelation()
                      ->with('category')
                      ->limit(4)
                      ->orderBy('created_at', 'desc')
                      ->get();
  • Make the eager loading further downwards at the client code like inside getNotifications(), getNotificationsNotRead()
/* src/Notifynder/Traits/NotifableBasic */
public function getNotifications($limit = null, $order = 'desc')
{
  $query = $this->getNotificationRelation()->with('category')->orderBy('created_at', $order);
  // ...
}
/* src/Notifynder/Traits/NotifableBasic */
public function getNotificationsNotRead($limit = null, $order = 'desc')
{
  $query = $this->getNotificationRelation()->with('category')->byRead(0)->orderBy('created_at', $order);
  // ...
  • Add a boolean for eagerloading (default false) in the getNotificationRelation method
/* src/Notifynder/Traits/Notifable */
public function getNotificationRelation($eagerLoadCategory = false)
{
  return $eagerLoadCategory ? $this->notifications()->with('category') : $this->notifications();
}

/* src/Notifynder/Traits/NotifableLaravel53 */
public function getNotificationRelation($eagerLoadCategory = false)
{
  return $eagerLoadCategory ? $this->notifynderNotifications()->with('category') : $this->notifynderNotifications();
}

@coveralls
Copy link

coveralls commented Jun 24, 2017

Coverage Status

Coverage increased (+1.4%) to 88.037% when pulling 05d3859 on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 24, 2017

Coverage Status

Coverage decreased (-0.4%) to 86.174% when pulling 69dc0aa on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 24, 2017

Coverage Status

Coverage decreased (-0.4%) to 86.174% when pulling 69dc0aa on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@gaussian1 gaussian1 changed the title Use Notification model category relation to allow for category eager … Remove Category::findOrFail from Notification Parser Jun 25, 2017
@gaussian1 gaussian1 changed the title Remove Category::findOrFail from Notification Parser Remove Category::findOrFail from NotificationParser Jun 25, 2017
@gaussian1 gaussian1 changed the title Remove Category::findOrFail from NotificationParser Remove NotificationCategory::findOrFail from NotificationParser Jun 25, 2017
@coveralls
Copy link

coveralls commented Jun 25, 2017

Coverage Status

Coverage increased (+1.1%) to 87.689% when pulling 274d4ee on gaussian1:fix-280 into fa9d1a3 on fenos:master.

2 similar comments
@coveralls
Copy link

Coverage Status

Coverage increased (+1.1%) to 87.689% when pulling 274d4ee on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

Coverage Status

Coverage increased (+1.1%) to 87.689% when pulling 274d4ee on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 25, 2017

Coverage Status

Coverage increased (+1.1%) to 87.689% when pulling 274d4ee on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 25, 2017

Coverage Status

Coverage increased (+1.1%) to 87.689% when pulling 274d4ee on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@Gummibeer
Copy link
Collaborator

I think a combination of the last two points would be the best - so add a boolean-flag in getNotificationRelation() and set it true in the wrapper methods getNotifications() and getNotificationsNotRead() etc.
Cause there the will be needed everytime and it's easy for the developer to load them.

On the other side, if we add eagerloading, it would be good to be able to load all relations (category, from, to). So what about a solution like:

/**
 * @param $eagerLoad array|bool
 */
public function getNotificationRelation($eagerLoad = false)
{
    $with = null;
    if($eagerLoad === true) {
        $with = ['category', 'from', 'to']; // all relations
    } elseif(is_array($eagerLoad)) {
        $with = $eagerLoad;
    }
    return $this->notifications()->with($with);
}

With this you can load nothing, everything or a specific set of relations.

@gaussian1
Copy link
Author

gaussian1 commented Jun 26, 2017

That's a nice solution!

One concern might be should the default value for $eagerLoad be false or true:

  • Argument for default false:
    The cost of doing an eager load by mistake just to update/count is too high.

  • Argument for default true:
    Less things to worry about for the programmer who is using the getNotificationRelation directly instead of the query helpers.

@Gummibeer
Copy link
Collaborator

For this I'm for keep itlike it is so atm we don't eager load anything and this should keep. Current running implementations shouldn't show a different behaviour just cause of a package update. So everyone who likes eagerloading can add it but we don't force developers to disable it in running projects.
The only way arround could be a new config value that is false by default. Than the default value of the parameter itself should be null.

/**
 * @param $eagerLoad array|bool
 */
public function getNotificationRelation($eagerLoad = null)
{
    $with = null;
    if(is_null($eagerLoad)) {
        $eagerLoad = notifynder_config('eager_load');
    }
    if($eagerLoad === true) {
        $with = ['category', 'from', 'to']; // all relations
    } elseif(is_array($eagerLoad)) {
        $with = $eagerLoad;
    }
    return $this->notifications()->with($with);
}

@gaussian1
Copy link
Author

gaussian1 commented Jun 26, 2017

I've added the eager_load configuration parameter and modified the code accordingly, I will be waiting for your feedback.

A Note on database-based tests:
I removed the previous database based query number testing, because it felt more like feature/e2e testing than unit testing. However, I couldn't think of a simple unit test that could detect something like the NotificationCategory::findOrFail call that bypassed eager loading and forced the query each time it was called.
I think to prevent an issue like that from happening in the future we can create a dedicated class for those "database based" tests which counts the queries, what do you think?

@coveralls
Copy link

coveralls commented Jun 26, 2017

Coverage Status

Coverage increased (+0.5%) to 87.061% when pulling 8abfa74 on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 26, 2017

Coverage Status

Coverage increased (+0.5%) to 87.061% when pulling 0c09dec on gaussian1:fix-280 into fa9d1a3 on fenos:master.

$category = NotificationCategory::findOrFail($categoryId);
$category = $notification->category;
if (is_null($category)) {
throw (new ModelNotFoundException)->setModel(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you make the $categoryId param useless - I'm not 100% sure if this is ok. I think that we can remove it and see what happens - unittests & community.

Copy link
Author

@gaussian1 gaussian1 Jun 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively we could set it to null, and if it is supplied, use the NotificationCategory::findOrFail to be backwards compatible, but I think its a bit of an overkill.

public function parse($notification, $categoryId = null)
{
  if (is_null($categoryId)) {
    $category = $notification->category;
    if (is_null($category)) {
      throw (new ModelNotFoundException)->setModel(
        NotificationCategory::class, $notification->category_id
      );
    }
  } else {
    $category = NotificationCategory::findOrFail($categoryId);
  }

* @return \Illuminate\Database\Eloquent\Relations\HasMany|\Illuminate\Database\Eloquent\Relations\MorphMany
*/
public function getNotificationRelation()
private function getLazyLoadedNotificationRelation()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls don't use private we are open for overrides and customization - so at least protected. Even if I don't get it 100% why we need it here? We can control the eager loading with the new parameter - so this is just a wrapper for getNotificationRelation(false).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the getLazyLoadedNotification method to avoid code duplication of the getNotificationRelation logic. Otherwise we would need to write the same logic in Notifable and NotifableL53

$this->assertModelHasNoLoadedRelations($notifications[0], ['to']);
}

private function assertModelHasLoadedRelations($model, $relationNames = [])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls protected

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although it is just a helper test function, I will modify it to be protected for the the sake of Open/Closed Principle

}
}

private function assertModelHasNoLoadedRelations($model, $relationNames = [])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls protected

@coveralls
Copy link

coveralls commented Jun 27, 2017

Coverage Status

Coverage increased (+2.9%) to 89.522% when pulling c517aff on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@gaussian1
Copy link
Author

gaussian1 commented Jun 27, 2017

I've discovered an issue, since the default behaviour of getNotificationRelation is to read the configuration parameter eager_load. If the eager_load config is set to true, it will always eager load even in count/update functions. I will have to strictly call getLazyNotificationRelation() where eager loading is not needed like for counts and updates. I will be updating the code to do that.

@coveralls
Copy link

coveralls commented Jun 27, 2017

Coverage Status

Coverage increased (+2.9%) to 89.524% when pulling 349ed53 on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 27, 2017

Coverage Status

Coverage increased (+3.01%) to 89.603% when pulling cd0d244 on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@coveralls
Copy link

coveralls commented Jun 27, 2017

Coverage Status

Coverage increased (+2.9%) to 89.524% when pulling 51a2ea4 on gaussian1:fix-280 into fa9d1a3 on fenos:master.

@gaussian1 gaussian1 changed the title Remove NotificationCategory::findOrFail from NotificationParser Add Notification Eager Load Support Jun 27, 2017
@gaussian1
Copy link
Author

gaussian1 commented Jun 28, 2017

All modifications are done, I am waiting for your feedback on the remaining review point.

@gaussian1
Copy link
Author

:octocat: Another PR lost in review, the place where most PRs die and only a brave few ever come back.

✌️ Farewell PR, You started as a simple query fix, increased in scope to include the actual eager loading, and now you are lost forever.🚶

@Gummibeer I will keep this PR open, maybe one day you would have enough time to review it, I understand how hectic it is to manage multiple repos 📦 📦, last thing I want to become is a feature troll.

@Gummibeer
Copy link
Collaborator

@gaussian1 Hey, I'm super sorry for the delay. I had a bicycle accident last week and prepare two new releases at work.

Your PR isn't lost or forgot. But I also don't want to merge it without a check - cause of the amount/chain of changes.
I can't/won't promise anything but I think that I will have time end of week or at the weekend.

Again sorry and I hope that you understand it!?

@Gummibeer
Copy link
Collaborator

I don't know if @fenos has time to check this?

@gaussian1
Copy link
Author

@Gummibeer Thanks for your quick reply, you don't need to apologize for anything. Sorry to hear about your bicycle incident 🤕

Take all the time you need, I was just poking you out 😋

@Gummibeer Gummibeer changed the base branch from master to develop July 19, 2017 13:24
@Gummibeer
Copy link
Collaborator

@gaussian1 Hey, sorry again for the long delay! I've issued two new builds on travis (master & develop) branch to be sure that everything works like expected and will review what you've done now.
I've already changed the base branch of your PR to develop to let this PR take the normal way with a rc tag.

@Gummibeer Gummibeer merged commit 31e9341 into fenos:develop Jul 19, 2017
@Gummibeer
Copy link
Collaborator

@gaussian1 thanks for this big PR - it's merged into develop now and after travis build is successful I will create a rc tag.
https://travis-ci.org/fenos/Notifynder/builds/255268398

@gaussian1 gaussian1 deleted the fix-280 branch July 19, 2017 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants