Not a bug here, but in npm itself: npm/npm#17287
Lost quite some time because of this, so mentioning this here in case others get bitten by this. ember-cli-addon-tests will symlink the addon under test into node_modules of the temporary app, but if you call npm install afterwards (e.g. by calling app.run('npm', 'install')), npm 5 will remove the symlink and install the addon from the npm registry (which is probably not the same code that you are working on). Does not happen with npm 3.