Feature/4263 replace optimizely with mixpanel

[kwonye]

Fixes #4263

Well that was easy...

One thing we do lose is Tracks integration, but since we are using Mixpanel for analytics anyways, it's not a big deal.

To test:

Check out Mixpanel and see that there is a "Magic Link" experiment and that Variation 1 has showMagicLink boolean set to true.

Needs review: @maxme

cc: @aerych

[maxme]

@kwonye did you try it?

I enabled it in Mixapnel and I tried like 20 times (resetting the emulator between each test), I wasn't able to get the magic link screen once.

From the Mixpanel documentation:

Mixpanel checks for any new experiments asynchronously each time your application is opened or resumed. After the response is received, experiment changes and tweaks are applied or removed where appropriate. To handle network availability, each experiment is cached on the device so they can be re-applied when the API call cannot be successfully made.

If you'd like more control over when this check for new experiments occurs, you can use the addOnMixpanelUpdatesReceivedListener listener and the joinExperimentIfAvailable method to download and apply experiments manually.

Also there is something weird, when I enabled it, I saw usage stats grow (current users already "counted" in "variant #1" even if this patch hasn't shipped yet).

[kwonye]

From the Mixpanel documentation:

Thanks @maxme I overlooked the asyncronous start of it. I'll figure out how we can show the correct login screen on start up.

Also there is something weird, when I enabled it, I saw usage stats grow (current users already "counted" in "variant #1" even if this patch hasn't shipped yet).

Yea, there's no way 400+ people saw that... I went ahead and deleted that test to make sure it wasn't doing anything funny. I'll investigate what was going on there.

[kwonye]

After investigation and conversation into what it will take to have this A/B test work with Mixpanel, we're going to leave Optimizely in for the Magic Link A/B test. Moving this PR to 5.7 and removing the Mixpanel magic link boolean.

[kwonye]

Ready to merge AFTER release/5.6 has been cut from develop

[kwonye]

@maxme ready for another pass as we have solid data from 5.6.x's Optimizely A/B test.

By 5.8 we'll remove the magic link test so it doesn't matter that it won't show this A/B test in the beginning.

[maxme]

I'm seeing this:

Can we wait until we have significant numbers?

[kwonye]

@maxme sorry I was unclear, my fault. I meant to say:

I believe we will make a decision about Magic Links by the time we ship 5.8, so we can remove Optimizely right now for develop.

[maxme]

I believe we will make a decision about Magic Links by the time we ship 5.8, so we can remove Optimizely right now for develop.

Ok, but then I'm not sure why you're using this Tweak (Mixpanel A/B test variable) to show the Magic Links or not, should we just wait for the AB test to end and then remove this? (Or eventually use a feature flag if we think we can make that feature better in the future?)

[kwonye]

Again, sorry, my fault. I was already working on having Magic Links be the default SignInActivity in another PR, but that was before you showed me that the significance was actually too low.

I've removed the A/B test from ActivityLauncher and we default to showing the original sign in screen until we make a decision 👍

[maxme]

The extra ; here breaks the build.

[kwonye]

... no excuses here, sorry about that.

[kwonye]

I'm sorry @maxme, that was a careless mistake on my part. Fixed and verified working.

[maxme]

Sorry one last pass:

• can you clean up the unused imports from ActivityLauncher
• and remove the optimizely token from WordPress/gradle.properties-example

[kwonye]

Wow, great catches. Fixed, thanks.

[maxme]

great to have faster launch speed for unsigned users ;)