BLD/CI: fix arm64 builds

https://travis-ci.org/github/pandas-dev/pandas/jobs/727691006

these are failing (and have been for a while).

I believe we *can* now use the numpy wheels so this should make this build fast enough to include in the main matrix (and avoid using the often ignored allow_failures).

- [ ] use wheels for numpy
- [ ] remove from the allow_failures matrix

this could be for 1.1.3 or 1.2