BigQuery: use union categorical to concatenate pages in `to_dataframe` when categorical dtype is requested

[dkapitan]

I am working on an enhancement for pandas-gbq, where we would like to reduce the memory footprint in pandas through downcasting. See googleapis/python-bigquery-pandas#275 for the full details.

I have made a first working version and getting erratic behaviour in the conversion of STRING to pandas category. It seems like it has to do with the size of the query results.

See attached notebook for the code example.
pandas-gbq-bugtracing.ipynb.zip

[tseaver]

@tswast I know you have some stuff in work related to the dtypes conversions.

[tswast]

If the size of the query results is large, then there are multiple pages to parse. Since we construct a dataframe for each page and then concatenate them, the results may not be as expected. Per: http://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html#concatenation

When two categorical series are concatenated with different types, they get converted to the "object" dtype. It seems this might be avoided with https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.types.union_categoricals.html

I'd love if there was a way to support this feature without having to add special logic for different dtypes.

Related: per googleapis/python-bigquery-pandas#275 (comment) there are more immediate ways that we can decrease peak memory usage. I've filed #8107 to track that issue separately.