Support `msgpack` format as `to_td` option

[akito19]

Purpose

Import object datatype values of DataFrame to Treasure Data as string type.

Behaviors

Currently, when we try to import these data into Treasure Data by using pandas_td.to_td, the values would be not necessarily that these are object:

e.g.)

import pytd.pandas_td as td
import pandas as pd

df = pd.DataFrame([{'id':'00123', 'num': '45678'}])
con = td.connect()

td.to_td(df, 'kasai_test.pytd', con, if_exists='replace')

In this case, both id and num are stored as long type in Treasure Data in spite of object type in DataFrame.
I guess that converting to CSV might cause this result since CSV doesn't have typing system.

However, fmt='msgpack' has been available in calling BulkImportWriter.write_dataframe since version 1.0.0.
Thus, we can store these values as string if we use the write_dataframe explicitly like below:

import pytd
import pandas as pd

df = pd.DataFrame({'id': '00324', 'test': '5678'}, index=range(2))
client = pytd.Client()

table = pytd.table.Table(client, "kasai_test", "pytd")
writer = pytd.writer.BulkImportWriter()
writer.write_dataframe(df, table, if_exists="overwrite", fmt="msgpack")

Because pytd has already enabled choosing format when using bulk_import, users might expect to be able to set format as well as write_dataframe.
Thus, I propose that pytd supports msgpack format in to_td options.

What do you think about it?

[chezou]

@akito19 Thanks for raising the issue. We also faced the same issue with td-client-python and the cause was reading CSV for bulk import may break type information for 0-padded number conversion.
treasure-data/td-client-python#83

I think there are two options to deal with the issue:

1. Support fmt="msgpack" to to_td option. This should be an easy fix for the issue.
2. Make fmt="msgpack" as default rather than fmt="csv".

For option 2, although this would introduce type inconsistency i.e., "00012" was ingested as integer 12 with CSV format, but will be ingested as "00012", I think this behavior should be unintentional and this is against to the principle of least astonishment.

[chezou]

@akito19 We merged #80 which allows passing fmt="msgpack" as an argument of to_td function. Could you try master branch? Detailed usage is written in the PR #80.

@takuti How do you feel to make fmt="msgpack" is default instead of CSV?

[akito19]

@chezou Thank you for the update!
I confirmed fmt="msgpack" was available in to_td fuction.

>>> import pytd.pandas_td as td
>>> import pandas as pd
>>> df = pd.DataFrame([{'id':'00123', 'num': '45678'}])
>>> client = td.Client()
>>> td.to_td(df, 'kasai_test.pytd', client, if_exists='replace', fmt='msgpack')
uploading data converted into a msgpack file
performing a bulk import job
[job id 708942835] imported 1 records.

[takuti]

@chezou Let's first improve documentation of fmt option e.g., describing the known issue of using fmt="csv" in:

pytd/pytd/writer.py

Lines 336 to 345 in 8d9a79f

	fmt : {'csv', 'msgpack'}, default: 'csv'
	Format for bulk_import.
	- csv
	Convert dataframe to temporary CSV file. Stable option but slower
	than msgpack option because pytd saves dataframe as temporary CSV file,
	then td-client converts it to msgpack.
	- msgpack
	Convert to temporary msgpack.gz file. Fast option but might be
	unstable because of skipping ``tdclient.API._prepare_file()``

Plus, simply copying-n-pasting docstrings of fmt and keep_list options to pytd.pandas_td.to_td and pytd.Table.import_dataframe would be helpful for users' reference.

When we become able to say fmt="msgpack" is stable enough, that'd be the time to change default fmt to msgpack. We improved MessagePack handling a lot lately, so do you think the file format is safe enough?

[chezou]

@takuti Agreed to add docstring and README.rst.

pandas 1.0 support improves fmt="msgpack" type conversion stability a lot, and the only difference I can come out with is 0-pad string handling, and that is unintended behavior. But, I'm ok to wait a certain time after updating the doc for to_td.

[chezou]

We now updated the document and supported fmt option on "to_td" function. Will create another issue for making fmt="msgpack" default option.