CSV ingest improvements

[landreev]

(this issue is for a clearly defined, short term goal; not another generic "improve xxx" issue;)

The original CSV parser was purposefully restrictive; strict formatting - one line per observation (no new lines in fields), fixed number of commas per line, etc. These requirements are no longer relevant. At the same time Gary specifically requested the CSV ingest to handle full text - with escaped new lines and rich punctuation, etc. Example: the file posts_all.tab (csv original) in https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/QSZMPD.

One way to define the goal would be to say that any Google/Excel spreadsheet columns exported as CSV should be parseable by our ingest. (I will add more details on how they escape punctuation characters and such).

A sensible way to achieve this would be to switch to some available open source parser (Apache seems like a good candidate), rather than maintaining our own.

[raprasad]

code

>>> import pandas as pd
>>> df = pd.read_csv('posts_all.csv')
>>> df.columns
Index([u'What.is.the.city.', u'folder', u'file',
       u'What.is.the.name.of.the.organization.making.posts.', u'url',
       u'content', u'site', u'What.is.the.account.name.of.the.person.posting.',
       u'PostDate', u'category', u'textseg'],
      dtype='object')
>>> df['What.is.the.city.'].describe()
count         43757
unique            1
top       Zhanggong
freq          43757
Name: What.is.the.city., dtype: object
>>> df.describe()
         category
count  188.000000
mean     3.936170
std      0.381624
min      3.000000
25%      4.000000
50%      4.000000
75%      4.000000
max      5.000000
>>> df['What.is.the.name.of.the.organization.making.posts.'].describe()
count     43722
unique      247
top         网宣办
freq       9159
Name: What.is.the.name.of.the.organization.making.posts., dtype: object

etc, etc

[pdurbin]

@raprasad heh. @mercecrosas also just mentioned to us that we could use the readxl R package, which just reached 1.0: https://blog.rstudio.org/2017/04/19/readxl-1-0-0/ . Using either Python or R for this sounds more like #2331 to me.

Back at #585 (comment) @bencomp mentioned using https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVParser.html , which I believe is what @landreev was referring to above.

In short, we need to decide what's in scope for this issue. I thought when we estimated this issue as an 8 on Wednesday we were talking about the Java/Apache Commons route, not the Python or R route. Perhaps we should re-estimate this issue and/or estimate #2331.