[WIP] Converts dataframe to/from named numpy arrays

[thunterdb]

I found this incredibly convenient to create small dataframes, here is how you can use it:

n = 5
A = rd.rand(n,4)
C = rd.randint(10, size=n)
df = conv.pack_DataFrame(a=A, c=C)

DataFrame[a: vector, c: bigint]

And the other conversion. It properly extracts the proper shape for vectors, matrices, etc.

Z = Converter.df_to_numpy(df)
# Each column is strictly equal to the original.
Z['a'] == A
Z['c'] == C

Currently missing are more tests, better names, and sparse vectors. Not sure how easy it is to support these because they have an irregular shape between row. It is probably easier to prevent it and force users to use the CSC conversion that you already wrote.

[jkbradley]

I assume this will be very slow for larger data? That's OK for now.

[thunterdb]

Yes it will; we can always improve it later.

[jkbradley]

The docs should list the supported input types and how they are handled: lists of common types, or lists of vector or numerical array types.

[thunterdb]

Done. I also added that we support a subset of numpy types (there are so many) and sql types.

[jkbradley]

Just had a couple more comments.

[thunterdb]

@jkbradley comments addressed

[vlad17]

This PR shoul unskip the following: test_cv_lasso_with_mllib_featurization (spark_sklearn.tests.test_grid_search_2.CVTests) ... SKIP: disable this test until we have numpy <-> dataframe conversion

[srowen]

I'm starting to look through the open PRs to see if we can merge them or whether they're stale -- @thunterdb is this one too old to resurrect?