DataFrame.from_records should have an optional "size" parameter

[tebeka]

Currently, data should have a __len__ method. However in some cases I'd like to pass a iterator with a known size (mostly in the case of database, where I can run the SELECT SUM(...) FROM table, for the size. Then SELECT * FROM table and pass the cursor as data.

Currently I need to create in memory copy of the rows in the cursor.

[tebeka]

From looking at DataFrame.from_records code, it looks like it first creates an array from all the rows and then allocate the DataFrame. If this is the case, then we still allocate twice the memory we need.

[wesm]

This is true. It is difficult to prevent this but we can take care of it internally. I'll create a separate issue