Iterableiterator

[toobaz]

If the approach is not overkill, it can be used also in UTF8Recoder and UnicodeReader from pandas/common/io.py. Closes #12153.

[jreback]

see #9496

[jreback]

move to pandas/io/common.py, inherit from object, call this BaseIterable, raise AbstractMethodError for __next__

[jreback]

make a parent class of Boto and UTFReader as well in`io/common.py``

[jreback]

you can also incorporate elements of TableIterator in io/pytables.py (and make this a sub-class)

[toobaz]

BaseIterable or BaseIterator? True, it makes an iterable out of an iterator, but most iterables will not be iterators.

[max-sixty]

inherit from object?

[toobaz]

TableIterator in io/pytables.py is still a bit too cryptic to me, but I can try to take a look at it next week.

[toobaz]

Each chunk has an index restarting from 0 (also in master). Isn't this a bug?

[jreback]

No, that shouldn't be the case

In [46]: s = DataFrame({'A' : range(10)}).to_csv(None)

In [47]: s
Out[47]: ',A\n0,0\n1,1\n2,2\n3,3\n4,4\n5,5\n6,6\n7,7\n8,8\n9,9\n'

In [48]: pd.read_csv(StringIO(s),index_col=0)
Out[48]: 
   A
0  0
1  1
2  2
3  3
4  4
5  5
6  6
7  7
8  8
9  9

In [49]: list(pd.read_csv(StringIO(s),index_col=0,chunksize=4))
Out[49]: 
[   A
 0  0
 1  1
 2  2
 3  3,    A
 4  4
 5  5
 6  6
 7  7,    A
 8  8
 9  9]

[toobaz]

I mean when the index is not loaded:

In [7]: list(pd.read_csv(StringIO(s), chunksize=4))
Out[7]: 
[   Unnamed: 0  A
 0           0  0
 1           1  1
 2           2  2
 3           3  3,    Unnamed: 0  A
 0           4  4
 1           5  5
 2           6  6
 3           7  7,    Unnamed: 0  A
 0           8  8
 1           9  9]

Wouldn't we prefer a consistent index?

[jreback]

hmm yes I agree

you'd have to keep state in the in the iterators I think though -
I do this is HDFStore iirc

but might be a bit non trivial

if u think u can fix easily - go ahead
otherwise create an issue

[toobaz]

No, I'd like to do it but it probably won't be immediate: #12185 .

[jreback]

ok on doing HDFStore iterator in another issue.

[jreback]

AbstractMethodError(self)

[toobaz]

I removed edits to BotoFileLikeReader because as far as I can judge the failing tests were due to it (through its ancestor key.Key) already being an iterable, and knowing nothing about this format/service it's hard for me to debug.

[jreback]

can you add tests for read_stata & read_sas

[toobaz]

yep, done

[jreback]

thanks @toobaz

I have marked off csv,stat,sas, remaining are hdf & msgpack which are in #9496