Index dataframes tables #716
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gonzaponte
requested changes
Apr 8, 2020
Collaborator
gonzaponte
left a comment
gonzaponte
left a comment
There was a problem hiding this comment.
Cool. I would like just one more test to be added.
gonzaponte
reviewed
Apr 9, 2020
invisible_cities/io/dst_io.py
Outdated
| from typing import Optional | ||
| from typing import Sequence | ||
|
|
||
| def _decode_df_to_str(df): |
Collaborator
There was a problem hiding this comment.
how about decode_str_columns?
invisible_cities/io/dst_io_test.py
Outdated
| with pytest.warns(UserWarning, match='does not exist'): | ||
| load_dsts([good_file, wrong_file], group, node) | ||
|
|
||
| def test_load_dst_converts_from_bytest(ICDATADIR, fixed_dataframe): |
Collaborator
There was a problem hiding this comment.
Suggested change
| def test_load_dst_converts_from_bytest(ICDATADIR, fixed_dataframe): | |
| def test_load_dst_converts_from_bytes(ICDATADIR, fixed_dataframe): |
small typo
mmkekic
force-pushed
the
fix_pandas_to_pytables
branch
from
cecbcc0 to
eccc348
Compare
mmkekic
force-pushed
the
fix_pandas_to_pytables
branch
2 times, most recently
from
b42e5be to
81e02d6
Compare
gonzaponte
approved these changes
Apr 10, 2020
Collaborator
gonzaponte
left a comment
gonzaponte
left a comment
There was a problem hiding this comment.
An improved version of the dataframe writer with table indexation. The writer is now renamed to df_writer, which is much more comfortable. This PR also includes the indexation of the event column in the output of Beersheva and Esmeralda.
This change suggests that we should ensure each city indexes its output properly, but that is left for a different PR.
The new feature is properly and thoughtfully tested. Nice job!
mmkekic
force-pushed
the
fix_pandas_to_pytables
branch
from
81e02d6 to
2eb3d05
Compare
The tests check: all elements in function argument are in tables.attrs.columns_to_index; KeyError with column name is raised if input list is not subset of dataframe columns; the argument is not given nothing is flagged.
The writer is now labeling columns_to_be_index. The table is not indexed since it should be done after the file is fully written.
The test where dataframe is read from Kr83_full_nexus_v5_03_01_ACTIVE_7bar_1evt.sim.h5 file that contains strings is added, and the test_store_pandas_as_tables_exact does not decode explicitly.
Since the writer has different structure than others the auxiliary _df_writer had to be defined
bpalmeiro
force-pushed
the
fix_pandas_to_pytables
branch
from
2eb3d05 to
794c684
Compare
andLaing
added a commit
to andLaing/IC
that referenced
this pull request
Apr 10, 2020
Remove _decode_str_columns which is now executed in load_dst. Change name of function used in mc_writer to new name approved in PR next-exp#716
andLaing
added a commit
to andLaing/IC
that referenced
this pull request
May 4, 2020
Remove _decode_str_columns which is now executed in load_dst. Change name of function used in mc_writer to new name approved in PR next-exp#716
andLaing
added a commit
to andLaing/IC
that referenced
this pull request
May 4, 2020
Remove _decode_str_columns which is now executed in load_dst. Change name of function used in mc_writer to new name approved in PR next-exp#716
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PR adds option to flag tables columns to be indexed.