Skip to content

test: check extreme DATE/DATETIME values can be loaded from pandas DataFrame #1078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gcf-merge-on-green merged 2 commits into from
Dec 2, 2021
Merged

test: check extreme DATE/DATETIME values can be loaded from pandas DataFrame #1078

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a25ec3c
test: check extreme DATE/DATETIME values can be loaded from pandas Da…
tswast Dec 1, 2021
3d3cfcc
Merge branch 'main' into issue1076-extreme-dates
tswast Dec 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 20 additions & 21 deletions tests/system/test_pandas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ def test_load_table_from_dataframe_w_nulls(bigquery_client, dataset_id):
See: https://github.com/googleapis/google-cloud-python/issues/7370
"""
# Schema with all scalar types.
scalars_schema = (
table_schema = (
bigquery.SchemaField("bool_col", "BOOLEAN"),
bigquery.SchemaField("bytes_col", "BYTES"),
bigquery.SchemaField("date_col", "DATE"),
Expand All @@ -283,15 +283,6 @@ def test_load_table_from_dataframe_w_nulls(bigquery_client, dataset_id):
bigquery.SchemaField("ts_col", "TIMESTAMP"),
)

table_schema = scalars_schema + (
# TODO: Array columns can't be read due to NULLABLE versus REPEATED
# mode mismatch. See:
# https://issuetracker.google.com/133415569#comment3
# bigquery.SchemaField("array_col", "INTEGER", mode="REPEATED"),
# TODO: Support writing StructArrays to Parquet. See:
# https://jira.apache.org/jira/browse/ARROW-2587
# bigquery.SchemaField("struct_col", "RECORD", fields=scalars_schema),
)
Copy link
Contributor

@tseaver tseaver Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we just bailing on those two TODO entries?

Copy link
Contributor

@plamut plamut Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, is this now off-plan?

Copy link
Contributor Author

@tswast tswast Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are actually already covered in the v3 branch.

python-bigquery/tests/system/test_pandas.py

Line 294 in 3b3ebff

[[-3], [-2], [-1]],

num_rows = 100
nulls = [None] * num_rows
df_data = [
Expand Down Expand Up @@ -372,7 +363,8 @@ def test_load_table_from_dataframe_w_explicit_schema(bigquery_client, dataset_id
# See:
# https://github.com/googleapis/python-bigquery/issues/61
# https://issuetracker.google.com/issues/151765076
scalars_schema = (
table_schema = (
bigquery.SchemaField("row_num", "INTEGER"),
bigquery.SchemaField("bool_col", "BOOLEAN"),
bigquery.SchemaField("bytes_col", "BYTES"),
bigquery.SchemaField("date_col", "DATE"),
Expand All @@ -387,17 +379,8 @@ def test_load_table_from_dataframe_w_explicit_schema(bigquery_client, dataset_id
bigquery.SchemaField("ts_col", "TIMESTAMP"),
)

table_schema = scalars_schema + (
# TODO: Array columns can't be read due to NULLABLE versus REPEATED
# mode mismatch. See:
# https://issuetracker.google.com/133415569#comment3
# bigquery.SchemaField("array_col", "INTEGER", mode="REPEATED"),
# TODO: Support writing StructArrays to Parquet. See:
# https://jira.apache.org/jira/browse/ARROW-2587
# bigquery.SchemaField("struct_col", "RECORD", fields=scalars_schema),
)

df_data = [
("row_num", [1, 2, 3]),
("bool_col", [True, None, False]),
("bytes_col", [b"abc", None, b"def"]),
("date_col", [datetime.date(1, 1, 1), None, datetime.date(9999, 12, 31)]),
Expand Down Expand Up @@ -464,6 +447,22 @@ def test_load_table_from_dataframe_w_explicit_schema(bigquery_client, dataset_id
assert tuple(table.schema) == table_schema
assert table.num_rows == 3

result = bigquery_client.list_rows(table).to_dataframe()
result.sort_values("row_num", inplace=True)

# Check that extreme DATE/DATETIME values are loaded correctly.
# https://github.com/googleapis/python-bigquery/issues/1076
assert result["date_col"][0] == datetime.date(1, 1, 1)
assert result["date_col"][2] == datetime.date(9999, 12, 31)
assert result["dt_col"][0] == datetime.datetime(1, 1, 1, 0, 0, 0)
assert result["dt_col"][2] == datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)
assert result["ts_col"][0] == datetime.datetime(
1, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc
)
assert result["ts_col"][2] == datetime.datetime(
9999, 12, 31, 23, 59, 59, 999999, tzinfo=datetime.timezone.utc
)


def test_load_table_from_dataframe_w_struct_datatype(bigquery_client, dataset_id):
"""Test that a DataFrame with struct datatype can be uploaded if a
Expand Down