ARROW-13138: [C++][R] Implement extract temporal components (year, month, day, etc) from date32/64 types #11075

aucahuasi · 2021-09-03T03:56:36Z

https://issues.apache.org/jira/browse/ARROW-13138

github-actions · 2021-09-03T03:56:53Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2021-09-03T12:37:47Z

https://issues.apache.org/jira/browse/ARROW-13138

lidavidm

Thanks for tackling this, this looks like a great start. I think my main questions are 1) whether we should support strftime, hour, minute, etc. on dates as I don't think they're meaningful, and 2) whether we can share the implementation of ISOCalendar instead of duplicating it.

lidavidm · 2021-09-03T13:30:50Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -466,8 +492,90 @@ struct Nanosecond {
 // Convert timestamps to a string representation with an arbitrary format

 #ifndef _WIN32
-template <typename Duration>
+template <typename Duration, typename InType>
 struct Strftime {


nit: since this is for dates only, maybe have the base struct Strftime have no implementation, and implement both dates and timestamps as specializations?

Actually, given that the regular strftime kernel won't format timestamps without timezones, I wonder if it's meaningful to have strftime work on dates. I would say if you want a string representation of a date, you should cast, since strftime would let you do misleading things like trying to extract the hour from a date.

I think you are right. Do you know if we have tests for casting date32/64 to strings?

We do, they're a bit thin though:

arrow/cpp/src/arrow/compute/kernels/scalar_cast_test.cc

Line 1245 in 5ead375

TEST(Cast, DateToString) {

Actually, given that the regular strftime kernel won't format timestamps without timezones, I wonder if it's meaningful to have strftime work on dates.

There's work to have Strftime work on timestamps without timezones (assuming UTC)
#10998. It's a philosophical discussion if it's correct or not but it will be available.

Sorry for the back-and-forth @aucahuasi. However, it might be prudent to split strftime support into a separate JIRA to minimize conflicts with the existing PR and so we can evaluate how best to share the implementation (to avoid too much code duplication).

No worries @lidavidm :)
Indeed, it makes sense to have a new jira. I can create the ticket if there is none.

I don't think there's a JIRA so please do file one.

Here is the jira ticket for strftime, thanks @lidavidm @rok
https://issues.apache.org/jira/browse/ARROW-13916

The Strftime ticket was merged yesterday so maybe something has changed here. Dunno.

cpp/src/arrow/compute/kernels/scalar_temporal_test.cc

lidavidm · 2021-09-03T13:36:00Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -588,8 +701,58 @@ inline std::array<int64_t, 3> GetIsoCalendar(int64_t arg, Localizer&& localizer)
          static_cast<int64_t>(weekday(ymd).iso_encoding())};
 }

-template <typename Duration>
+template <typename Duration, typename InType>
 struct ISOCalendar {


Would it be possible to avoid specialization by instead making the helper functions used work with timestamps and dates? For instance, you could modify GetInputTimezone to be GetInputTimezone<InType> where it would always return an empty string for InType == Date32Type.

(I suppose you'd also need to change functions to templates to be able to specialize things like that...)

Thanks @lidavidm !

I think the provided implementation (using specialization) is good enough.
Please note that this function is returning a const ref string:
const std::string& GetInputTimezone
So the compiler will throw an error if we want to return local empty strings.
And if I change the declaration to return a string (without const ref) we will be copying the timezones instead of using the refs.

Also, for the case of ISOCalendar with TimestampType we had this code:
string timezone = GetInputTimezone
and we had been performing a copy.
So I changed that to
const auto& timezone = GetInputTimezone

We could get around that with a static string somewhere, or by returning const std::string*. Or the actual implementation could be factored out, and the date kernel could call it with a literal string. At the very least, I'm not super enthused about essentially copy-pasting this kernel.

Sure, that's a good idea, I can try to have a single implementation.
Where should I put the empty static string? in which file?

It would go in this file, though FWIW, it would probably be easiest to just factor things into functions since these are all static methods anyways.

@lidavidm I think this last version reuse the kernel logic and it specializes only the parts related to the timestamp logic. Please let me know if it is good enough.
TODO: I need to fix the some R tests (I'll tackle this tomorrow)

lidavidm · 2021-09-03T13:40:03Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -670,26 +833,36 @@ std::shared_ptr<ScalarFunction> MakeTemporal(
  auto func =
      std::make_shared<ScalarFunction>(name, Arity::Unary(), doc, default_options);

+  {


Frankly I think kernels like hour should just not work on dates - it's not meaningful.

Make sense, pandas doesn't has that behavior too

pd.to_datetime("2019-01-03", format='%Y-%m-%d', errors='coerce').date().hour() AttributeError: 'datetime.date' object has no attribute 'hour'

I'll remove the support of date32/64 for hour, min, sec ...

lidavidm · 2021-09-03T17:20:09Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

    class ExecTemplate,
    typename OutType>
-std::shared_ptr<ScalarFunction> MakeTemporal(
+std::shared_ptr<ScalarFunction> MakeTemporal(bool enable_date,


Instead of a bool, I think it'd be cleaner to split the date kernel registration into a separate function:

template < template <typename...> class Op, template <template <typename...> class OpExec, typename Duration, typename InType, typename OutType> class ExecTemplate, typename OutType> std::shared_ptr<ScalarFunction> AddDateKernels(ScalarFunction* func, const std::shared_ptr<arrow::DataType>& out_type, KernelInit init = nullptr) { { auto exec = ExecTemplate<Op, days, Date32Type, OutType>::Exec; DCHECK_OK(func->AddKernel({date32()}, out_Type, std::move(exec), init); } { auto exec = ExecTemplate<Op, std::chrono::milliseconds, Date64Type, OutType>::Exec; DCHECK_OK(func->AddKernel({date64()}, out_Type, std::move(exec), init); } }

Then you can just call (for instance) AddDateKernels<Year, TemporalComponentExtract, Int64Type>(year.get(), int64()); below.

Thanks, but I suggest to keep the provided implementation in this PR:

It requires less code/It change less parts.

It integrates well into the MakeTemporal* functions.

It is setting up the date kernels (optionally) at the moment of creating the function.

Also, I think that if I add AddDateKernels we would need 2 versions: one for MakeTemporal and other for MakeSimpleUnaryTemporal.
So I think it makes sense to keep the current code. Let me know what do you think here @lidavidm

Hmm, it's reasonable. In that case, below, can we do the following, so it's clear what the bare boolean parameter is for?

auto quarter = MakeTemporal<Quarter, TemporalComponentExtract, Int64Type>( /*enable_date=*/true, "quarter", int64(), &quarter_doc);

I will note though, a struct like this can collapse MakeTemporal and MakeSimpleUnaryTemporal into one function:

template <template <typename...> class Op, typename Duration, typename InType> struct SimpleUnaryTemporal { static Status Exec(KernelContext* ctx, const ExecBatch& batch, Datum* out) { return SimpleUnary<Op<Duration, InType>>(ctx, batch, out); } };

You would then replace OutType with typename... Args in MakeTemporal and you could replace MakeSimpleUnaryTemporal<Foo> with MakeTemporal<Foo, SimpleUnaryTemporal>. Then you wouldn't need two versions of AddDateKernels.

Also note that the function creation is only done once on initialization, so it doesn't really matter whether it's done in the same function or not.

Thanks, I added /*enable_date=*/ to indicate the intention of the boolean parameter.
Regarding the SimpleUnaryTemporal struct, sounds like a nice idea for a future refactor, however if you feel strongly about carrying out these sort of changes in this PR, let me know please!

lidavidm · 2021-09-03T17:23:44Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -588,8 +701,58 @@ inline std::array<int64_t, 3> GetIsoCalendar(int64_t arg, Localizer&& localizer)
          static_cast<int64_t>(weekday(ymd).iso_encoding())};
 }

-template <typename Duration>
+template <typename Duration, typename InType>
 struct ISOCalendar {


We could get around that with a static string somewhere, or by returning const std::string*. Or the actual implementation could be factored out, and the date kernel could call it with a literal string. At the very least, I'm not super enthused about essentially copy-pasting this kernel.

cpp/src/arrow/compute/kernels/scalar_temporal_test.cc

lidavidm · 2021-09-07T12:46:54Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

    class ExecTemplate,
    typename OutType>
-std::shared_ptr<ScalarFunction> MakeTemporal(
+std::shared_ptr<ScalarFunction> MakeTemporal(bool enable_date,


Hmm, it's reasonable. In that case, below, can we do the following, so it's clear what the bare boolean parameter is for?

auto quarter = MakeTemporal<Quarter, TemporalComponentExtract, Int64Type>( /*enable_date=*/true, "quarter", int64(), &quarter_doc);

lidavidm · 2021-09-07T13:31:32Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -588,20 +614,85 @@ inline std::array<int64_t, 3> GetIsoCalendar(int64_t arg, Localizer&& localizer)
          static_cast<int64_t>(weekday(ymd).iso_encoding())};
 }

+template <typename Duration, typename InType>
+struct ISOCalendarWrapper {
+  static Status get(const Scalar& in, std::array<int64_t, 3>& iso_calendar) {


I think this factorization is good, however, just a nit: methods should use UpperCamelCase unless they're const getters so this should be called Get.

Done, thanks @lidavidm @rok !

lidavidm · 2021-09-07T13:34:08Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

    class ExecTemplate,
    typename OutType>
-std::shared_ptr<ScalarFunction> MakeTemporal(
+std::shared_ptr<ScalarFunction> MakeTemporal(bool enable_date,


I will note though, a struct like this can collapse MakeTemporal and MakeSimpleUnaryTemporal into one function:

template <template <typename...> class Op, typename Duration, typename InType> struct SimpleUnaryTemporal { static Status Exec(KernelContext* ctx, const ExecBatch& batch, Datum* out) { return SimpleUnary<Op<Duration, InType>>(ctx, batch, out); } };

You would then replace OutType with typename... Args in MakeTemporal and you could replace MakeSimpleUnaryTemporal<Foo> with MakeTemporal<Foo, SimpleUnaryTemporal>. Then you wouldn't need two versions of AddDateKernels.

Also note that the function creation is only done once on initialization, so it doesn't really matter whether it's done in the same function or not.

lidavidm · 2021-09-07T13:37:50Z

For the failed R test, it looks like it expected year(date) to fail but of course that's no longer the case. Seems this can be changed to expect_dplyr_equal:

arrow/r/tests/testthat/test-dplyr-lubridate.R

Lines 37 to 48 in 2ef0edc

    
           # We can support this feature when ARROW-13138 is resolved 
        
           test_that("date32 objects are not supported", { 
        
             date <- ymd("2017-01-01") 
        
             df <- tibble::tibble(date = date) 
        
             expect_error( 
        
               Table$create(df) %>% 
        
                 mutate(x = year(date)) %>% 
        
                 collect(), 
        
               "Function year has no kernel matching input types" 
        
             ) 
        
           })

lidavidm

Thanks!

One last thing that I forgot earlier, sorry: we should update compute.rst as well: https://github.com/apache/arrow/blob/f40856a768f2c397082da70af034080994587807/docs/source/cpp/compute.rst#temporal-component-extraction

lidavidm · 2021-09-07T16:12:53Z

Oh, it also seems we need to rebase here.

cpp/src/arrow/compute/kernels/scalar_temporal.cc

cpp/src/arrow/compute/kernels/scalar_temporal_test.cc

aucahuasi · 2021-09-07T23:47:59Z

For the failed R test, it looks like it expected year(date) to fail but of course that's no longer the case.
Seems this can be changed to expect_dplyr_equal:

@lidavidm Thank you! I fixed the issue with the R test and added some additional R tests!

aucahuasi · 2021-09-07T23:54:49Z

One last thing that I forgot earlier, sorry: we should update compute.rst as well:

Done!

aucahuasi · 2021-09-08T03:10:44Z

It seems there are some build issues on windows. Tomorrow I'll check that!

lidavidm · 2021-09-08T12:25:09Z

Note the TestNonexistentTimezone test is also failing on Linux.

…functions Reuse most of the iso_calendar algorithm for different types cleaning & minor changes fix and add R tests for year, month, etc functions on date types update cpp docs for temporal compute functions c++ format format all fix windows build fix TestNonexistentTimezone test and clang errors

aucahuasi · 2021-09-08T23:33:19Z

Note the TestNonexistentTimezone test is also failing on Linux.

Thanks, fixed!

rok · 2021-09-09T09:57:27Z

Looks good. Appveyor fail doesn't seem related.

It would probably be good to also add Python tests.

lidavidm

LGTM, thank you.

I filed https://issues.apache.org/jira/browse/ARROW-13957 for the flaky AppVeyor test.

I'm not sure if we need explicit Python tests, maybe comparing against Pandas would be nice, but unlike R we don't have higher level bindings we're trying to test.

pitrou · 2021-09-09T13:02:04Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc


+template <typename Duration, typename InType>
+struct ISOCalendarWrapper {
+  static Status Get(const Scalar& in, std::array<int64_t, 3>& iso_calendar) {


For later: either pass a const reference or a non-const pointer. Non-const references require more care from the caller.

pitrou · 2021-09-09T13:02:26Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

+struct ISOCalendarVisitValueFunction {
+  static Status Get(std::vector<BuilderType*>& field_builders, const ArrayData&,
+                    StructBuilder* struct_builder,
+                    std::function<Status(typename InType::c_type arg)>& visit_value) {


Same here: visit_value should be passed by pointer.

lidavidm · 2021-09-09T13:03:41Z

Thanks @pitrou, I filed ARROW-13960

aucahuasi · 2021-09-09T16:28:23Z

Thanks @lidavidm @rok @pitrou !

…nth, day, etc) from date32/64 types https://issues.apache.org/jira/browse/ARROW-13138 Closes apache#11075 from aucahuasi/temporal-functions-date32 Authored-by: Percy Camilo Triveño Aucahuasi <percy.camilo.ta@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>

github-actions bot added the Component: C++ label Sep 3, 2021

aucahuasi changed the title ~~Implement extract temporal components (year, month, day, etc) from date types~~ Implement extract temporal components (year, month, day, etc) from date32/64 types Sep 3, 2021

lidavidm self-requested a review September 3, 2021 12:37

lidavidm changed the title ~~Implement extract temporal components (year, month, day, etc) from date32/64 types~~ ARROW-13138: [C++] Implement extract temporal components (year, month, day, etc) from date32/64 types Sep 3, 2021

lidavidm reviewed Sep 3, 2021

View reviewed changes

aucahuasi marked this pull request as ready for review September 3, 2021 15:53

lidavidm reviewed Sep 3, 2021

View reviewed changes

lidavidm reviewed Sep 7, 2021

View reviewed changes

rok reviewed Sep 7, 2021

View reviewed changes

cpp/src/arrow/compute/kernels/scalar_temporal.cc Outdated Show resolved Hide resolved

cpp/src/arrow/compute/kernels/scalar_temporal.cc Outdated Show resolved Hide resolved

cpp/src/arrow/compute/kernels/scalar_temporal_test.cc Outdated Show resolved Hide resolved

aucahuasi force-pushed the temporal-functions-date32 branch 2 times, most recently from 8f366d7 to 19b1cf6 Compare September 7, 2021 17:50

aucahuasi changed the title ~~ARROW-13138: [C++] Implement extract temporal components (year, month, day, etc) from date32/64 types~~ ARROW-13138: [C++][R] Implement extract temporal components (year, month, day, etc) from date32/64 types Sep 7, 2021

aucahuasi force-pushed the temporal-functions-date32 branch from 19b1cf6 to a839acd Compare September 7, 2021 23:45

github-actions bot added the Component: R label Sep 7, 2021

aucahuasi force-pushed the temporal-functions-date32 branch from a839acd to f7c9e3a Compare September 7, 2021 23:54

aucahuasi force-pushed the temporal-functions-date32 branch 2 times, most recently from 130ea03 to 41d6beb Compare September 8, 2021 03:09

aucahuasi force-pushed the temporal-functions-date32 branch from 41d6beb to 49ee568 Compare September 8, 2021 22:18

aucahuasi force-pushed the temporal-functions-date32 branch from 49ee568 to a47b78a Compare September 8, 2021 23:22

lidavidm approved these changes Sep 9, 2021

View reviewed changes

lidavidm closed this in 4b5ed4e Sep 9, 2021

pitrou reviewed Sep 9, 2021

View reviewed changes

This was referenced Sep 9, 2021

[C++] Implement kernel to extract datetime components (year, month, day, etc) from date type objects #28839

Closed

[C++] Implement strftime on date32/64 types #29529

Closed

[C++] Fix use of non-const references in temporal kernels #29566

Closed

ARROW-13138: [C++][R] Implement extract temporal components (year, month, day, etc) from date32/64 types #11075

ARROW-13138: [C++][R] Implement extract temporal components (year, month, day, etc) from date32/64 types #11075

Uh oh!

Conversation

aucahuasi commented Sep 3, 2021

Uh oh!

github-actions bot commented Sep 3, 2021

Uh oh!

github-actions bot commented Sep 3, 2021

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aucahuasi Sep 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aucahuasi Sep 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Sep 7, 2021

Uh oh!

lidavidm left a comment

aucahuasi Sep 7, 2021 •

edited

Loading

aucahuasi Sep 7, 2021 •

edited

Loading

aucahuasi commented Sep 8, 2021 •

edited

Loading