[Variant] Align cast logic for `variant_get` to cast kernel

[klion26]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, we only support safe cast, but no lossy cast(e.g., 123.34(Decimal) -> 123(Int)), we need to support this after #8086 and #8873

Describe the solution you'd like

Add a lossy cast for all data types in variant_get, we may need to do each data type separately.

Describe alternatives you've considered

Additional context

[klion26]

As cast kernel makes lossy=true the default from its introduction(without the flag), so we'll align the behavior to cast kernel here, this one can be used to track this one

[klion26]

@alamb @scovich
The plan is to handle the cast logic within every Variant::as_xx function. Do you think this is the right direction? If so, I'll move forward with it.

The reasons for implementing the cast logic in Variant::as_xx are as follows:

1. Currently, arrow cast kernels are column-based, while the casts in variant_get are row-based; we can't directly use the cast kernel; we can only reuse the logic in the cast kernel as much as possible.
2. This requires maintaining the cast logic in two places, but the basic cast logic remains changed little over time, so it should be manageable.

[scovich]

Ultimately, the columnar casts have to cast each individual value, correct? There's just a (really) big difference in the wiring harness around that scalar casting logic? But from what I've seen, the scalar casting logic is embedded in the various columnar kernels, so unifying them would require factoring out a common scalar cast capability?

[klion26]

@scovich, sorry for the late reply; this notification mail is located in a different folder this time.

so unifying them would require factoring out a common scalar cast capability?

Yes, we can unify the common cast logic for most of them, but some may not be easy to unify; we'll unify/reuse the common logic as much as possible.

• like Int32 -> Date32 cast kernel use (Int32, Date32) => cast_reinterpret_arrays::<Int32Type, Date32Type>(array), but for variant, current is Variant::as_naive_date (will return NaiveDate) and will do Date32Type::from_naive_date(x) when builder.append(value)
• like Timestamp -> Timestamp, it may not be easy to unify without performance sacrifice(did not find a solution, but need to confirm it), the logic is as follows

  arrow-rs/arrow-cast/src/cast/mod.rs

  Lines 1782 to 1803 in 92a239a

  	(Timestamp(from_unit, from_tz), Timestamp(to_unit, to_tz)) => {
  	let array = cast_with_options(array, &Int64, cast_options)?;
  	let time_array = array.as_primitive::<Int64Type>();
  	let from_size = time_unit_multiple(from_unit);
  	let to_size = time_unit_multiple(to_unit);
  	// we either divide or multiply, depending on size of each unit
  	// units are never the same when the types are the same
  	let converted = match from_size.cmp(&to_size) {
  	Ordering::Greater => {
  	let divisor = from_size / to_size;
  	time_array.unary::<_, Int64Type>(|o| o / divisor)
  	}
  	Ordering::Equal => time_array.clone(),
  	Ordering::Less => {
  	let mul = to_size / from_size;
  	if cast_options.safe {
  	time_array.unary_opt::<_, Int64Type>(|o| o.checked_mul(mul))
  	} else {
  	time_array.try_unary::<_, Int64Type, _>(|o| o.mul_checked(mul))?
  	}
  	}
  	};

Anyway, seems we need to do the cast logic in Variant::as_xx, will push one pr to explain it better

[sdf-jkl]

@klion26 What's next? 😃

[klion26]

@sdf-jkl Thanks for tagging me, #9564 has been merged, I'll do this for the remaining types

[AdamGS]

I've been looking into that and realized the the variant-specific casts don't allow truncating timestamps while arrow does allow some, I couldn't find any explicit reason why, and would be happy to take on it if that's the desired final behavior here.

My motivation here is that I realized that we currently just pull typed_value only if the types match exactly, but theoretically we could cast it (when supported and according to CastOptions), which should be much faster than going through the row-wise builder.