REF: BlockManager.apply_allow_failures

[jbrockmendel]

cc @mroeschke w/r/t rolling ._apply, @WillAyd w/r/t _cython_agg_blocks, and anyone else w/r/t DataFrame._reducewith numeric_only=None, apply.FrameApply with ignore_failures=True.

All of these do something along the lines of:

results = []
exclude = []
for i, block in enumerate(mgr.blocks):
    try:
          res = func(block.values)
          results.append(res)
    except:
          exclude.append(i)

out = reconstruct(results, exclude)

Two things should be done with this pattern:

1. Deprecate it, since it is a disproportionate maintainence burden
2. implement something like BlockManager.apply_with_ignore_failures that would resemble BlockManager.apply but with the try/except logic*

* We could add that logic into BlockManager.apply, but BM.apply assumes the output has the same shape as the original frame, which I don't think holds for the cases mentioned above, so that would be a little more invasive than just adding a try/except.

[WillAyd]

I don’t know about deprecating. I think I tried to remove this pattern from GroupBy in the past and it killed performance. Would be on board with consolidating for sure though

…

Sent from my iPhone

On Jun 11, 2020, at 10:44 AM, jbrockmendel ***@***.***> wrote:  cc @mroeschke w/r/t rolling ._apply, @WillAyd w/r/t _cython_agg_blocks, and anyone else w/r/t DataFrame._reducewith numeric_only=None, apply.FrameApply with ignore_failures=True. All of these do something along the lines of: results = [] exclude = [] for i, block in enumerate(mgr.blocks): try: res = func(block.values) results.append(res) except: exclude.append(i) out = reconstruct(results, exclude) Two things should be done with this pattern: Deprecate it, since it is a disproportionate maintainence burden implement something like BlockManager.apply_with_ignore_failures that would resemble BlockManager.apply but with the try/except logic* * We could add that logic into BlockManager.apply, but BM.apply assumes the output has the same shape as the original frame, which I don't think holds for the cases mentioned above, so that would be a little more invasive than just adding a try/except. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mroeschke]

FWIW, if we're open to allowing numba in the internals (with some opt-in, global execution engine setting), this is a great candidate for numba parallelism and can ameliorate the performance hit if we decide to deprecate this behavior.