progress: limit maximum running bars

[casperdcl]

• introduce Tqdm.MAX_BARS = 20 for parallel display
  • add ... (more hidden) ... message for overflow bars
• auto-detect screen height? Should be done in upstream add nrows tqdm/tqdm#918
• use tqdm>=4.44.0 which auto-detects screen height with a fallback of 20
• TODO (future): combine sub bars into a single high-level bar
• fixes pull: going completely crazy on large dataset #2905
• Replaces/closes [WIP] pull: display a lot of jobs #3453
• related progress: inconsistent nested format when no main bar #3452

issue & fix:

(from #3453 (cc @JIoJIaJIu): script for reproducing issue & fix)

[casperdcl]

ideally auto-detect; see tqdm/tqdm@80be780:tqdm/utils.py#L263-L330 for inspiration

[efiop]

@casperdcl Could you show before/after capture, please?

[efiop]

@casperdcl Also, wouldn't it make sense to patch tqdm itself to tidy up unused bars as it goes?

[casperdcl]

Could you show before/after capture, please?

The script and asciinema recordings from @JIoJIaJIu (see description at the top here) still apply. I can re-generate them but they'd look identical.

Also, wouldn't it make sense to patch tqdm itself to tidy up unused bars as it goes?

This isn't about tidying up unused bars. It's about hiding running bars.

Specifically:

• completed bars have never been an issue - they just print normally upon completion (assuming leave==True)
• running disable=False bars could scroll off screen if there are more than SCREEN_HEIGHT of them running. This PR dynamically hides them.

By "dynamically" I mean:

• completed leave=True bars would still print normally only after completion
• when a displaying bar completes it may be replaced by one which was hidden

[casperdcl]

The script and asciinema recordings from @JIoJIaJIu (see description at the top here) still apply. I can re-generate them but they'd look identical.

Actually not true. Updated:

[efiop]

@casperdcl By unused bars i meant blank lines, that make current pbars look like they are flickering in a mess. I like how the new approach looks like 👍 But maybe this should be done in tqdm itself, after all? Seems like a feature everyone could benefit from. To be honest, I'm not too deep into the implementation details here, so I'm simply judging by the looks.

[gurobokum]

Actually not true. Updated

But you launched with --tqdm flag that means you use library tqdm version.

I tried and noticed that it works! 🎉 And it look nicer from the code side
What interesting when I was working on the issue it was my first approach - usage display and clean methods, but it didn't help, but now I cannot reproduce it and everything works

One disadvantage - that it flicks too much. I think cause all pbars going up (pos -=1) on removing pbar. It would be better if we can just put hidden bar in the place of the closed one

[casperdcl]

But you launched with --tqdm flag that means you use library tqdm version.

Yes but at the end of the animation it's run again without the flag

[casperdcl]

It would be better if we can just put hidden bar in the place of the closed one

Interesting idea. I prefer moving up logically as newer bars at the bottom will be less complete and higher bars will be more complete - e.g. assuming similar file sizes:

bar10: #################
bar12: ###########
bar13: #######
bar14: ####

Rather than more randomly ordered:

bar14: ####
bar10: #################
bar13: #######
bar12: ###########

Maybe for a different issue/PR?

[casperdcl]

Upstream issue for fixing this properly: tqdm/tqdm#918

[casperdcl]

updated with a ... (more hidden) ... message:

Think this is mergeable now as-is.

[efiop]

For the record: had a discussion about it last Sat and decided to wait for this functionality to become available in the upstream tqdm.

[casperdcl]

... aaand new version!