feat: let pylance use sub-level logger of logging

[yanghua]

Closes #3195

[yanghua]

cc @westonpace @wjones127 Please have a look, thanks.

[wjones127]

This seems alright. Though most of our logging happens at the Rust layer. That does make me wonder if it is even that valuable to let users tweak the logger, given it only controls a small fraction of our logs.

I made a suggestion on how they could share configuration. Otherwise, I think this seems fine.

[wjones127]

We already have a logging level for pylance defined as LANCE_LOG

https://github.com/lancedb/lance/blob/f21397d47a1c6b3998e6d3dc50a858262ec17960/python/src/lib.rs#L106-L109

However, the syntax is a bit more complex:

LANCE_LOG=[target][=][level][,...]

For example:

LANCE_LOG=lance-core=debug,info

Means provide lance-core logs at debug level, everything else at info level.

I think it might be preferable to re-use and parse that environment variable.

Something like:

def get_log_level():
    lance_log = os.environ.get("LANCE_LOG", "INFO").upper()
    lance_log = [entry for entry in lance_log.split(',')
                 if '=' not in entry]
    if len(lance_log) > 0:
        return lance_log[0]
    else:
        return "INFO"

[yanghua]

Thanks for your explanation. Reasonable. Accept.

By the way, it seems there is no description about the env LANCE_LOG in the whole codebase(not sure about documentation in the official site).

If you agree, we should document it for users. WDYT?

[wjones127]

Nice work. And agreed, we should add some documentation about the logging settings.