api: let open() return the file object (like builtin open)

[jorgeorpinel]

Sort of extracted from #3426 (review)

Should dvc.api.open() behave more like builtin open() in that it just returns the file object? (Probably an explicit close() would be needed though). But keeping also the existing possibility of being used as a context manager (with keyboard)?

CC @shcheklein please add any further details if needed, thanks!

[shcheklein]

@Suor what are your thoughts on this? Is it possible to imitate open() at all or we have to go with context manager or regular object return value?

[Suor]

The reason for using context manager is to manage resources, i.e. remove temporary clone dirs after use. Now that we use caching those, this only applies to external_repo(..., for_write=True) calls, which dvcx uses. Everything is cleaned up in then end of dvc run with dvc.external_repo.clean_repos().

I should note that clean_repos() is not safe in the sense that it does not check whether any repo is still in use. This is not an issue since it happens just before dvc exits, however, if people start using api calls, including .open() then at some point we might want to call or let people call clean_repos() and update that to only remove dirs not used anymore, i.e. make it safe.

If we go away from using context managers then we can't really make clean_repos() safe because we don't know what is used.

[shcheklein]

It's also about closing file descriptors, right?

Can we make our own fd.close() that also updates external_repo object so that clean_repos can be run safely?

we should still find a way to cache repos in memory btw, and not depend of fs and git clone - might it change the approach to all these cached repos?