Switch to python 3.8

[rwols]

Before we can use asyncio in all of its glory, we need to switch to the Python 3.8 runtime. To do that, we have to wait for a few things:

• Package Control must implement an upgrade path for existing "dependencies"
• The backrefs dependency must be upgraded to 3.8
• The dependency bracex must be upgraded to 3.8
• The dependency markupsafe must be upgraded to 3.8
• The dependency mdpopups must be upgraded to 3.8
• The dependency pygments must be upgraded to 3.8
• The dependency pymdownx must be upgraded to 3.8
• The dependency python-jinja2 must be upgraded to 3.8
• The dependency python-markdown must be upgraded to 3.8
• The dependency pyyaml must be upgraded to 3.8
• The dependency wcmatch must be upgraded to 3.8
• sublime_lib must be upgraded to 3.8
• lsp_utils must be upgraded to 3.8
• have a pull request ready for switching to 3.8 for the LSP package
• announce deadline for switching all helper packages

As far as I understand, this is also going to be a hard backwards-incompatible break. So we have to announce it.

[rwols]

Package Control WIP branch is at https://github.com/wbond/package_control/tree/four-point-oh

[rchl]

Realistically also lsp_utils and sublime_lib would have to be updated or most of the LSP-* packages won't work.

I only really use ResourcePath from sublime_lib so I could vendor it but it will probably be updated anyway since it's from an active ST member.

[predragnikolic]

I think the time for Python 3.8 if finally here :)

Just a small note,
the list of deps that LSP currently uses is this:

            "bracex",
            "mdpopups",
            "pathlib",
            "wcmatch",

I did a quick experiment.

I have only LSP and LSP-pyright setup.
I created a .python-version file at the root of LSP and LSP-pyright.
The file just contains 3.8.

Once I did that, I expected that I need to change something in lsp_utils,
but I didn't. I restarted ST, both LSP and LSP-pyright were running in 3.8, which is great.

I don't know why I didn't need to change anything in lsp_utils...
it looked like PC4 already added lsp_utils to Lib/python38 with the rest of the required LSP dependencies:

~/.config/sublime-text/Lib/python38  ls
bracex                    lsp_utils-3.1.0.dist-info  sublime_lib-1.5.2.dist-info
bracex-2.1.1.dist-info    mdpopups                   wcmatch
coverage                  mdpopups-4.2.2.dist-info   wcmatch-1.2.1.dist-info
coverage-7.4.1.dist-info  package_control.py
lsp_utils                 sublime_lib

[jwortmann]

I don't really know how Package Control works, but I can see two unmerged pull requests sublimehq/package_control_channel#8713 and sublimehq/packagecontrol.io#157 for the channel and for the website, whose descriptions indicate that they are required to install dependencies/libraries on the 3.8 environment:

2. opt-in compatible libraries to python 3.8

Why channel_v4.json

Package Control 4.0 handles both 3.0.0 and 4.0.0 schemes well. Just requires 4.0.0 to pull dependencies/libraries for python 3.8.

Notably, the pathlib dependency seems to be still restricted to 3.3 in the package_control_channel PR (note that it's also not listed in your ls output):

So, as far as I understand, the linked PR needs to be merged first, and even after that we can't upgrade to Python 3.8 unless we drop the pathlib dependency. Is that correct?

Edit: Ah, pathlib is of course part of the Python 3.8 stdlib, so we can just remove it from the dependencies and import from the stdlib directly in the code. I guess then it makes sense why the dependency is only for Python 3.3 😄

---

Also as far as I understand, all LSP-* helper packages will need a .python-version file with 3.8 as well to keep working. So we need to update them all at the same time with a new LSP 2.0.0 (Python 3.8) release. I think this occasion could potentially be used in case we want to do some other breaking API changes. If we want to do something like that, maybe we should start another thread/issue to collect and discuss such possible API changes, before pushing for a 3.8 update. Let me start here with a first example:

• Remove the SessionViewProtocol and SessionBufferProtocol classes - as far as I can tell, they don't have any purpose, and in my opinion they make working with parts of the code base very annoying because "Goto definition" for methods from the SessionView and SessionBuffer classes doesn't work (it only opens the empty method stubs from the Protocol classes). Also I have never seen something like this where an interface/protocol is introduced for something that will ever only have a single class which uses the interface. So I'd vote to remove those two Protocol classes.

[rchl]

• Remove the SessionViewProtocol and SessionBufferProtocol classes - as far as I can tell, they don't have any purpose, and in my opinion they make working with parts of the code base very annoying because "Goto definition" for methods from the SessionView and SessionBuffer classes doesn't work (it only opens the empty method stubs from the Protocol classes). Also I have never seen something like this where an interface/protocol is introduced for something that will ever only have a single class which uses the interface. So I'd vote to remove those two Protocol classes.

I believe the purpose is to clearly define what is a public interface and what is internal. Especially for the plugin packages accessing those. I'm not sure how well it's working in practice... I suppose there are cases already where the plugins just override the type and access things anyway.

[jwortmann]

I believe the purpose is to clearly define what is a public interface and what is internal. Especially for the plugin packages accessing those. I'm not sure how well it's working in practice... I suppose there are cases already where the plugins just override the type and access things anyway.

Besides the fact that there is no way in Python to enforce this (those are only type annotations in the plugin class methods anyway) and as you wrote plugins can easily workaround it, Python already has a convention to mark private methods with an underscore. So in my opinion those protocol classes have zero advantages and only make working with the code harder if you use tools like Pyright which use the type annotations as source for "Goto Definition". And around half of the methods from the protocol classes are not meant to be used by plugins anyway and instead were just added there to satify the type checker. Furthermore, there are other classes exposed in the plugin API which apparently also work fine without having a protocol class, e.g. Session without a SessionProtocol etc.

I can also see no indication in https://peps.python.org/pep-0544/ that the intention of typing.Protocol would be to hide internal methods, so I still think this is a misuse or at least bad practice here.

[rchl]

I get your point. It seems a bit excessive given that we have only one implementation for each of those protocols.

That said, regardless if misuse or not, it does make things easier for plugin developers to limit the stuff that they can see (type-wise). For example SessionView has a lot of methods that the plugins should not touch (like on_before_remove, on_capability_added_async and other) but at the same time those can't be marked as private as we want to treat those as public within the LSP code base.

I do think removing those would simplify things within the LSP code base which would be nice. But do we want to accept the before-mentioned drawback? Probably the ideal way would be to have more restricted public Session interface that would then return those more restricted interfaces and those would only be used by external plugins, without affecting internal code.

[rwols]

From what I remember, the reason the two protocol classes SessionViewProtocol and SessionBufferProtocol exist is because they avoid a circular import problem. If the code in sessions.py imported one of the two concrete implementation classes then that would result in a circular import. So, the initial reason for them existing was a technical reason.

Maybe that circular import problem doesn't exist anymore, in which case it's totally fine to remove them if possible (from a technical standpoint).