re-support python3.6 on the setup

[isidentical]

Resolves #445

[isidentical]

Apparently newer versions of moto either doesn't preserve the version order, or use a reversed order. I couldn't find anything related this on the spec (if I am missing, please correct me) but it seems list-object-versions API seems like having no promises on the order. I've converted lists to sets in the tests and adjusted the cases to work with orderless versions.

[martindurant]

Ok, looks good - thank you.

[martindurant]

Can you also add py3.9 to the CI? Seems like a good time to do it.

[isidentical]

@martindurant done!

[isidentical]

@martindurant, when you are available could you make a new release (so that we can fetch it from pypi with that version)?

[efiop]

@martindurant Would it be possible to create a new release, please? 🙏

[martindurant]

On my list

[efiop]

@martindurant Sorry for the noise and thank you! 🙏

Btw, we've noticed that this (and other projects) don't have github actions to automate releases, so I suppose you are doing all of them by hand, right? Is there a particular reason for that? Would you be open for a PR from us to automate that? You'll need to only add your pypi token to repo secrets.

If you are interested in such gha, which release workflow would you prefer?

1. Automatically release tags when they are created
2. create a release through github releases (we use it in dvc, but it is mostly for historic reasons)
3. something else?

Also, would you be interested in (automatic?) pre-releases, similar to moto: https://pypi.org/project/moto/#history They don't show up as regular pypi releases, requiring a special flag to install, but IIRC they could be explicitly used in requirements, which is nice for projects that use s3fs, as pypi doesn't like dependencies directly to github.

[martindurant]

There was a workflow to do the release in this or one of the other repos, but wasn't working when I experimented with it.

I would use option 1, which is essentially what I do now. Building locally is pretty painless. I have never seen the point of github releases, since it's pypi that matters.

In my circle, I have not come across anyone using pypi pre-releases, people seem to be happy to point to github when necessary. Is there a downside?

[efiop]

I would use option 1, which is essentially what I do now. Building locally is pretty painless. I have never seen the point of github releases, since it's pypi that matters.

Agreed, but there cases like when you are also building other package types or artifacts (e.g. we build dvc with pyinstaller as well) and want to upload them to github releases.

If you are happy with manual builds and publishing, we are also okay with it, since you've been very helpful, responsive and quick with all of the releases that we've requested so far, we really appreciate that 🙏 Just wondering if there is anything we could do to make the release workflow easier for you. So please do let us know 🙂

In my circle, I have not come across anyone using pypi pre-releases, people seem to be happy to point to github when necessary. Is there a downside?

Yeah, you won't be able to publish your package to pypi if it has a direct dependency (pypi will literally complain about it and refuse to publish), which is a major problem for us.

[martindurant]

Perhaps the best thing that you can do for me is to run your test suite against fsspec/s3fs/gcsfs master/main (yes, two have changed, one has not) as well as released versions. Many projects have a "test-dev" or "upstream" environment for this purpose.

hat do you think of the versions-related points here ?

[efiop]

Perhaps the best thing that you can do for me is to run your test suite against fsspec/s3fs/gcsfs master/main (yes, two have changed, one has not) as well as released versions. Many projects have a "test-dev" or "upstream" environment for this purpose.

Do you mean something like daily cron tests with upstream fsspec/s3fs/gcsfs(and probably adlfs)? Not sure how much value there is for you in that, as our tests are fairly limited, we only use a subset of filesystem functionality (at least right now), but we could indeed at least keep an eye on that in dvc. Unless you are talking about fsspec API stability, kinda like integration tests of sorts, then yeah, that makes sense. Especially once we migrate DvcFileSystem that we have to fsspec. Please let me know if I'm missing some point here.

hat do you think of the versions-related points here ?

I can share my opinion, but I wonder what @isidentical thinks about these, since he has been dealing with it much more.

move to calendar versioning, since the "semantics" of x.y.z are often confusing for a dev-facing library like this

I agree, this would prevent people from assuming that fsspec has semantic versioning.

always release fsspec, s3fs and gcsfs together

👍 that's the problem we've also ran into with 0.9 release

explicitly pin s3fs and gcsfs's version to the co-released fsspec. That means more work for me.

That would be great and hopefully, the burden of pinning that will reduce over time as fsspec becomes more and more stable. Maybe even dependabot could help with constantly keeping an eye on fsspec releases.

[martindurant]

The more external projects running some portion of fsspec regularly, the better :)

[isidentical]

hat do you think of the versions-related points here ?

The points are great, one concern though

explicitly pin s3fs and gcsfs's version to the co-released fsspec

I assume more and more implementations (adlfs etc) will start to pin their fsspec version with the calver transition. One side effect of this would be for the projects who need to have many different implementations installed at the same time with one fsspec version (e.g dvc installed with dvc[s3, gs, azure] or dvc[all]). This will certainly make things a bit harder (and at the same time a bit safe) for us, since we'd end up with waiting all projects to migrate to certain version of fsspec.

I think overall, it will be a nice change.

[martindurant]

@hayesgb @TomAugspurger , what do you think of the prospect of either releasing adlfs every single time fsspec does, and pinning to that fsspec version, or testing some minversion/maxversion scheme in your CI so that you know, at your release time, what version bounds to use?

[TomAugspurger]

I'd prefer to avoid needing to tie adlfs tightly to fsspec's release schedule. Ideally we would avoid the need for that in the first place.

adlfs should be testing against more versions of fsspec though. At the very least it should test the minimum required, latest, and development version of fsspec. I'll get to that today.

[efiop]

@martindurant Btw, what are the reasons for avoiding semantic versioning for fsspec? Seems like that would work best for the ecosystem of fsspec-based implementations.

[martindurant]

what are the reasons for avoiding semantic versioning

This was the direction chosen by dask-related repos recently. We found that it was likely that, as new features were added, something in the API could always be considered "braking", even as other bugs got fixed for each release.

For instance, ReferenceFileSystem has undergone rapid change recently, but is still experimental. It will surely have braking changes in every single release, even if the main user API is stable. Indeed, internal implementation details might change as we fix and improve things, so that libraries like s3fs see braking changes (i.e., they need updating) even when end users don't notice.