deb/rpm: build granular packages

[Abrosimov-a-a]

1. Big package size.
2. Librarys duplication on client system.
3. Need to upgrade package, when dependency is upgraded.
4. The patches of the distribution developers are not used. As example: setupterm: could not find terminal #2768
5. Perhaps you build the package by hand?
6. You can find many other information about that issue.

What is your current way to build packages?
What about the system for automatically assembling the correct packages? I can help you with that.

Roadmap:

• 1.1. Determine the initial status.
• 1.2. Build all dependencies in auto mode.
• 1.3. Build the primary package: python3-dvc
• 1.4. Build all extra dependencies in auto mode.
• 1.5. Issue dvc/dagascii: Use pager instead of AsciiCanvas._do_draw #2815 is merged.
• 1.6. Now new package system is ready for use.
• 2.1. Make debian/ for all packages.
• 2.2. Make pull requests to the upstreams.
• 2.3. Build doc packages.
• 2.4. Passing Debian lint checks.
• 2.5. Now DVC is ready for Debian Experimental.

Current status:

Already in Debian repository:

• python3-ply
• python3-future
• python3-colorama
• python3-configobj
• python3-setuptools
• python3-schema
• python3-requests
• python3-distro
• python3-appdirs
• python3-inflect
• python3-humanize
• python3-pathspec
• python3-shortuuid
• python3-tqdm
• python3-packaging
• python3-yaml
• python3-ruamel.yaml
• python3-git
• python3-networkx
• python3-flufl.lock
• python3-voluptuous

Not in Debian:

• configparser:
  • https://github.com/jaraco/configparser/
  • Do you use this package instead of the built-in?
• nanotime:
  • https://github.com/jbenet/nanotime/tree/master/python
  • Perhaps abandoned.
• jsonpath-ng
  • https://github.com/h2non/jsonpath-ng
  • Perhaps abandoned.
  • Debian dependencies:
    • python3-ply
    • python3-six
    • python3-decorator
• grandalf
  • https://github.com/bdcht/grandalf
  • Debian dependencies:
    • python3-pyparsing
    • python3-future
• treelib
  • https://github.com/caesar0301/treelib
• funcy
  • https://github.com/Suor/funcy
• asciimatics (Thrown out of code!)
  • https://github.com/peterbrittain/asciimatics
  • Debian dependencies:
    • python3-pyfiglet
    • python3-future
    • python3-wcwidth
    • python3-pillow

[efiop]

Hi @Abrosimov-a-a ! Great question! We package it like that for simplicity, as we don't have to deal with packaging all our dependencies into deb and rpm packages. The way we build current deb and rpm packages is fully reflected by https://github.com/iterative/dvc/blob/master/scripts/build_posix.sh . We build binaries by pyinstaller and then package them into deb/rpm using fpm tool. Then we publish them on our s3 bucket using https://github.com/iterative/dvc-s3-repo . Turning these packages into proper python-based, would require packaging all our dependencies into deb/rpm packages first, which is tiresome, but could be simplified a lot by using that fpm tool, which supports converting python packages into deb/rpm. If you feel like it, we would very appreciate you taking a look 🙂

[Abrosimov-a-a]

Hi @efiop !
I will see how it is more convenient to organize dependencies and building. Now I have found 5 dependencies with the problems. It's not so much.

What do you think about SCons?

[efiop]

@Abrosimov-a-a Yep, 5 deps doesn't sound too bad, indeed 🙂

What do you think about SCons?

First time I hear about it, tbh. Could you elaborate please?

Also worth mentioning that we are about to have a snap package #2778 thanks to @casperdcl 🙏 , but it does seem to suffer from similar limitations, as it runs in an isolated container.

[casperdcl]

@efiop SCons is a pythonic replacement for CMake.

[Abrosimov-a-a]

Next questions:

1. There are two different ways. Simple (for your own repository). Hard (for Debian repository). What the way you need?
2. What we gona do with dependencyes? Include to DVC package or build another packages?

[efiop]

@Abrosimov-a-a

There are two different ways. Simple (for your own repository). Hard (for Debian repository). What the way you need?

Creating missing packages for our dependencies and placing them on our s3 deb/rpm repo should be good enough.

What we gona do with dependencyes? Include to DVC package or build another packages?

I guess we need to package each dependency, to set this up correctly, right?

[Abrosimov-a-a]

@efiop

I guess we need to package each dependency, to set this up correctly, right?

Yes. It's more preferrable.

[casperdcl]

@efiop @shcheklein this is one reason people create *.deb python packages. apt-get install python-dvc could depend on libffi-dev (or libffi6) for example.

[efiop]

@Abrosimov-a-a Please let us know if you have any questions or need any help 🙂

@casperdcl Good point!