source: Switch to xz for packages and tools where possible

[diizzyy]

Adds a slightly higher compression level to xz by default which roughly raises memory usage from 100MiB to about 200MiB during compression, about 10MiB for decompression. (Source: xz manpage)
Changes git packages to xz
Removes mirror checksums to avoid mismatches with tarballs as they differ due to checkout date and time of files.
Changes a few source tarballs to xz if available upstream.

Signed-off-by: Daniel Engberg daniel.engberg.lists@pyret.net

[diizzyy]

Not sure if PKG_RELEASE should be bumped or not.

[ignisf]

openwrt/packages#149

Incrementing PKG_RELEASE has been a bit hazy to me, too.

Has the PKG_RELEASE convention been documented anywhere in the wiki? I've forgotten to bump the PKG_RELEASE a few times and my patches have been merged anyways.

Is the linked convention adopted by LEDE?

If it is, is it OK if I added it to https://wiki.openwrt.org/doc/devel/packages?

[neocturne]

Generally, PKG_RELEASE should be incremented when the resulting binary package will change (either the contained files or the metadata like dependencies), so opkg will notice the update.

For build fixes or changing to another equivalent source URL or tarball (like here) no increment should be done, as there is nothing for opkg to update.

Other comments:

• Switching downloaded tarballs to .xz seems reasonable to me
• The tarball for the package xz should be kept as bz2, so xz can be built when it is not available on the host system (bzip2 is checked as a prerequisite of LEDE, xz isn't; adding xz to the prereqs might be an alternative)

How much smaller do the tarballs get with -7e? The effect will only be noticable for huge tarballs, but in my experience it's not worth the increased time needed for the compression of big files.

[diizzyy]

I think it's resonable to expect xz support these days, I did try to find how prereqs were generated but failed, if someone can give me pointers I'll add that too.

Most base packages see a slightly smaller size (it's usually within 1-2KiB) however it's more beneficial for external packages which can be quite a bit larger and I don't see much of a difference in CPU time for base packages given their overall size. @jow- is also planning to make git packages reproducible (for mirroring with checksums) and if we can save space and bandwidth I think it's worth the slight overhead it may inflict.

[diizzyy]

I've added xz-utils (xz) to make prereq

[blogic]

there is a commented out line here

[diizzyy]

Couldn't find any documentation or figure out what it actually did, there are more occurrences of stray lines in package Makefiles so I'd rather do a tree-wide cleanup afterwards.

[blogic]

remove the whole line if it is not needed and explain this change in the commit message. this is unrealated to changing the compression to xz

[diizzyy]

It's an issue because git doesn't date and time-stamp files during cloning so each package will be different unless you clone at the exact same time and at the exact same speed. As someone who doesn't have access to the "fallback" download server it's impossible/doesn't make any sense to add a checksum for a package that's more than likely mismatch compared to what's going to be uploaded if any at all. There are more Makefiles in the tree which have empty PKG_MIRROR_MD5SUM vars so I figured it would be more consistent to leave those empty rather than removing them.

[blogic]

why not delete the line ?

[diizzyy]

See first comment

[blogic]

remove it if it not needed

[diizzyy]

See second comment

[diizzyy]

Due to my silly mistake not using branches makes this much harder to fix. Continued (and fixed) in #375