diff does not report updated packages

[nscheer]

When comparing images using diff, there seem to be a lot of packages missing, specifically those from a "yum upgrade" run.

Expected behavior

I'd expect to see the packages that got upgraded during "yum upgrade":

[...]
================================================================================
 Package                Arch       Version                    Repository   Size
================================================================================
Updating:
 bind-license           noarch     32:9.9.4-73.el7_6          updates      87 k
 device-mapper          x86_64     7:1.02.149-10.el7_6.3      updates     292 k
 device-mapper-libs     x86_64     7:1.02.149-10.el7_6.3      updates     320 k
 glibc                  x86_64     2.17-260.el7_6.3           updates     3.7 M
 glibc-common           x86_64     2.17-260.el7_6.3           updates      12 M
 krb5-libs              x86_64     1.15.1-37.el7_6            updates     803 k
 nss                    x86_64     3.36.0-7.1.el7_6           updates     835 k
 nss-sysinit            x86_64     3.36.0-7.1.el7_6           updates      62 k
 nss-tools              x86_64     3.36.0-7.1.el7_6           updates     515 k
 nss-util               x86_64     3.36.0-1.1.el7_6           updates      78 k
 openldap               x86_64     2.4.44-21.el7_6            updates     356 k
 systemd                x86_64     219-62.el7_6.3             updates     5.1 M
 systemd-libs           x86_64     219-62.el7_6.3             updates     406 k
 tzdata                 noarch     2018i-1.el7                updates     490 k

Transaction Summary
================================================================================
Upgrade  14 Packages
[...]

Actual behavior

Only package tzdata shows up:

-----RPM-----

Packages found only in centos:7: None

Packages found only in centos:7-upgraded:
NAME               VERSION         SIZE
-gpg-pubkey        f4a80eb5        0

Version differences:
PACKAGE        IMAGE1 (centos:7)        IMAGE2 (centos:7-upgraded)
-tzdata        2018g, 1.9M              2018i, 1.9M

Information

• container-diff version: 0.14.0
• Operating system: CentOS Linux release 7.6.1810 (Core)
• Docker version: docker-ce 18.09.2

Steps to reproduce the behavior

1. Fetch CentOS image

[nscheer@docker ~]$ docker pull centos:7
7: Pulling from library/centos
Digest: sha256:184e5f35598e333bfa7de10d8fb1cebb5ee4df5bc0f970bf2b1e7c7345136426
Status: Image is up to date for centos:7

2. Build an upgraded version

docker build -t centos:7-upgraded -f - . <<EOF
FROM centos:7
RUN yum makecache fast && yum -y upgrade && yum clean all
EOF

The upgrade process shows and upgrades the aformentioned packages.

3. Run diff

./container-diff-linux-amd64 diff --type=rpm daemon://centos:7 daemon://centos:7-upgraded

4. Diff results in the output shown in "Actual behavior"

[nkubala]

hey @scopev24, thanks for the issue. container-diff does OS package analysis for packages installed with apt only; it doesn't know anything about yum, or specifically, the locations where yum installs packages in the filesystem. it's not surprisingly to me that you're not seeing correct results here.

would you be interested in contributing a yum differ to container-diff?

[nscheer]

Hi,

sorry for the delay.
I don't quite get it... You say, that contaner-diff does package analysis only for apt - but what about the rpm differ? That's what I'm talking about here.

I wrote about yum because that's what you typically use.
Of course you can use rpm as well.

So let me rephrase the issue I think I'm seeing:

If you just run a bash in the aforementioned images, and the run

rpm -qa --qf '%{NAME} %{VERSION}-%{RELEASE}.%{ARCH}\n'

in both - you'll get a list of packages. Just save that as a text file and diff it using your favorite tool.
You'll see a number of packages that changed their version. These packages are missing from the container-diff run.

So, since container-diff has a rpm differ, I'd expect it to pickup all changed packages.

I implemented my own simple rpm differ, which does just that, run the rpm list command mentioned above and diffs the two resulting lists.

Here's the output of container-diff when diffing centos:7.6.1810 and centos:7.5.1804:

-----RPM-----

Packages found only in centos:7.6.1810:
NAME                 VERSION        SIZE
-json-c              0.11           64.1K
-libsmartcols        2.23.2         160.8K

Packages found only in centos:7.5.1804: None

Version differences:
PACKAGE                             IMAGE1 (centos:7.6.1810)        IMAGE2 (centos:7.5.1804)
-audit-libs                         2.8.4, 250.4K                   2.8.1, 250.2K
-cryptsetup-libs                    2.0.3, 1.2M                     1.7.4, 947.2K
-device-mapper                      1.02.149, 331.9K                1.02.146, 331K
-device-mapper-libs                 1.02.149, 391.2K                1.02.146, 391.2K
-elfutils-default-yama-scope        0.172, 1.8K                     0.170, 1.8K
-elfutils-libelf                    0.172, 892K                     0.170, 915K
-elfutils-libs                      0.172, 774.5K                   0.170, 730K
-glib2                              2.56.1, 11.6M                   2.54.2, 11.4M
-gobject-introspection              1.56.1, 834.2K                  1.50.0, 814.6K

And here's the output of my own tool:

- comparing 'centos:7.5.1804' with 'centos:7.6.1810'
- packages only in centos:7.5.1804:

- packages only in centos:7.6.1810:
  json-c                      0.11-4.el7_0.x86_64
  libsmartcols                2.23.2-59.el7.x86_64

- updated packages:
  audit-libs                  2.8.1-3.el7.x86_64            ->  2.8.4-4.el7.x86_64
  bash                        4.2.46-30.el7.x86_64          ->  4.2.46-31.el7.x86_64
  bind-license                9.9.4-61.el7.noarch           ->  9.9.4-72.el7.noarch
  binutils                    2.27-27.base.el7.x86_64       ->  2.27-34.base.el7.x86_64
  centos-release              7-5.1804.el7.centos.2.x86_64  ->  7-6.1810.2.el7.centos.x86_64
  coreutils                   8.22-21.el7.x86_64            ->  8.22-23.el7.x86_64
  cryptsetup-libs             1.7.4-4.el7.x86_64            ->  2.0.3-3.el7.x86_64
  curl                        7.29.0-46.el7.x86_64          ->  7.29.0-51.el7.x86_64
  dbus                        1.10.24-7.el7.x86_64          ->  1.10.24-12.el7.x86_64
  dbus-libs                   1.10.24-7.el7.x86_64          ->  1.10.24-12.el7.x86_64
  device-mapper               1.02.146-4.el7.x86_64         ->  1.02.149-8.el7.x86_64
  device-mapper-libs          1.02.146-4.el7.x86_64         ->  1.02.149-8.el7.x86_64
  dracut                      033-535.el7.x86_64            ->  033-554.el7.x86_64
  elfutils-default-yama-scope 0.170-4.el7.noarch            ->  0.172-2.el7.noarch
  elfutils-libelf             0.170-4.el7.x86_64            ->  0.172-2.el7.x86_64
  elfutils-libs               0.170-4.el7.x86_64            ->  0.172-2.el7.x86_64
  file-libs                   5.11-33.el7.x86_64            ->  5.11-35.el7.x86_64
  findutils                   4.5.11-5.el7.x86_64           ->  4.5.11-6.el7.x86_64
  glib2                       2.54.2-2.el7.x86_64           ->  2.56.1-2.el7.x86_64
  glibc                       2.17-222.el7.x86_64           ->  2.17-260.el7.x86_64
  glibc-common                2.17-222.el7.x86_64           ->  2.17-260.el7.x86_64
  gnupg2                      2.0.22-4.el7.x86_64           ->  2.0.22-5.el7_5.x86_64
  gobject-introspection       1.50.0-1.el7.x86_64           ->  1.56.1-1.el7.x86_64
  kmod                        20-21.el7.x86_64              ->  20-23.el7.x86_64
  kmod-libs                   20-21.el7.x86_64              ->  20-23.el7.x86_64
  kpartx                      0.4.9-119.el7.x86_64          ->  0.4.9-123.el7.x86_64
  krb5-libs                   1.15.1-19.el7.x86_64          ->  1.15.1-34.el7.x86_64
  libblkid                    2.23.2-52.el7.x86_64          ->  2.23.2-59.el7.x86_64
  libcom_err                  1.42.9-12.el7_5.x86_64        ->  1.42.9-13.el7.x86_64
  libcurl                     7.29.0-46.el7.x86_64          ->  7.29.0-51.el7.x86_64
  libgcc                      4.8.5-28.el7_5.1.x86_64       ->  4.8.5-36.el7.x86_64
  libmount                    2.23.2-52.el7.x86_64          ->  2.23.2-59.el7.x86_64
  libselinux                  2.5-12.el7.x86_64             ->  2.5-14.1.el7.x86_64
  libsemanage                 2.5-11.el7.x86_64             ->  2.5-14.el7.x86_64
  libsepol                    2.5-8.1.el7.x86_64            ->  2.5-10.el7.x86_64
  libssh2                     1.4.3-10.el7_2.1.x86_64       ->  1.4.3-12.el7.x86_64
  libstdc++                   4.8.5-28.el7_5.1.x86_64       ->  4.8.5-36.el7.x86_64
  libuuid                     2.23.2-52.el7.x86_64          ->  2.23.2-59.el7.x86_64
  nss                         3.36.0-5.el7_5.x86_64         ->  3.36.0-7.el7_5.x86_64
  nss-pem                     1.0.3-4.el7.x86_64            ->  1.0.3-5.el7.x86_64
  nss-sysinit                 3.36.0-5.el7_5.x86_64         ->  3.36.0-7.el7_5.x86_64
  nss-tools                   3.36.0-5.el7_5.x86_64         ->  3.36.0-7.el7_5.x86_64
  openldap                    2.4.44-15.el7_5.x86_64        ->  2.4.44-20.el7.x86_64
  openssl-libs                1.0.2k-12.el7.x86_64          ->  1.0.2k-16.el7.x86_64
  procps-ng                   3.3.10-17.el7_5.2.x86_64      ->  3.3.10-23.el7.x86_64
  python                      2.7.5-68.el7.x86_64           ->  2.7.5-76.el7.x86_64
  python-libs                 2.7.5-68.el7.x86_64           ->  2.7.5-76.el7.x86_64
  python-urlgrabber           3.10-8.el7.noarch             ->  3.10-9.el7.noarch
  rpm                         4.11.3-32.el7.x86_64          ->  4.11.3-35.el7.x86_64
  rpm-build-libs              4.11.3-32.el7.x86_64          ->  4.11.3-35.el7.x86_64
  rpm-libs                    4.11.3-32.el7.x86_64          ->  4.11.3-35.el7.x86_64
  rpm-python                  4.11.3-32.el7.x86_64          ->  4.11.3-35.el7.x86_64
  setup                       2.8.71-9.el7.noarch           ->  2.8.71-10.el7.noarch
  shadow-utils                4.1.5.1-24.el7.x86_64         ->  4.1.5.1-25.el7.x86_64
  systemd                     219-57.el7.x86_64             ->  219-62.el7.x86_64
  systemd-libs                219-57.el7.x86_64             ->  219-62.el7.x86_64
  tar                         1.26-34.el7.x86_64            ->  1.26-35.el7.x86_64
  util-linux                  2.23.2-52.el7.x86_64          ->  2.23.2-59.el7.x86_64
  vim-minimal                 7.4.160-4.el7.x86_64          ->  7.4.160-5.el7.x86_64
  yum                         3.4.3-158.el7.centos.noarch   ->  3.4.3-161.el7.centos.noarch
  yum-plugin-fastestmirror    1.1.31-45.el7.noarch          ->  1.1.31-50.el7.noarch
  yum-plugin-ovl              1.1.31-45.el7.noarch          ->  1.1.31-50.el7.noarch
  yum-utils                   1.1.31-45.el7.noarch          ->  1.1.31-50.el7.noarch
  zlib                        1.2.7-17.el7.x86_64           ->  1.2.7-18.el7.x86_64

As you can see, a whole lot more packages changed.

It seems as if container-diff only compares the rpm version, not the release. As such, if the version changes (e.g. audit-libs from 2.8.1 to 2.8.4) the change is picked up. But if only the release changes (e.g. zlib from 1.2.7-17 to 1.2.7-18) the change is omitted.

These are mainly security updates, so I think it is very important for them to show up in the rpm diff.

[nkubala]

@scopev24 ahh, sorry I misunderstood your issue originally. I think you're right, it looks like the rpm differ is ignoring the release when retrieving the list of packages from the database:

container-diff/differs/rpm_diff.go

Lines 52 to 55 in 2f96a09

	var rpmCmd = []string{
	"rpm", "--nodigest", "--nosignature",
	"-qa", "--qf", "%{NAME}\t%{VERSION}\t%{SIZE}\n",
	}

I haven't tested it, but I think just by changing the format of the query passed to rpm this issue could be fixed. not totally sure though, I didn't write this differ :) are you interested in digging into this a little and sending a PR?

[nscheer]

Hi!
Thanks! Sorry, but I didn't find the time to look into it...

[donmccasland]

NP. Looks like we've got a fix for it.