-
-
Notifications
You must be signed in to change notification settings - Fork 62
Add CentOS/RHEL 8 support with py3 and no mistral #630
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks much cleaner! Great work! 👍
I've added a few more minor comments to address before final merge.
|
Looks like the build takes 15mins for el8, which is highest among all other distros. Based on build log, during These dependencies should be ideally already shipped within the Test Docker image so we don't need to re-install them over and over again at least for this stage. |
|
@armab I moved the rpm installs to dockerfiles: StackStorm/st2packaging-dockerfiles#80 |
Co-Authored-By: Eugen C. <armab@users.noreply.github.com>
|
So the build time is still ridiculous. I think we should merge it when it’s time though, and I’ll dig in in a few weeks to see what I can do to reduce it. I need to go and chat with the folks over at Freenode#RPM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code in PR now looks pretty good to me.
However while trying to install produced st2 rpm package in CentOS8 Vagrant VM I hit the following issue:
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction check error:
file /usr/lib/.build-id/16/d25f8f7c82ee5383c3a140e6398270731546f3 from install of st2-3.2dev-1.x86_64 conflicts with file from package platform-python-3.6.8-1.el8.0.1.x86_64
Error Summary
-------------
Steps to reproduce
vagrant init centos/8; vagrant up; vagrant ssh- wget
st2-3.2dev-1.x86_64.rpmfrom CircleCI Artifacts: https://circleci.com/gh/syncurity-exchange/st2-packages/47#artifacts/containers/0 sudo yum install ./st2-3.2dev-1.x86_64.rpm
Result:
[vagrant@centos8 ~]$ sudo yum install ./st2-3.2dev-1.x86_64.rpm
Last metadata expiration check: 0:01:09 ago on Fri 10 Jan 2020 06:39:22 PM UTC.
Dependencies resolved.
=====================================================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================================================
Installing:
st2 x86_64 3.2dev-1 @commandline 29 M
Installing dependencies:
dwz x86_64 0.12-9.el8 AppStream 109 k
efi-srpm-macros noarch 3-2.el8 AppStream 22 k
ghc-srpm-macros noarch 1.4.2-7.el8 AppStream 9.3 k
git x86_64 2.18.1-3.el8 AppStream 186 k
git-core x86_64 2.18.1-3.el8 AppStream 4.1 M
git-core-doc noarch 2.18.1-3.el8 AppStream 2.3 M
go-srpm-macros noarch 2-16.el8 AppStream 14 k
ocaml-srpm-macros noarch 5-4.el8 AppStream 9.4 k
openblas-srpm-macros noarch 2-2.el8 AppStream 7.9 k
perl-Digest noarch 1.17-395.el8 AppStream 27 k
perl-Digest-MD5 x86_64 2.55-396.el8 AppStream 37 k
perl-Error noarch 1:0.17025-2.el8 AppStream 46 k
perl-Git noarch 2.18.1-3.el8 AppStream 77 k
perl-Net-SSLeay x86_64 1.85-6.el8 AppStream 358 k
perl-TermReadKey x86_64 2.37-7.el8 AppStream 40 k
perl-URI noarch 1.73-3.el8 AppStream 116 k
perl-libnet noarch 3.11-3.el8 AppStream 121 k
perl-srpm-macros noarch 1-25.el8 AppStream 11 k
platform-python-devel x86_64 3.6.8-1.el8.0.1 AppStream 242 k
python-rpm-macros noarch 3-37.el8 AppStream 14 k
python-srpm-macros noarch 3-37.el8 AppStream 14 k
python3-pip noarch 9.0.3-13.el8 AppStream 18 k
python3-rpm-generators noarch 5-4.el8 AppStream 24 k
python3-rpm-macros noarch 3-37.el8 AppStream 13 k
python36 x86_64 3.6.8-2.module_el8.0.0+33+0a10c0e1 AppStream 19 k
python36-devel x86_64 3.6.8-2.module_el8.0.0+33+0a10c0e1 AppStream 16 k
qt5-srpm-macros noarch 5.11.1-2.el8 AppStream 11 k
redhat-rpm-config noarch 116-1.el8.0.1 AppStream 82 k
rust-srpm-macros noarch 5-2.el8 AppStream 9.2 k
emacs-filesystem noarch 1:26.1-5.el8 BaseOS 69 k
keyutils-libs-devel x86_64 1.5.10-6.el8 BaseOS 48 k
krb5-devel x86_64 1.16.1-22.el8 BaseOS 546 k
libcom_err-devel x86_64 1.44.3-2.el8 BaseOS 37 k
libffi-devel x86_64 3.1-18.el8 BaseOS 28 k
libkadm5 x86_64 1.16.1-22.el8 BaseOS 184 k
libselinux-devel x86_64 2.8-6.el8 BaseOS 199 k
libsepol-devel x86_64 2.8-2.el8 BaseOS 85 k
libverto-devel x86_64 0.3.0-5.el8 BaseOS 18 k
openssl-devel x86_64 1:1.1.1-8.el8 BaseOS 2.3 M
pcre2-devel x86_64 10.32-1.el8 BaseOS 605 k
pcre2-utf16 x86_64 10.32-1.el8 BaseOS 228 k
pcre2-utf32 x86_64 10.32-1.el8 BaseOS 220 k
perl-Carp noarch 1.42-396.el8 BaseOS 30 k
perl-Data-Dumper x86_64 2.167-399.el8 BaseOS 58 k
perl-Encode x86_64 4:2.97-3.el8 BaseOS 1.5 M
perl-Errno x86_64 1.28-416.el8 BaseOS 76 k
perl-Exporter noarch 5.72-396.el8 BaseOS 34 k
perl-File-Path noarch 2.15-2.el8 BaseOS 38 k
perl-File-Temp noarch 0.230.600-1.el8 BaseOS 63 k
perl-Getopt-Long noarch 1:2.50-4.el8 BaseOS 63 k
perl-HTTP-Tiny noarch 0.074-1.el8 BaseOS 58 k
perl-IO x86_64 1.38-416.el8 BaseOS 141 k
perl-MIME-Base64 x86_64 3.15-396.el8 BaseOS 31 k
perl-PathTools x86_64 3.74-1.el8 BaseOS 90 k
perl-Pod-Escapes noarch 1:1.07-395.el8 BaseOS 20 k
perl-Pod-Perldoc noarch 3.28-396.el8 BaseOS 86 k
perl-Pod-Simple noarch 1:3.35-395.el8 BaseOS 213 k
perl-Pod-Usage noarch 4:1.69-395.el8 BaseOS 34 k
perl-Scalar-List-Utils x86_64 3:1.49-2.el8 BaseOS 68 k
perl-Socket x86_64 4:2.027-2.el8 BaseOS 59 k
perl-Storable x86_64 1:3.11-3.el8 BaseOS 98 k
perl-Term-ANSIColor noarch 4.06-396.el8 BaseOS 46 k
perl-Term-Cap noarch 1.17-395.el8 BaseOS 23 k
perl-Text-ParseWords noarch 3.30-395.el8 BaseOS 18 k
perl-Text-Tabs+Wrap noarch 2013.0523-395.el8 BaseOS 24 k
perl-Time-Local noarch 1:1.280-1.el8 BaseOS 34 k
perl-Unicode-Normalize x86_64 1.25-396.el8 BaseOS 82 k
perl-constant noarch 1.33-396.el8 BaseOS 25 k
perl-interpreter x86_64 4:5.26.3-416.el8 BaseOS 6.3 M
perl-libs x86_64 4:5.26.3-416.el8 BaseOS 1.6 M
perl-macros x86_64 4:5.26.3-416.el8 BaseOS 72 k
perl-parent noarch 1:0.237-1.el8 BaseOS 20 k
perl-podlators noarch 4.11-1.el8 BaseOS 118 k
perl-threads x86_64 1:2.21-2.el8 BaseOS 61 k
perl-threads-shared x86_64 1.58-2.el8 BaseOS 48 k
python3-setuptools noarch 39.2.0-4.el8 BaseOS 162 k
unzip x86_64 6.0-41.el8 BaseOS 193 k
zip x86_64 3.0-23.el8 BaseOS 270 k
zlib-devel x86_64 1.2.11-10.el8 BaseOS 56 k
Installing weak dependencies:
perl-IO-Socket-IP noarch 0.39-5.el8 AppStream 47 k
perl-IO-Socket-SSL noarch 2.060-2.el8 AppStream 289 k
perl-Mozilla-CA noarch 20160104-7.el8 AppStream 15 k
Enabling module streams:
python36 3.6
Transaction Summary
=====================================================================================================================================================================
Install 83 Packages
Total size: 53 M
Total download size: 25 M
Installed size: 198 M
Is this ok [y/N]: y
Downloading Packages:
(1/82): ghc-srpm-macros-1.4.2-7.el8.noarch.rpm 8.3 kB/s | 9.3 kB 00:01
(2/82): efi-srpm-macros-3-2.el8.noarch.rpm 19 kB/s | 22 kB 00:01
(3/82): dwz-0.12-9.el8.x86_64.rpm 81 kB/s | 109 kB 00:01
(4/82): git-2.18.1-3.el8.x86_64.rpm 573 kB/s | 186 kB 00:00
(5/82): go-srpm-macros-2-16.el8.noarch.rpm 186 kB/s | 14 kB 00:00
(6/82): ocaml-srpm-macros-5-4.el8.noarch.rpm 144 kB/s | 9.4 kB 00:00
(7/82): openblas-srpm-macros-2-2.el8.noarch.rpm 90 kB/s | 7.9 kB 00:00
(8/82): perl-Digest-1.17-395.el8.noarch.rpm 379 kB/s | 27 kB 00:00
(9/82): perl-Digest-MD5-2.55-396.el8.x86_64.rpm 188 kB/s | 37 kB 00:00
(10/82): perl-Error-0.17025-2.el8.noarch.rpm 525 kB/s | 46 kB 00:00
(11/82): perl-Git-2.18.1-3.el8.noarch.rpm 570 kB/s | 77 kB 00:00
(12/82): perl-IO-Socket-IP-0.39-5.el8.noarch.rpm 797 kB/s | 47 kB 00:00
(13/82): perl-IO-Socket-SSL-2.060-2.el8.noarch.rpm 748 kB/s | 289 kB 00:00
(14/82): perl-Mozilla-CA-20160104-7.el8.noarch.rpm 191 kB/s | 15 kB 00:00
(15/82): git-core-doc-2.18.1-3.el8.noarch.rpm 1.3 MB/s | 2.3 MB 00:01
(16/82): perl-Net-SSLeay-1.85-6.el8.x86_64.rpm 741 kB/s | 358 kB 00:00
(17/82): perl-TermReadKey-2.37-7.el8.x86_64.rpm 578 kB/s | 40 kB 00:00
(18/82): perl-libnet-3.11-3.el8.noarch.rpm 1.2 MB/s | 121 kB 00:00
(19/82): perl-URI-1.73-3.el8.noarch.rpm 712 kB/s | 116 kB 00:00
(20/82): perl-srpm-macros-1-25.el8.noarch.rpm 166 kB/s | 11 kB 00:00
(21/82): python-rpm-macros-3-37.el8.noarch.rpm 179 kB/s | 14 kB 00:00
(22/82): python-srpm-macros-3-37.el8.noarch.rpm 209 kB/s | 14 kB 00:00
(23/82): python3-pip-9.0.3-13.el8.noarch.rpm 233 kB/s | 18 kB 00:00
(24/82): platform-python-devel-3.6.8-1.el8.0.1.x86_64.rpm 825 kB/s | 242 kB 00:00
(25/82): python3-rpm-generators-5-4.el8.noarch.rpm 367 kB/s | 24 kB 00:00
(26/82): python3-rpm-macros-3-37.el8.noarch.rpm 174 kB/s | 13 kB 00:00
(27/82): python36-3.6.8-2.module_el8.0.0+33+0a10c0e1.x86_64.rpm 266 kB/s | 19 kB 00:00
(28/82): python36-devel-3.6.8-2.module_el8.0.0+33+0a10c0e1.x86_64.rpm 251 kB/s | 16 kB 00:00
(29/82): qt5-srpm-macros-5.11.1-2.el8.noarch.rpm 152 kB/s | 11 kB 00:00
(30/82): rust-srpm-macros-5-2.el8.noarch.rpm 92 kB/s | 9.2 kB 00:00
(31/82): redhat-rpm-config-116-1.el8.0.1.noarch.rpm 498 kB/s | 82 kB 00:00
(32/82): git-core-2.18.1-3.el8.x86_64.rpm 1.4 MB/s | 4.1 MB 00:02
(33/82): keyutils-libs-devel-1.5.10-6.el8.x86_64.rpm 79 kB/s | 48 kB 00:00
(34/82): emacs-filesystem-26.1-5.el8.noarch.rpm 109 kB/s | 69 kB 00:00
(35/82): libffi-devel-3.1-18.el8.x86_64.rpm 202 kB/s | 28 kB 00:00
(36/82): libcom_err-devel-1.44.3-2.el8.x86_64.rpm 243 kB/s | 37 kB 00:00
(37/82): libkadm5-1.16.1-22.el8.x86_64.rpm 485 kB/s | 184 kB 00:00
(38/82): krb5-devel-1.16.1-22.el8.x86_64.rpm 559 kB/s | 546 kB 00:00
(39/82): libselinux-devel-2.8-6.el8.x86_64.rpm 436 kB/s | 199 kB 00:00
(40/82): libsepol-devel-2.8-2.el8.x86_64.rpm 537 kB/s | 85 kB 00:00
(41/82): libverto-devel-0.3.0-5.el8.x86_64.rpm 232 kB/s | 18 kB 00:00
(42/82): pcre2-utf16-10.32-1.el8.x86_64.rpm 606 kB/s | 228 kB 00:00
(43/82): pcre2-devel-10.32-1.el8.x86_64.rpm 1.0 MB/s | 605 kB 00:00
(44/82): pcre2-utf32-10.32-1.el8.x86_64.rpm 819 kB/s | 220 kB 00:00
(45/82): perl-Carp-1.42-396.el8.noarch.rpm 333 kB/s | 30 kB 00:00
(46/82): perl-Data-Dumper-2.167-399.el8.x86_64.rpm 677 kB/s | 58 kB 00:00
(47/82): perl-Errno-1.28-416.el8.x86_64.rpm 977 kB/s | 76 kB 00:00
(48/82): perl-Exporter-5.72-396.el8.noarch.rpm 388 kB/s | 34 kB 00:00
(49/82): perl-File-Path-2.15-2.el8.noarch.rpm 426 kB/s | 38 kB 00:00
(50/82): perl-File-Temp-0.230.600-1.el8.noarch.rpm 623 kB/s | 63 kB 00:00
(51/82): openssl-devel-1.1.1-8.el8.x86_64.rpm 1.8 MB/s | 2.3 MB 00:01
(52/82): perl-Getopt-Long-2.50-4.el8.noarch.rpm 697 kB/s | 63 kB 00:00
(53/82): perl-HTTP-Tiny-0.074-1.el8.noarch.rpm 617 kB/s | 58 kB 00:00
(54/82): perl-IO-1.38-416.el8.x86_64.rpm 810 kB/s | 141 kB 00:00
(55/82): perl-MIME-Base64-3.15-396.el8.x86_64.rpm 352 kB/s | 31 kB 00:00
(56/82): perl-Pod-Escapes-1.07-395.el8.noarch.rpm 231 kB/s | 20 kB 00:00
(57/82): perl-PathTools-3.74-1.el8.x86_64.rpm 517 kB/s | 90 kB 00:00
(58/82): perl-Encode-2.97-3.el8.x86_64.rpm 1.7 MB/s | 1.5 MB 00:00
(59/82): perl-Pod-Perldoc-3.28-396.el8.noarch.rpm 519 kB/s | 86 kB 00:00
(60/82): perl-Pod-Usage-1.69-395.el8.noarch.rpm 230 kB/s | 34 kB 00:00
(61/82): perl-Scalar-List-Utils-1.49-2.el8.x86_64.rpm 717 kB/s | 68 kB 00:00
(62/82): perl-Pod-Simple-3.35-395.el8.noarch.rpm 872 kB/s | 213 kB 00:00
(63/82): perl-Socket-2.027-2.el8.x86_64.rpm 682 kB/s | 59 kB 00:00
(64/82): perl-Storable-3.11-3.el8.x86_64.rpm 715 kB/s | 98 kB 00:00
(65/82): perl-Term-ANSIColor-4.06-396.el8.noarch.rpm 572 kB/s | 46 kB 00:00
(66/82): perl-Term-Cap-1.17-395.el8.noarch.rpm 165 kB/s | 23 kB 00:00
(67/82): perl-Text-ParseWords-3.30-395.el8.noarch.rpm 222 kB/s | 18 kB 00:00
(68/82): perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch.rpm 167 kB/s | 24 kB 00:00
(69/82): perl-Time-Local-1.280-1.el8.noarch.rpm 389 kB/s | 34 kB 00:00
(70/82): perl-Unicode-Normalize-1.25-396.el8.x86_64.rpm 571 kB/s | 82 kB 00:00
(71/82): perl-constant-1.33-396.el8.noarch.rpm 318 kB/s | 25 kB 00:00
(72/82): perl-macros-5.26.3-416.el8.x86_64.rpm 734 kB/s | 72 kB 00:00
(73/82): perl-parent-0.237-1.el8.noarch.rpm 222 kB/s | 20 kB 00:00
(74/82): perl-podlators-4.11-1.el8.noarch.rpm 634 kB/s | 118 kB 00:00
(75/82): perl-threads-2.21-2.el8.x86_64.rpm 663 kB/s | 61 kB 00:00
(76/82): perl-libs-5.26.3-416.el8.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00
(77/82): perl-threads-shared-1.58-2.el8.x86_64.rpm 568 kB/s | 48 kB 00:00
(78/82): python3-setuptools-39.2.0-4.el8.noarch.rpm 1.8 MB/s | 162 kB 00:00
(79/82): zip-3.0-23.el8.x86_64.rpm 2.6 MB/s | 270 kB 00:00
(80/82): unzip-6.0-41.el8.x86_64.rpm 1.0 MB/s | 193 kB 00:00
(81/82): zlib-devel-1.2.11-10.el8.x86_64.rpm 708 kB/s | 56 kB 00:00
(82/82): perl-interpreter-5.26.3-416.el8.x86_64.rpm 3.7 MB/s | 6.3 MB 00:01
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.1 MB/s | 25 MB 00:11
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction check error:
file /usr/lib/.build-id/16/d25f8f7c82ee5383c3a140e6398270731546f3 from install of st2-3.2dev-1.x86_64 conflicts with file from package platform-python-3.6.8-1.el8.0.1.x86_64
Error Summary
-------------
Looks like /usr/lib/.build-id/16/d25f8f7c82ee5383c3a140e6398270731546f3 points to /usr/libexec/platform-python3.6:
[vagrant@centos8 ~]$ ls -la /usr/lib/.build-id/16/
total 16
drwxr-xr-x. 2 root root 4096 Oct 25 18:38 .
drwxr-xr-x. 258 root root 8192 Jun 28 2019 ..
lrwxrwxrwx. 1 root root 44 May 11 2019 25faf9306f5775275cb8119a1c9a367826413f -> ../../../../usr/lib64/xtables/libxt_limit.so
lrwxrwxrwx. 1 root root 37 May 13 2019 42d890c1400d19933886ffdf46da12e25188e3 -> ../../../../usr/lib64/gconv/KOI8-R.so
lrwxrwxrwx. 1 root root 27 May 11 2019 486063f71410ddbb078e3183ad77c899b5fa9f -> ../../../../usr/sbin/xfs_io
lrwxrwxrwx. 1 root root 23 May 11 2019 803878e62dc202f9a2f7245bc06965599b9052 -> ../../../../usr/bin/tic
lrwxrwxrwx. 1 root root 37 May 23 2019 b75c68c4b4cc59d0aee2226e7ff7ced597286e -> ../../../../usr/libexec/sssd/sssd_nss
lrwxrwxrwx. 1 root root 25 May 11 2019 bc908355febcba32f386fb10e4786c45c5da4f -> ../../../../usr/bin/teamd
lrwxrwxrwx. 1 root root 53 May 11 2019 d1544525dd8a56eca03a04993f2b7861005a93 -> ../../../../usr/lib64/samba/libserver-id-db-samba4.so
lrwxrwxrwx. 1 root root 42 May 22 2019 d25f8f7c82ee5383c3a140e6398270731546f3 -> ../../../../usr/libexec/platform-python3.6
lrwxrwxrwx. 1 root root 43 May 22 2019 d25f8f7c82ee5383c3a140e6398270731546f3.1 -> ../../../../usr/libexec/platform-python3.6m
Found about platform-python in EL8: https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/
Current installed version of platform-python:
[vagrant@centos8 ~]$ sudo yum list installed platform-python
Installed Packages
platform-python.x86_64 3.6.8-1.el8.0.1 @anaconda
|
@armab Regarding the python conflict, I resolved it in my latest commit, and tested. It's related to rpm unique build ids. It's build 48 in circle, and here's the rpm file. I was able to run your Vagrant test commands and here's my output: |
| %endif | ||
|
|
||
| %if 0%{?rhel} >= 8 | ||
| %global _build_id_links none |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! 👍 Haven't heard about it.
| EHD | ||
| } | ||
|
|
||
| install_st2mistral() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Installing mistral is not needed anymore with EL8 and could be removed completely out of this script to clean up the code.
| } | ||
|
|
||
| install_st2chatops() { | ||
| # Temporary hack until proper upstream fix https://bugs.centos.org/view.php?id=13669 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this hack is EL7-related only. Let's please remove it and we'll see if we need it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work and thank you very much for EL8 core packages 👍
As st2.rpm package is working now, - merging it.
Definitely wasn't an easy nut to crack!
I left some comments around bash installer scripts to address in next PR iteration.
Successful EL8 (CentOS8) build with py3.
Supports: StackStorm/st2#4761
Major Changes:
Significant changes:
Examples:
pip==19.1.1cryptography-2.8-cp27-cp27m-manylinux1_x86_64.whlvscryptography-2.8-py2.py3-none-any_x86_64.whl) we need to build the cryptography package locally so it's available when needed. This also adds a layer to the EL8 docker build since we need to update a few packages and build and install the crypto one.definestatements were moved around since we needed to know if we were using EL8 and/or py3 before we could either populate variables or run some commands.pythoncommand (eg. notpython2/python3)Gemfileis now included because we need a specific version of the net-ssh pack to avoid conflicts.PYTHON_ALT_BINARY := python3. This allows us to let theBUILDROOT venvuse it's version of python3 three which is not/usr/bin/python3.echoedso expose the fact that we are in fact building with python3/pip3/el8.Requires:statements for EL8 to fail earlier in the build if we don't have the right packages needed in ourBUILDROOTredhatdirectory in the path to the bytecompile functions in rpmspec/helpers.spec and packages/python/st2python.spec has been removed. The needed files are in/usr/lib/rpm/redhatas well as /usr/lib/rpm` so this works universally and supports the new home for these files in EL7+setup_requiresargument.