Latest httpd-container (ubi8/httpd-24:1-230) stopped working on OCP by default

[mbaldessari]

Just noticed this today on my OCP clusters (both 4.10.x and 4.11.x) when running a container with the image: registry.access.redhat.com/ubi8/httpd-24

Error:

[ec2-user@bastion multicloud-gitops]$ oc logs -n config-demo config-demo-c4995556-7nbzf
=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
---> Generating SSL key pair for httpd...
Could not write to /opt/app-root/src/dhparams.pem. Check directory permissions.

The previous version (1-226 works fine, I just tested it). I assume 5be03ef is somewhat related. No big deal I guess, but I thought I'd at least mention it as it might affect others. The full deployment yaml we use is here https://github.com/hybrid-cloud-patterns/multicloud-gitops/blob/main/charts/all/config-demo/templates/config-demo-deployment.yaml

[pkubatrh]

Hi @mbaldessari and thanks for the report!
#138 did not get into latest UBI8-based images, so that one will be unrelated. Looking at the diff from last release, there is only #145 which does not look like it would be related to the issue either. :/

@phracek Can you take a more detailed look?

[pkubatrh]

@uhliarik might also be able to help

[toastbrotch]

same issue here with https://catalog.redhat.com/software/containers/rhel8/httpd-24

[Faradome]

It seems like the step to generate the SSL key pair on container startup in /usr/share/container-scripts/httpd/common.sh attempts to write a dhparams.pem file as part of this step to whatever is the current working directory. If this directory is not writable then the command fails.

[phracek]

@Faradome @mbaldessari I was trying to reproduce the error, but the usual command oc new-app registry.access.redhat.com/ubi8/httpd-24~https://github.com/sclorg/httpd-ex.git

works properly

oc logs httpd-ex-566d88694d-w6lt2
=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.129.2.165. Set the 'ServerName' directive globally to suppress this message
[Tue Nov 15 09:20:26.406876 2022] [ssl:warn] [pid 1:tid 140331149623552] AH01909: 10.129.2.165:8443:0 server certificate does NOT include an ID which matches the server name
[Tue Nov 15 09:20:26.407039 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/) configured.
[Tue Nov 15 09:20:26.407045 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity: APR compiled version="1.6.3"; loaded version="1.6.3"
[Tue Nov 15 09:20:26.407048 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity: PCRE compiled version="8.42 "; loaded version="8.42 2018-03-20"
[Tue Nov 15 09:20:26.407055 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity: LUA compiled version="Lua 5.3"
[Tue Nov 15 09:20:26.407056 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity: YAJL compiled version="2.1.0"
[Tue Nov 15 09:20:26.407057 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity: LIBXML compiled version="2.9.7"
[Tue Nov 15 09:20:26.407058 2022] [:notice] [pid 1:tid 140331149623552] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.129.2.165. Set the 'ServerName' directive globally to suppress this message
[Tue Nov 15 09:20:26.493075 2022] [ssl:warn] [pid 1:tid 140331149623552] AH01909: 10.129.2.165:8443:0 server certificate does NOT include an ID which matches the server name
[Tue Nov 15 09:20:26.493206 2022] [lbmethod_heartbeat:notice] [pid 1:tid 140331149623552] AH02282: No slotmem from mod_heartmonitor
[Tue Nov 15 09:20:26.497161 2022] [mpm_event:notice] [pid 1:tid 140331149623552] AH00489: Apache/2.4.37 (Red Hat Enterprise Linux) OpenSSL/1.1.1k configured -- resuming normal operations
[Tue Nov 15 09:20:26.497180 2022] [core:notice] [pid 1:tid 140331149623552] AH00094: Command line: 'httpd -D FOREGROUND'

It seems this is not your case.
Can you please provide us reproducer?

[uhliarik]

I'm not aware of any recent change in httpd, that could cause this behavior. As phracek requested, could you please provide us a reproducer?

[Faradome]

The developer who brought this to me was using this as a base image in an external build process and not in a S2I build process.
His docker file is as follows, what broke the build was the WORKDIR /data in which /data does not exist in the container.

FROM registry.redhat.io/rhel8/httpd-24:latest
WORKDIR /data
COPY endep-client/dist/ /var/www/html/
COPY ./endep-client/custom-httpd.conf /etc/httpd/conf.d/custom.conf
ENV TZ='America/New_York'

While in this case setting WORKDIR to a non-existent directory is improper the same behavior can be triggered when the WORKDIR is not writable by the running container which some may do for security. A simple replication of this case can be done by building as:

FROM registry.redhat.io/rhel8/httpd-24:latest
USER 0
RUN chmod 644 /opt/app-root/src
USER 1001

docker build . -t local/test:latest
...

docker run --rm local/test:latest
find: Failed to restore initial working directory: /opt/app-root/src: Permission denied
=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
---> Generating SSL key pair for httpd...
Could not write to /opt/app-root/src/dhparams.pem. Check directory permissions.

From what I am able to tell the sscg command is seemingly is missing the --dhparams-file argument:
https://github.com/sclorg/httpd-container/blob/master/2.4/root/usr/share/container-scripts/httpd/common.sh#L31

  echo "---> Generating SSL key pair for httpd..."
  if [ -x "/usr/bin/sscg" ]; then
    sscg -q                                                           \
       --cert-file           $sslcert                                 \
       --cert-key-file       $sslkey                                  \
       --ca-file             $sslcert                                 \
       --lifetime            365                                      \
       --hostname            $fqdn                                    \
       --email               root@$fqdn
  else
...

Without specifying where to write the dhparams-file the command attempts to write the file to whatever is the current working directory for the container, being that the WORKDIR is not writable the sscg command fails and causes the container start to fail.

In terms of changes this is wrapped in a if/else where if /usr/bin/sscg does not exist openssl is called to do this generation and there is no reference to writing a dhparams file. It would be interesting to know when the sscg command became available in the image.

[notroj]

Good comment @Faradome - and sscg was rebased in 8.7 so maybe that's what has changed since 8.6?

[notroj]

I filed https://bugzilla.redhat.com/show_bug.cgi?id=2143206 to see if the maintainer will accept this as a regression.