Unable to `chef-server-ctl reconfigure` a new 15.3.2 install on Ubuntu 22.04

[dkowis]

Chef Server Version

Chef-server standalone installation 15.3.2

Platform Details

Ubuntu 22.04
Installing on a ProxmoxVM, with iSCSI backed disks on a TrueNAS.
16GB ram, 50GB overall disk space

Configuration

Standalone, new install. Have also tried upgradding. The farthest I can upgrade to seems to be 14.14.1. If I try to upgrade to anything newer than that, I get all the index corrupt issues I'll paste below.

Scenario

I'm trying to run chef-server-ctl reconfigure after a fresh install. I have also done it after an upgrade and have failed.

opensearch log

opensearch-log.zip

Steps to Reproduce

1. Create a VM on proxmox:
   

2. Install ubuntu 22.04 from the ubuntu server ISO.

3. Download the chef-infra-server package from: https://www.chef.io/downloads/tools/infra-server?v=15.3.2

4. apt install ./chef-server-core_15.3.2-1_amd64.deb

5. chef-server-ctl reconfigure

Expected Result

I was expecting to have a working chef server that I could then try restoring backups to.

Actual Result

There's the open search log which happens at this phase of the chef-server-ctl reconfigure

* runit_service[opensearch] action reload_log
      - reload log service

Recipe: infra-server::opensearch
  * component_runit_service[opensearch] action start
  Recipe: <Dynamically Defined Resource>
    * service[opensearch] action nothing (skipped due to action :nothing)
    * runit_service[opensearch] action start (up to date)
     (up to date)
Recipe: infra-server::opensearch
  * chef_sleep[10] action sleep
    - sleep 10 seconds
  * execute[add internal user to opensearch security plugin] action run

[dkowis]

I have tested this on a KVM based VM on my desktop, which has NVMe Backed storage, and it worked fine.

Perhaps there's some race condition during the setup that is causing the opensearch/elasticsearch to become corrupt after 14.14.x ?

I have confirmed I have no other data corruption problems on my server hardware, and the iSCSI disks.

[uskf]

I have confirmed same "CorruptIndexException" log with chef-server 15.3.2 on Ubuntu 20.04.
Perhaps this problem is caused by zlib v1.2.12 and resolved by zlib v1.2.13.
madler/zlib#613

Replacing libz.so.1.2.12 to libz.so.1.2.13 resolved this problem.

$ tar zxf zlib-1.2.13.tar.gz
$ cd zlib-1.2.13
$ ./configure
$ make
$ sudo cp libz.so.1.2.13 /opt/opscode/embedded/lib/
$ cd /opt/opscode/embedded/lib/
$ sudo ln -sf libz.so.1.2.13 libz.so
$ sudo ln -sf libz.so.1.2.13 libz.so.1

[dkowis]

Replacing libz.so.1.2.12 to libz.so.1.2.13 resolved this problem.

To get the order of operations correct:

1. Install from package
2. update zlib as you described, installing it into the opscode embedded
3. chef-server-ctl reconfigure so that it installs

Yes?

[uskf]

@dkowis Yes!

AFAIK, this problem occurs on cpu that does not support SSE 4.2.
If the physical cpu supports SSE 4.2, enabling SSE 4.2 on the virtual cpu resolves this problem.

[dkowis]

Ah yeah that'd do it, these CPUs are slightly too old for that: https://ark.intel.com/content/www/us/en/ark/products/75277/intel-xeon-processor-e52680-v2-25m-cache-2-80-ghz.html

Thanks for finding this! I hope the chef team updates it!

[dkowis]

Ah yeah that'd do it, these CPUs are slightly too old for that: https://ark.intel.com/content/www/us/en/ark/products/75277/intel-xeon-processor-e52680-v2-25m-cache-2-80-ghz.html

Actually, I think this CPU should support SSE4.2 It's supposedly Sandy Bridge era, which, in theory, supports SSE 4.2 HRMMM

[dkowis]

Yeah, I don't see it passed through in proxmox, so I've gotta figure that part out!

[uskf]

probably same problem #3451