[bug]: `Inconsistency detected by ld.so` when generating or switching models

[brucethemoose]

Is there an existing issue for this?

• I have searched the existing issues

OS

Linux

GPU

cuda

VRAM

6GB

What happened?

In the git version of invokeai, switching models or trying to generate an images results in this error, with no stacktrace:

Inconsistency detected by ld.so: ../elf/dl-tls.c: 618: _dl_allocate_tls_init: Assertion `listp != NULL' failed!

Looking this up it looks to be... some kind of glibc issue? 😮 It seems to persist with another version of glibc though.

These are the last few lines of the linux strace command:

openat(AT_FDCWD, "/home/alpha/Storage/AIModels/invokeinstall/models/diffusers/models--stabilityai--sd-vae-ft-mse/snapshots/ad7ac2cf88578c68f660449f60fe9496f35a1cbf/config.json", O_RDONLY|O_CLOEXEC) = 40
newfstatat(40, "", {st_mode=S_IFREG|0644, st_size=547, ...}, AT_EMPTY_PATH) = 0
ioctl(40, TCGETS, 0x7ffd566ea0f0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(40, 0, SEEK_CUR) = 0
lseek(40, 0, SEEK_CUR) = 0
newfstatat(40, "", {st_mode=S_IFREG|0644, st_size=547, ...}, AT_EMPTY_PATH) = 0
read(40, "{\n  \"_class_name\": \"AutoencoderK"..., 548) = 547
read(40, "", 1) = 0
close(40)             = 0
brk(0xaa2f3000)       = 0xaa2f3000
brk(0xaa773000)       = 0xaa773000
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0a3bf00000
openat(AT_FDCWD, "/home/alpha/Storage/AIModels/invokeinstall/models/diffusers/models--stabilityai--sd-vae-ft-mse/snapshots/ad7ac2cf88578c68f660449f60fe9496f35a1cbf/diffusion_pytorch_model.safetensors", O_RDONLY|O_CLOEXEC) = 40
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address)
statx(40, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=334643276, ...}) = 0
mmap(NULL, 334643276, PROT_READ, MAP_SHARED, 40, 0) = 0x7f0a27e00000
openat(AT_FDCWD, "/home/alpha/Storage/AIModels/invokeinstall/models/diffusers/models--stabilityai--sd-vae-ft-mse/snapshots/ad7ac2cf88578c68f660449f60fe9496f35a1cbf/diffusion_pytorch_model.safetensors", O_RDONLY) = 41
fstat(41, {st_mode=S_IFREG|0644, st_size=334643276, ...}) = 0
mmap(NULL, 334643276, PROT_READ|PROT_WRITE, MAP_PRIVATE, 41, 0) = 0x7f09fc000000
close(41)             = 0
munmap(0x7f0a27e00000, 334643276) = 0
close(40)             = 0
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f0a3b6ff000
mprotect(0x7f0a3b700000, 8388608, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
writev(2, [{iov_base="Inconsistency detected by ld.so:"..., iov_len=33}, {iov_base="../elf/dl-tls.c", iov_len=15}, {iov_base=": ", iov_len=2}, {iov_base="618", iov_len=3}, {iov_base=": ", iov_len=2}, {iov_base="_dl_allocate_tls_init", iov_len=21}, {iov_base=": ", iov_len=2}, {iov_base="Assertion `", iov_len=11}, {iov_base="listp != NULL", iov_len=13}, {iov_base="' failed!\n", iov_len=10}], 10Inconsistency detected by ld.so: ../elf/dl-tls.c: 618: _dl_allocate_tls_init: Assertion `listp != NULL' failed!

Looks like its erroring it when it tries to use the VAE? The issue persists after deleting the whole VAE folder and letting it redownload.

The last few lines of the python trace module are less informative:

grad_mode.py(152):         self.mode = mode
module.py(807):                 param_applied = fn(param)
 --- modulename: module, funcname: convert
module.py(1127):             if convert_to_format is not None and t.dim() in (4, 5):
module.py(1130):             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
module.py(806):             with torch.no_grad():
 --- modulename: grad_mode, funcname: __exit__
grad_mode.py(58):         torch.set_grad_enabled(self.prev)
 --- modulename: grad_mode, funcname: __init__
grad_mode.py(150):         self.prev = torch.is_grad_enabled()
grad_mode.py(151):         torch._C._set_grad_enabled(mode)
grad_mode.py(152):         self.mode = mode
module.py(808):             should_use_set_data = compute_should_use_set_data(param, param_applied)
 --- modulename: module, funcname: compute_should_use_set_data
module.py(787):             if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
module.py(796):                 return not torch.__future__.get_overwrite_module_params_on_conversion()
 --- modulename: __future__, funcname: get_overwrite_module_params_on_conversion
__future__.py(21):     return _overwrite_module_params_on_conversion
module.py(809):             if should_use_set_data:
module.py(810):                 param.data = param_applied
module.py(811):                 out_param = param
module.py(818):             if param.grad is not None:
module.py(800):         for key, param in self._parameters.items():
module.py(829):         for key, buf in self._buffers.items():
module.py(833):         return self
module.py(783):         for module in self.children():
 --- modulename: module, funcname: children
module.py(2167):         for name, module in self.named_children():
 --- modulename: module, funcname: named_children
module.py(2186):         for name, module in self._modules.items():
module.py(2187):             if module is not None and module not in memo:
module.py(2188):                 memo.add(module)
module.py(2189):                 yield name, module
module.py(2168):             yield module
module.py(784):             module._apply(fn)
 --- modulename: module, funcname: _apply
module.py(783):         for module in self.children():
 --- modulename: module, funcname: children
module.py(2167):         for name, module in self.named_children():
 --- modulename: module, funcname: named_children
module.py(2185):         memo = set()
module.py(2186):         for name, module in self._modules.items():
module.py(786):         def compute_should_use_set_data(tensor, tensor_applied):
module.py(800):         for key, param in self._parameters.items():
module.py(801):             if param is None:
module.py(806):             with torch.no_grad():
 --- modulename: grad_mode, funcname: __init__
grad_mode.py(49):         if not torch._jit_internal.is_scripting():
 --- modulename: _jit_internal, funcname: is_scripting
_jit_internal.py(1105):     return False
grad_mode.py(50):             super().__init__()
grad_mode.py(51):         self.prev = False
 --- modulename: grad_mode, funcname: __enter__
grad_mode.py(54):         self.prev = torch.is_grad_enabled()
grad_mode.py(55):         torch.set_grad_enabled(False)
 --- modulename: grad_mode, funcname: __init__
grad_mode.py(150):         self.prev = torch.is_grad_enabled()
grad_mode.py(151):         torch._C._set_grad_enabled(mode)
grad_mode.py(152):         self.mode = mode
module.py(807):                 param_applied = fn(param)
 --- modulename: module, funcname: convert
module.py(1127):             if convert_to_format is not None and t.dim() in (4, 5):
module.py(1130):             return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
Inconsistency detected by ld.so: ../elf/dl-tls.c: 618: _dl_allocate_tls_init: Assertion `listp != NULL' failed!

I've got some other environment changes to test, but so far this has me stumped.

[brucethemoose]

A plain 🧨 diffusers pipeline works within the venv:

invokeai ❯ python test1.py
2023-01-23 02:06:25.881503: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Fetching 16 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 8921.68it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:02<00:00,  5.41it/s]

~/Storage/clone/StableDiffusion-Video2Video main* 11s

But I'm not sure if it works with the VAE? I guess I will see what code invoke uses to load it, to test that.

[brucethemoose]

Full reinstall of the python venv did not fix this...

[brucethemoose]

Yeah this is definitely related to loading FP32 SD 1.5 and the legacy VAE safetensors file. Trying to load waifu diffusion works fine:

* --web was specified, starting web server...
* Initializing, be patient...
>> Initialization file /home/alpha/Storage/AIModels/invokeinstall/invokeai.init found. Loading...
>> Started Invoke AI Web Server!
Point your browser at http://localhost:7680 or use the host's DNS name or IP address.
>> System config requested
>> patchmatch.patch_match: ERROR - patchmatch failed to load or compile (/home/alpha/.local/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1: version `GOMP_5.0' not found (required by /usr/lib/libvtkCommonCore.so.1)).
>> patchmatch.patch_match: INFO - Refer to https://github.com/invoke-ai/InvokeAI/blob/main/docs/installation/INSTALL_PATCHMATCH.md for installation instructions.
>> Patchmatch not loaded (nonfatal)
>> System config requested
>> Deleting Model: inpainting-1.5
>> Model Deleted: inpainting-1.5
>> Model change requested: waifu-diffusion-1.4
>> Current VRAM usage:  2.60G
>> Offloading stable-diffusion-2.1-768 to CPU
>> Loading diffusers model from hakurei/waifu-diffusion
  | Using faster float16 precision
Fetching 15 files: 100%|██████████████████████████████████████████████| 15/15 [00:00<00:00, 11799.43it/s]
  | Calculating sha256 hash of model files
  | sha256 = 66a719095ce3e8b1e5357ae9172b3d5cb883a870dfebaf84accd21724e29ba06 (31 files hashed in 3.98s)
  | Default image dimensions = 512 x 512
>> Model loaded in 7.08s
>> Max VRAM used to load the model: 2.16G
>> Current VRAM usage:2.16G
>> Textual inversions available:
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
>> Model change requested: stable-diffusion-1.5
>> Current VRAM usage:  2.16G
>> Cache limit (max=2) reached. Purging stable-diffusion-2.1-768
>> Offloading waifu-diffusion-1.4 to CPU
>> Loading diffusers model from runwayml/stable-diffusion-v1-5
  | Using faster float16 precision
  | Loading diffusers VAE from stabilityai/sd-vae-ft-mse
  | Half-precision version of model not available; fetching full-precision instead
Downloading: 100%|████████████████████████████████████████████████████| 335M/335M [00:11<00:00, 29.5MB/s]
Downloading: 100%|███████████████████████████████████████████████████████| 547/547 [00:00<00:00, 457kB/s]
Inconsistency detected by ld.so: ../elf/dl-tls.c: 618: _dl_allocate_tls_init: Assertion `listp != NULL' failed!

[brucethemoose]

Turns out I made a mistake in earlier testing, and this is a pytorch 2.0 issue.