Skip to content

nut-scanner: additional multiarch libraries search path; general: fixes to (debug) logging#2082

Merged
jimklimov merged 16 commits intonetworkupstools:masterfrom
jimklimov:issue-317
Oct 1, 2023
Merged

nut-scanner: additional multiarch libraries search path; general: fixes to (debug) logging#2082
jimklimov merged 16 commits intonetworkupstools:masterfrom
jimklimov:issue-317

Conversation

@jimklimov
Copy link
Copy Markdown
Member

@jimklimov jimklimov commented Sep 30, 2023

Closes: #317

On one hand, this PR does what issue #317 asked for, to consider "multiarch" path components (if suggested by compiler) when we consider dynamic library loading at run-time (nut-scanner, eventually DMF, etc.) Resulting paths are constructed per https://wiki.debian.org/Multiarch/LibraryPathOverview (prefixing (/usr)/lib(bits) essentially).

On another, it uncovered and addressed some earlier issues (bugs and annoyances):

  • the get_libname_in_pathset() method (added after 2.8.0 release, largely as part of NUT for Windows effort) had some issues both with accounting the number of directories checked, and with looking at them in the reentrant fashion (strtok() involved in parsing mangled the original string, such as getenv("PATH") original, adding 0x00 chars where it found separators).
  • the upsdebugx() => vupslog() methods limited the printable reports to LARGEBUF (1024 bytes currently) as compiled into the program. This truncated reports of PATH (quite long on Windows) for example, and was not hard to replace by a dynamically growing buffer to fit the conversions. Default behavior should be same as before, and it should fit the majority of practical cases, but there should be fewer truncations now. Note that receivers like syslog may do their own truncations of very long lines.
  • nut-scanner also failed to parse CIDR on Windows (needs special init, missed here)

Something not addressed here was a lingering idea to check if the initially successfully prepared buf string length is comparable with bufsize (minus... how much?) and if it is - also extend, so that our later suffixes (errno details, etc.) would fit.

…ols#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
@jimklimov jimklimov added bug enhancement Windows nut-scanner portability We want NUT to build and run everywhere possible labels Sep 30, 2023
@jimklimov jimklimov added this to the 2.8.1 milestone Sep 30, 2023
Also bumps public API version of libnutscan due to added symbol.

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…ts [networkupstools#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…ual search order of get_libname() with more locations possibly checked [networkupstools#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…RGET_ALIAS as/if suggested by compiler [networkupstools#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…thset string

It was unexpected to see just one entry in getenv("PATH") after the first loop...

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…rectory counter

Be sure to bump the number, not the pointer!

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…later calls do not corrupt what we want to report; fix declarations-on-top for windows builds

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…e for longer printouts, e.g. PATH in upsdebugx_report_search_paths() [networkupstools#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…essage

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
@jimklimov
Copy link
Copy Markdown
Member Author

  • Example output via nutscanner -h on Linux:
Run-time loadable library search paths used by this build of NUT:
        Via LD_LIBRARY_PATH:    /home/localuser/nut/tools/nut-scanner/.libs
        Built-in:       /usr/local/ups/lib
        Built-in:       /usr/usr/local/ups/lib
        Built-in:       /usr/local/usr/local/ups/lib
        Built-in:       /usr/lib/64/x86_64-linux-gnu
        Built-in:       /usr/lib64/x86_64-linux-gnu
        Built-in:       /lib/64/x86_64-linux-gnu
        Built-in:       /lib64/x86_64-linux-gnu
        Built-in:       /usr/lib/x86_64-linux-gnu
        Built-in:       /lib/x86_64-linux-gnu
        Built-in:       /usr/lib/64
        Built-in:       /usr/lib64
        Built-in:       /usr/lib
        Built-in:       /lib/64
        Built-in:       /lib64
        Built-in:       /lib
        Built-in:       /usr/local/lib/64
        Built-in:       /usr/local/lib64
        Built-in:       /usr/local/lib
        Built-in:       /usr/lib/x86_64-linux-gnu
        Built-in:       /usr/lib/gcc/x86_64-linux-gnu
        Built-in:       /usr/lib/x86_64-pc-linux-gnu
        Built-in:       /usr/lib/gcc/x86_64-pc-linux-gnu
  • Example output via nutscanner -D with a Windows cross-build on Linux (early in nutscan-init in its library):
:;  ./scripts/Windows/nut_install/bin/nut-scanner.exe -D -h
   0.000000     [D1] Run-time loadable library search paths used by this build of NUT:
   0.003001     [D1]    Built-in:       //lib
   0.004002     [D1]    Built-in:       /usr//lib
   0.006000     [D1]    Built-in:       /usr/local//lib
   0.007002     [D1]    Built-in:       /usr/lib/64
   0.009001     [D1]    Built-in:       /usr/lib64
   0.010001     [D1]    Built-in:       /usr/lib
   0.012002     [D1]    Built-in:       /lib/64
   0.013001     [D1]    Built-in:       /lib64
   0.014002     [D1]    Built-in:       /lib
   0.015000     [D1]    Built-in:       /usr/local/lib/64
   0.017002     [D1]    Built-in:       /usr/local/lib64
   0.018002     [D1]    Built-in:       /usr/local/lib
   0.020001     [D1]    Built-in:       /usr/lib/x86_64-mingw32
   0.022000     [D1]    Built-in:       /usr/lib/gcc/x86_64-mingw32
   0.023001     [D1]    Built-in:       /usr/lib/x86_64-w64-mingw32
   0.024999     [D1]    Built-in:       /usr/lib/gcc/x86_64-w64-mingw32
   0.028002     [D1]    Built-in:       .
   0.029000     [D1]    Windows near EXE:       \\wsl$\Ubuntu\home\localuser\nut\scripts\Windows\nut_install\bin
   0.032000     [D1]    Windows PATH_LIB (\..\lib):     \\wsl$\Ubuntu\home\localuser\nut\scripts\Windows\nut_install\bin\..\lib
   0.036001     [D1]    Windows "lib" dir near EXE:     \\wsl$\Ubuntu\home\localuser\nut\scripts\Windows\nut_install\bin/../lib
WARNING: vupslog: vsnprintf needed more than 1024 bytes: -1 (34 => Result too large), extending to 2048
   0.043002     [D1]    Windows via PATH:       C:\Program Files\Eclipse Adoptium\jdk-11.0.17.8-hotspot\bin;C:\Python310\Scripts\;C:\Python310\;C:\Program Files\Eclipse Adoptium\jdk-8.0.342.7-hotspot\bin;C:\Program Files\Eclipse Adoptium\jdk-17.0.3.7-hotspot\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\Program Files\PuTTY\;%LOCALAPPDATA%\Atlassian\SourceTree\git_local\bin;C:\Program Files\Git\cmd;C:\Program Files\WinMerge;c:\progra~1\Apache\maven\bin;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files\Microsoft VS Code\bin;C:\Program Files\dotnet\;C:\Program Files\Docker\Docker\resources\bin;C:\Users\localuser\AppData\Local\Microsoft\WindowsApps;C:\Users\localuser\AppData\Roaming\npm;C:\Program Files\ffmpeg-2022-10-13-git-9e8a327e68-full_build\bin;C:\Program Files\7-Zip;C:\Users\localuser\Local Settings\Android\Sdk\platform-tools;C:\msys64\mingw64\bin;C:\msys64\usr\bin;C:\Users\localuser\.dotnet\tools
   0.085000     [D1] nutscan_init: skipped loading the library for LibUSB: was absent during NUT build

Notably, there is further area of improvement to de-duplicate search_path[] entries (or generally those to visit), to reduce the overheads of searching and logging in vain, and for Windows builds - maybe to extend the %ENVVARNAME%/bin sort of paths.

@AppVeyorBot

This comment was marked as resolved.

@jimklimov
Copy link
Copy Markdown
Member Author

On native Ubuntu LTS both the gcc and clang report a -print-multiarch value, but on the same system a cross-building x86_64-w64-mingw32-gcc does not. MSYS2 (semi-native Windows builds) also uses complicated path naming, but not a multiarch from the compiler.

Sample NUT config report for where it works:

NUT Build/Target system info:
-----------------------------

* Compact version of C compiler:        gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
* Compact version of C++ compiler:      g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
* Compact version of C preprocessor:    gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
* host env spec we run on:      x86_64-pc-linux-gnu
* host env spec we built on:    x86_64-pc-linux-gnu
* host env spec we built for:   x86_64-pc-linux-gnu
* host OS short spec we run on: x86_64-linux-gnu
* host OS short spec we built on:       x86_64-linux-gnu
* host OS short spec we built for:      x86_64-linux-gnu
* host multiarch spec we build for (as suggested by compiler being used):       x86_64-linux-gnu

…o retry from one mode to another)

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…ore getaddrinfo() et al

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
@jimklimov
Copy link
Copy Markdown
Member Author

On Windows (semi-native build with MSYS2):

Note: many scanning options depend on further loadable libraries.
Run-time loadable library search paths used by this build of NUT:
        Via LD_LIBRARY_PATH:    C:\msys64\home\localuser\nut-win\clients;C:\msys64\home\localuser\nut-win\clients\.libs
        Built-in:       /mingw32/lib
        Built-in:       /usr/mingw32/lib
        Built-in:       /usr/local/mingw32/lib
        Built-in:       /usr/lib/64
        Built-in:       /usr/lib64
        Built-in:       /usr/lib
        Built-in:       /lib/64
        Built-in:       /lib64
        Built-in:       /lib
        Built-in:       /usr/local/lib/64
        Built-in:       /usr/local/lib64
        Built-in:       /usr/local/lib
        Built-in:       /usr/lib/i686-mingw32
        Built-in:       /usr/lib/gcc/i686-mingw32
        Built-in:       /usr/lib/i686-w64-mingw32
        Built-in:       /usr/lib/gcc/i686-w64-mingw32
        Built-in:       .
        Windows near EXE:       C:\msys64\home\localuser\nut-win\tools\nut-scanner\.libs
        Windows PATH_LIB (\..\lib):     C:\msys64\home\localuser\nut-win\tools\nut-scanner\.libs\..\lib
        Windows "lib" dir near EXE:     C:\msys64\home\localuser\nut-win\tools\nut-scanner\.libs/../lib
        Windows via PATH:       C:\msys64\home\localuser\nut-win\tools\nut-scanner\.libs;C:\msys64\home\localuser\nut-win\tools\nut-scanner\.libs;C:\msys64\mingw32\lib;C:\msys64\mingw32\bin;C:\msys64\mingw64\bin;C:\msys64\mingw32\bin;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\msys64\usr\bin\site_perl;C:\msys64\usr\bin\vendor_perl;C:\msys64\usr\bin\core_perl

@AppVeyorBot

This comment was marked as resolved.

@jimklimov
Copy link
Copy Markdown
Member Author

With pre-filter (on Linux) over "hard-coded" paths:

   0.000001     [D5] nut_prepare_search_paths: SKIP unreachable directory #0 : /usr/local/ups/lib
   0.000029     [D5] nut_prepare_search_paths: SKIP unreachable directory #1 : /usr/usr/local/ups/lib
   0.000032     [D5] nut_prepare_search_paths: SKIP unreachable directory #2 : /usr/local/usr/local/ups/lib
   0.000034     [D5] nut_prepare_search_paths: SKIP unreachable directory #3 : /usr/lib/64/x86_64-linux-gnu
   0.000036     [D5] nut_prepare_search_paths: SKIP unreachable directory #4 : /usr/lib64/x86_64-linux-gnu
   0.000038     [D5] nut_prepare_search_paths: SKIP unreachable directory #5 : /lib/64/x86_64-linux-gnu
   0.000039     [D5] nut_prepare_search_paths: SKIP unreachable directory #6 : /lib64/x86_64-linux-gnu
   0.000043     [D5] nut_prepare_search_paths: ADD existing unique directory #0 : /usr/lib/x86_64-linux-gnu
   0.000046     [D5] nut_prepare_search_paths: ADD existing unique directory #1 : /lib/x86_64-linux-gnu
   0.000047     [D5] nut_prepare_search_paths: SKIP unreachable directory #9 : /usr/lib/64
   0.000050     [D5] nut_prepare_search_paths: ADD existing unique directory #2 : /usr/lib64
   0.000054     [D5] nut_prepare_search_paths: ADD existing unique directory #3 : /usr/lib
   0.000056     [D5] nut_prepare_search_paths: SKIP unreachable directory #12 : /lib/64
   0.000058     [D5] nut_prepare_search_paths: ADD existing unique directory #4 : /lib64
   0.000061     [D5] nut_prepare_search_paths: ADD existing unique directory #5 : /lib
   0.000063     [D5] nut_prepare_search_paths: SKIP unreachable directory #15 : /usr/local/lib/64
   0.000065     [D5] nut_prepare_search_paths: SKIP unreachable directory #16 : /usr/local/lib64
   0.000067     [D5] nut_prepare_search_paths: ADD existing unique directory #6 : /usr/local/lib
   0.000071     [D5] nut_prepare_search_paths: SKIP duplicate directory #19 : /usr/lib/x86_64-linux-gnu
   0.000076     [D5] nut_prepare_search_paths: ADD existing unique directory #7 : /usr/lib/gcc/x86_64-linux-gnu
   0.000078     [D5] nut_prepare_search_paths: SKIP unreachable directory #20 : /usr/lib/x86_64-pc-linux-gnu
   0.000080     [D5] nut_prepare_search_paths: SKIP unreachable directory #21 : /usr/lib/gcc/x86_64-pc-linux-gnu
   0.000083     [D1] Run-time loadable library search paths used by this build of NUT:
   0.000084     [D1]    Via LD_LIBRARY_PATH:    /home/localuser/nut/tools/nut-scanner/.libs
   0.000085     [D1]    NOTE: Reporting raw built-in paths:
   0.000086     [D1]    Built-in:       /usr/local/ups/lib
   0.000087     [D1]    Built-in:       /usr/usr/local/ups/lib
   0.000087     [D1]    Built-in:       /usr/local/usr/local/ups/lib
   0.000088     [D1]    Built-in:       /usr/lib/64/x86_64-linux-gnu
   0.000089     [D1]    Built-in:       /usr/lib64/x86_64-linux-gnu
   0.000089     [D1]    Built-in:       /lib/64/x86_64-linux-gnu
   0.000090     [D1]    Built-in:       /lib64/x86_64-linux-gnu
   0.000091     [D1]    Built-in:       /usr/lib/x86_64-linux-gnu
   0.000091     [D1]    Built-in:       /lib/x86_64-linux-gnu
   0.000092     [D1]    Built-in:       /usr/lib/64
   0.000092     [D1]    Built-in:       /usr/lib64
   0.000093     [D1]    Built-in:       /usr/lib
   0.000094     [D1]    Built-in:       /lib/64
   0.000094     [D1]    Built-in:       /lib64
   0.000095     [D1]    Built-in:       /lib
   0.000096     [D1]    Built-in:       /usr/local/lib/64
   0.000096     [D1]    Built-in:       /usr/local/lib64
   0.000097     [D1]    Built-in:       /usr/local/lib
   0.000098     [D1]    Built-in:       /usr/lib/x86_64-linux-gnu
   0.000098     [D1]    Built-in:       /usr/lib/gcc/x86_64-linux-gnu
   0.000099     [D1]    Built-in:       /usr/lib/x86_64-pc-linux-gnu
   0.000100     [D1]    Built-in:       /usr/lib/gcc/x86_64-pc-linux-gnu
...

Note: many scanning options depend on further loadable libraries.
Run-time loadable library search paths used by this build of NUT:
        Via LD_LIBRARY_PATH:    /home/localuser/nut/tools/nut-scanner/.libs
        NOTE: Reporting filtered (existing unique) built-in paths:
        Built-in:       /usr/lib/x86_64-linux-gnu
        Built-in:       /lib/x86_64-linux-gnu
        Built-in:       /usr/lib64
        Built-in:       /usr/lib
        Built-in:       /lib64
        Built-in:       /lib
        Built-in:       /usr/local/lib
        Built-in:       /usr/lib/gcc/x86_64-linux-gnu

…er the usable set [networkupstools#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…ilable, and so xstrdup()/free() in general [networkupstools#317]

Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
@jimklimov jimklimov merged commit db66837 into networkupstools:master Oct 1, 2023
@jimklimov jimklimov deleted the issue-317 branch October 1, 2023 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug enhancement nut-scanner portability We want NUT to build and run everywhere possible Windows

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

nut-scanner: additional multiarch libraries search path

2 participants