From fcf16b56588ba11f3b5f9233f5489a49aa3bc2c5 Mon Sep 17 00:00:00 2001 From: William A Rowe Jr Date: Tue, 23 Jun 2020 09:58:10 -0500 Subject: [PATCH 1/6] The beginnings of a PORTING.md project doc for contributors - Soliciting contributions to build on this after it is merged to master from both Microsoft contributors and OS/X champions. - It is also worthwhile to solicit more contributions to build on these notes from the Apple iOS and Android porting efforts. Signed-off-by: Sunjay Bhatia Co-authored-by: Sunjay Bhatia Signed-off-by: William A Rowe Jr Co-authored-by: William A Rowe Jr --- DEVELOPER.md | 2 + PORTING.md | 128 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 130 insertions(+) create mode 100644 PORTING.md diff --git a/DEVELOPER.md b/DEVELOPER.md index 465644c0e02ce..867cc76dfdff2 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -30,6 +30,8 @@ Below is a list of additional documentation to aid the development process: - [Performance testing Envoy with `tcmalloc`/`pprof`](https://github.com/envoyproxy/envoy/blob/master/bazel/PPROF.md) +- [Porting Envoy and common porting issues](https://github.com/envoyproxy/envoy/blob/master/bazel/PORTING.md) + And some documents on components of Envoy architecture: - [Envoy flow control](https://github.com/envoyproxy/envoy/blob/master/source/docs/flow_control.md) diff --git a/PORTING.md b/PORTING.md new file mode 100644 index 0000000000000..923f5bba6f955 --- /dev/null +++ b/PORTING.md @@ -0,0 +1,128 @@ +## Envoy Porting + +* Linux (the original/baseline implementation) +* Mac OS/X port +* Windows Win32 API port + +## Troubleshooting Common Porting Issues + +Between the use of gcc, clang, the optional use of libstdc++ vs different flavors of libc++, +and other platform specific compilers including SDK compilers for mobile, there are a number +of disagreements over what constitutes "correct" stdc++14 (the current baseline.) For a very +thorough reference, see; + +* [Compiler stdc++ feature support chart](https://en.cppreference.com/w/cpp/compiler_support) + +It is often useful to comparatively compile code fragments, and most developers don't have access +to a wide array of different compilers at hand. This tool helps porters evaluate how different +compilers (and even different revisions of the same compiler) compile the offending code +down to the assembly level; + +* [Compiler Explorer](https://godbolt.org/) + +Note that each port has its own development channel on envoyproxy slack. For example there +are champions at `#envoy-windows-dev`, `#envoy-osx-dev` etc. who are happy to answer quick +questions and support contributors who are encountering architecture-specific CI failures. +There are also tags for issues and PR's such as `area/windows` which can help to raise +issues to specific maintainers of these ports. + +### General Porting Issues with respect to Microsoft Visual C++ (cl.exe) + +The LLVM clang compiler mirrors many of the gcc behaviors and quirks, resulting in two very +similar compilation results. This is not true of MSVC cl.exe, which has a number of unique quirks, +and deviations from non-standard clang/gcc quirks. Here are some links documenting these deviations +from ISO stdc++; + +* [MSVC non-standard behaviors](https://docs.microsoft.com/en-us/cpp/cpp/nonstandard-behavior?view=vs-2019) + +Note that Envoy project has elected to distribute a monolithic envoy-static.exe including +all components compiled into the single executable. This includes the libcmt (C Runtime) +and cpp libraries; all code is compiled with /MT (or /MTd for debug runtime). Do not inject +the /nodefaultlib flag to the linker; this masks errors in the compilation phase and also +mixes dynamic and static runtime implementation macros in the resulting binary. + +## Specific Coding Issues + +Most porting issues can be summarized in a handful of assumptions to be avoided. + +The size of an 'int' or 'long' integer on Windows is 32 bits, while pointers are 64 bits +(we refer to this as a 64P architecture.) On Linux, OS/X and other *nix variants, the 'int', +'long' and pointer types are all 64 bits (64ILP architectures). +This can prove frustrating to contributors, as the Envoy project prefers explicit-size types +such as int64_t, uint32_t etc. but most coders default to using 'int' and similar in loops +and other local variables. + +The Standard C++ library interfaces must conform to the actual size of the platform's 'int', +'long', 'ptrdiff_t' and similarly declared types. The use of 'auto' is often helpful but cannot +resolve this transition between C++ definitions and Envoy explicit-sized types. + +The Windows MSVC compiler began but never completed a proper POSIX implementation, and this +has not been enhanced in over two decades. Many commonly used POSIX data types and functions +aren't available, or differ in name or implementation details. A prime example is the absence +of the POSIX, non-ISO/C++ type 'ssize_t' on Windows. The ISO/C++ 'ptrdiff_t' type is available, +and part of the language standard. Use 'ptrdiff_t' in place of 'ssize_t', and do not presume +that this value fits in a 32-bit `int`. + +The Microsoft compiler performs K&R preprocessor line evaluation, so it is not possible to +embed `#if`/`#endif` and other preprocessor evaluation within the argument list of a macro +expression. + +The Microsoft compiler has a limit of 2k characters in any single literal string expression +such as R"" multi-line expressions. However, this applies only to each individual token, so +string concatenation needs to be used to merge multiple string expressions into very large +string constants. + +Alternative operator representations such as 'not', 'and', 'compl' etc. which are alternatives +to '!', '&&', '~' etc. which had been ISO standard workarounds for very limited character sets and +locale-specific keyboards are still supported by gcc and clang, but were never supported by the +Microsoft compiler and are now deprecated by the language spec. + +Whenever platform-specific header files are needed, include the envoy/common/platform.h file to pick +up the most common inclusions. This file makes a number of adjustments to define common constructs +that may be missing on a particular environment. This file also adjusts for system file misbehavior. +(For example, the Windows.h header file #define's many common words that break even the simplest C++ +source code; platform.h reverts some of this damage). + +The portable implementations of FilesystemImpl, IoSocketImpl and low-level OsSysCallsImpl using +os_fd_t descriptors and the error constants defined in `platform.h` should be used in favor of raw +POSIX APIs when authoring tests. + +## Dependency behavioral differences + +The libraries that Envoy depends on will exhibit different behaviors when built for Windows +versus Linux versus BSD architectures. In some cases, Envoy has patched the library to either +correct misbehavior, or to make the library follow expected linux conventions more closely, +and every attempt is made to push such changes as optional features to upstream. Some notable +discrepancies include; + +The googletest library has a number of Windows-specific quirks. At the time of this writing, +this library is incompatible with the LLVM clang-cl. It uses the POSIX re (regular expression) +system library by default. There is an option to use the PCRE library instead, which the +project has not adopted. On Windows, it uses a massively simplified expression parser which +is missing support for most common Re expressions. (The solution to this, substituting the +Google Re library, is proposed but has not been adopted by the googletest project yet). + +The opentracing library is missing a Windows build schema (it relies entirely on automake), +so all tracing-related Envoy extensions are disabled on Windows. + +## Bazel behavior differences + +There are several circumstances that bazel will end up misbehaving in one environment or another. +For example, during RBE CI builds, the environment, paths, mounts etc. may have longer paths or +otherwise hit limits that aren't observed on the contributor's environment. + +Similarly, when building on Windows the tools themselves have different path or command line length +restrictions than on the typical Linux/clang tool chain. These limits include; + +* The individual path elements for -Inclusion (this restricts the total path name lengths available to the project). +* The length of the embedded command line stored in the debug info record of the COFF object file (48kb on Windows). + +Note that the rules_foreign_cc macros for bazel helps to ensure compilation is consistent across +architectures. For the greatest portability, we rely on native bazel BUILD rules, or CMakeLists.txt +input to CMake which is handled somewhat painlessly by rules_foreign_cc. + +Many tests rely on command line scripting or tool invocation. Bazel and Envoy rely heavily on bash scripts, executing +on Windows via msys2. Inherent discrepancies between the msys2 execution environment and a typical bash shell can +cause confusing errors. Be sure to use cmd.exe on Windows in any test scripts that intend to create symlinks. + + From fce74d2356b64e1eaede453f40514cc70628984b Mon Sep 17 00:00:00 2001 From: William A Rowe Jr Date: Thu, 25 Jun 2020 00:54:36 -0500 Subject: [PATCH 2/6] Silly commit to cooerce silly CI to rebuild Signed-off-by: William A Rowe Jr --- DEVELOPER.md | 1 - PORTING.md | 2 -- 2 files changed, 3 deletions(-) diff --git a/DEVELOPER.md b/DEVELOPER.md index 867cc76dfdff2..2fc0e5687188c 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -37,4 +37,3 @@ And some documents on components of Envoy architecture: - [Envoy flow control](https://github.com/envoyproxy/envoy/blob/master/source/docs/flow_control.md) - [Envoy's subset load balancer](https://github.com/envoyproxy/envoy/blob/master/source/docs/subset_load_balancer.md) - diff --git a/PORTING.md b/PORTING.md index 923f5bba6f955..662f0b1e218eb 100644 --- a/PORTING.md +++ b/PORTING.md @@ -124,5 +124,3 @@ input to CMake which is handled somewhat painlessly by rules_foreign_cc. Many tests rely on command line scripting or tool invocation. Bazel and Envoy rely heavily on bash scripts, executing on Windows via msys2. Inherent discrepancies between the msys2 execution environment and a typical bash shell can cause confusing errors. Be sure to use cmd.exe on Windows in any test scripts that intend to create symlinks. - - From 5467b1e0ef56c82522fe14f85726c171c0317b90 Mon Sep 17 00:00:00 2001 From: William A Rowe Jr Date: Tue, 30 Jun 2020 11:10:54 -0400 Subject: [PATCH 3/6] Note MSVC conformance history URL Signed-off-by: Sunjay Bhatia Co-authored-by: Sunjay Bhatia Signed-off-by: William A Rowe Jr Co-authored-by: William A Rowe Jr --- PORTING.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/PORTING.md b/PORTING.md index 662f0b1e218eb..f60e4977bd1d4 100644 --- a/PORTING.md +++ b/PORTING.md @@ -35,6 +35,11 @@ from ISO stdc++; * [MSVC non-standard behaviors](https://docs.microsoft.com/en-us/cpp/cpp/nonstandard-behavior?view=vs-2019) +Microsoft documents their conformance fixes by Virtual Studio toolchain revision (the CI attempts +to keep up with the most recent release); + +* [MSVC conformance improvements](https://docs.microsoft.com/en-us/cpp/overview/cpp-conformance-improvements?view=vs-2019) + Note that Envoy project has elected to distribute a monolithic envoy-static.exe including all components compiled into the single executable. This includes the libcmt (C Runtime) and cpp libraries; all code is compiled with /MT (or /MTd for debug runtime). Do not inject From 93c4d8d1b33694cb0854ab95abc9cd380009c466 Mon Sep 17 00:00:00 2001 From: William A Rowe Jr Date: Tue, 14 Jul 2020 09:46:08 -0500 Subject: [PATCH 4/6] Correct DEVELOPER.md link to live at envoy root Signed-off-by: William A Rowe Jr --- DEVELOPER.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPER.md b/DEVELOPER.md index 2fc0e5687188c..074ee5457748d 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -30,7 +30,7 @@ Below is a list of additional documentation to aid the development process: - [Performance testing Envoy with `tcmalloc`/`pprof`](https://github.com/envoyproxy/envoy/blob/master/bazel/PPROF.md) -- [Porting Envoy and common porting issues](https://github.com/envoyproxy/envoy/blob/master/bazel/PORTING.md) +- [Porting Envoy and common porting issues](https://github.com/envoyproxy/envoy/blob/master/PORTING.md) And some documents on components of Envoy architecture: From f886669accc94ee9bd01ceb0e3101591aa0d7dda Mon Sep 17 00:00:00 2001 From: William A Rowe Jr Date: Tue, 21 Jul 2020 10:44:03 -0500 Subject: [PATCH 5/6] Decorate as link Signed-off-by: William A Rowe Jr --- PORTING.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/PORTING.md b/PORTING.md index 662f0b1e218eb..5b5e912172796 100644 --- a/PORTING.md +++ b/PORTING.md @@ -1,4 +1,4 @@ -## Envoy Porting +# Envoy Porting * Linux (the original/baseline implementation) * Mac OS/X port @@ -20,11 +20,11 @@ down to the assembly level; * [Compiler Explorer](https://godbolt.org/) -Note that each port has its own development channel on envoyproxy slack. For example there -are champions at `#envoy-windows-dev`, `#envoy-osx-dev` etc. who are happy to answer quick -questions and support contributors who are encountering architecture-specific CI failures. -There are also tags for issues and PR's such as `area/windows` which can help to raise -issues to specific maintainers of these ports. +Note that each port has its own development channel on (envoyproxy slack)[https://envoyproxy.slack.com/]. +For example there are champions at `#envoy-windows-dev`, `#envoy-osx-dev` etc. who are happy to answer +quick questions and support contributors who are encountering architecture-specific CI failures. There are +also tags for issues and PR's such as `area/windows` which can help to raise issues to specific maintainers +of these ports. ### General Porting Issues with respect to Microsoft Visual C++ (cl.exe) From d3fc8355e4ff53959341fc264a6f3b5d4cf6f28d Mon Sep 17 00:00:00 2001 From: William A Rowe Jr Date: Tue, 21 Jul 2020 16:05:08 -0400 Subject: [PATCH 6/6] Fix at lizan's suggestion for macOS's own representation Signed-off-by: William A Rowe Jr --- PORTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PORTING.md b/PORTING.md index 4fcf6223b61f7..9ec1cde732769 100644 --- a/PORTING.md +++ b/PORTING.md @@ -1,7 +1,7 @@ # Envoy Porting * Linux (the original/baseline implementation) -* Mac OS/X port +* macOS port * Windows Win32 API port ## Troubleshooting Common Porting Issues