Add some troubleshooting notes for the development environment by fpacifici · Pull Request #170 · getsentry/streams

fpacifici · 2025-06-27T23:57:13Z

We had a number of issues in managing the development environment.
This collects all the learnings

untitaker

i recommend against documenting N workarounds for things that should already just work. if we document it, people will work around the broken automation instead of raising those issues and fixing them.

untitaker · 2025-06-30T12:56:07Z

sentry_streams/docs/source/devenv.rst

+  a. MacOs note: There are multiple ways to install the python environment on MacOS.
+     We tested successfully the installation via `brew`. Other ways, like `pyenv` and
+     `uv`, should work as well but we saw incompatibilities with the way `maturin` expects
+     environment variables to be set. See below for more details if you want to use
+     `pyenv` or `uv` to install python.


this should be handled by the envrc and direnv allow. if 3.11 does not exist uv will download it. if that leads to problems let's treat it like any other bug

if that leads to problems let's treat it like any other bug

We are treating it as any other bug. https://linear.app/getsentry/issue/STREAM-284/automate-all-the-validation-step-in-the-dev-env-guide . Still till that is not fixed people should at least know a way to make it work otherwise nobody would try to fix the issue.

untitaker · 2025-06-30T12:56:18Z

sentry_streams/docs/source/devenv.rst

+     environment variables to be set. See below for more details if you want to use
+     `pyenv` or `uv` to install python.
+
+4. Run `make install-dev` from the root of the repo. This will instal `uv` and build


this is done by direnv allow

untitaker · 2025-06-30T12:56:43Z

sentry_streams/docs/source/devenv.rst

+Some environment variables are needed to successfully build the library. These are set by direnv.
+They are explained here to give a better understanding in case of issues.
+
+- `CMAKE_POLICY_VERSION_MINIMUM=3.5`` - This is needed to build librdkafka on
+  newer versions of cmake
+
+Needed environment variables to allow Rust to start a Python interpreter properly:
+
+- `PYTHONPATH=.` - This is needed in order to make the Python interpreter started by
+  Rust find the virtual environment.
+
+- `PYO3_PYTHON` - This is needed to make the Python interpreter find the right `site-packages`
+  directory when started by the Rust code. See `rust-envvars <https://github.com/getsentry/streams/blob/main/scripts/rust-envvars>`_


all of this is set by the envrc

Yes, this paragraph is not part of the happy path. You should only land here if something went wrong. If something goes wrong knowing what variables are needed for what is needed to allow people to try to fix the issue.

if this just "neutrally" documents what the envrc does, shouldn't it be possible to document it next to the code that sets those envvars, i.e. as code comments?

That makes sense to me. I'd move the explanation in the code and reference that from here.

fpacifici · 2025-06-30T18:44:01Z

i recommend against documenting N workarounds for things that should already just work

We have a problem to solve. Multiple people needed hours of pairing to get their environment working. The absence of documentation explaining how things worked did not prompt people to fix them but to skip tests. Obviously this is not acceptable.
Would you agree that we need both the incentive to fix the environment and enough shared knowledge to understand what is not working so people know where to start from to fix the environment?

The first (incentive to fix) is both tracked https://linear.app/getsentry/issue/STREAM-284/automate-all-the-validation-step-in-the-dev-env-guide. I doubt we can expect people to fix the environment if the way it works is not clear.

untitaker · 2025-06-30T18:46:34Z

We have a problem to solve. Multiple people needed hours of pairing to get their environment working.

and now it does, right? i mean most of the envvars you are suggesting in this guide are already set by envrc. i don't think we should suggest to set them explicitly. if they are missing, it's an indication that their direnv setup is broken, and this is the part that should be fixed.

The absence of documentation explaining how things worked did not prompt people to fix them but to skip tests. Obviously this is not acceptable.

my concern is that we end up with dev environments that drift further away from the golden path. because everybody applied manual fixes on top.

untitaker

don't want to block this too much. maybe this though: instead of "set this envvar" i'd like to rather suggest "figure out why this envvar isn't being set"

untitaker · 2025-06-30T18:48:24Z

sentry_streams/docs/source/devenv.rst

+
+Needed environment variables to allow Rust to start a Python interpreter properly:
+
+- `PYTHONPATH=.` - This is needed in order to make the Python interpreter started by


this envvar is already set (STREAMS_TEST_PYTHONPATH)

specifically overwriting this envvar like this will fix one bug while re-introducing another. i think if PYTHONPATH needs to be set for cargo test, this points to an issue with direnv. otherwise it's a new bug to be investigated and should be discussed.

also since we now overwrite sys.path within the rust process i'm no longer sure PYTHONPATH has any effect at all

untitaker · 2025-06-30T18:48:32Z

sentry_streams/docs/source/devenv.rst

+Some environment variables are needed to successfully build the library. These are set by direnv.
+They are explained here to give a better understanding in case of issues.
+
+- `CMAKE_POLICY_VERSION_MINIMUM=3.5`` - This is needed to build librdkafka on


this is also definitely set by envrc.

fpacifici · 2025-06-30T18:52:52Z

i don't think we should suggest to set them explicitly. if they are missing, it's an indication that their direnv setup is broken, and this is the part that should be fixed.

Suggesting to set environment variables manually was not my intent. The happy path is only this https://github.com/getsentry/streams/pull/170/files#diff-a5fb029817e5f0c5efdcb0de0545265c5c80aefe309baab2141c676f9c26849eR32-R56.
Fair: I need to remove the reference to make install-dev. Will do that.

The rest is all about how to understand the process when things go wrong.

I'd be happy to clarify better what is the happy path vs what is troubleshooting. What if I explained that explicitly at the beginning of this section https://github.com/getsentry/streams/pull/170/files#diff-a5fb029817e5f0c5efdcb0de0545265c5c80aefe309baab2141c676f9c26849eR32

untitaker · 2025-06-30T18:56:37Z

sentry_streams/docs/source/devenv.rst

+Some environment variables are needed to successfully build the library. These are set by direnv.
+They are explained here to give a better understanding in case of issues.
+
+- `CMAKE_POLICY_VERSION_MINIMUM=3.5`` - This is needed to build librdkafka on
+  newer versions of cmake
+
+Needed environment variables to allow Rust to start a Python interpreter properly:
+
+- `PYTHONPATH=.` - This is needed in order to make the Python interpreter started by
+  Rust find the virtual environment.
+
+- `PYO3_PYTHON` - This is needed to make the Python interpreter find the right `site-packages`
+  directory when started by the Rust code. See `rust-envvars <https://github.com/getsentry/streams/blob/main/scripts/rust-envvars>`_


if this just "neutrally" documents what the envrc does, shouldn't it be possible to document it next to the code that sets those envvars, i.e. as code comments?

evanh

I think this is great!

evanh · 2025-07-03T13:54:00Z

sentry_streams/docs/source/devenv.rst

+     environment variables to be set. See below for more details if you want to use
+     `pyenv` or `uv` to install python.
+
+4. Run `make install-dev` from the root of the repo. This will instal `uv` and build


Suggested change

4. Run `make install-dev` from the root of the repo. This will instal `uv` and build

4. Run `make install-dev` from the root of the repo. This will install `uv` and build

fpacifici added 2 commits June 27, 2025 09:12

Document devenv

62cf720

Provide dev env troubleshooting guide

0bfb1de

fpacifici requested a review from a team as a code owner June 27, 2025 23:57

Merge branch 'main' into fpacifici/doc_env

e364dde

untitaker reviewed Jun 30, 2025

View reviewed changes

untitaker approved these changes Jun 30, 2025

View reviewed changes

evanh approved these changes Jul 3, 2025

View reviewed changes


		Needed environment variables to allow Rust to start a Python interpreter properly:

		- `PYTHONPATH=.` - This is needed in order to make the Python interpreter started by

	4. Run `make install-dev` from the root of the repo. This will instal `uv` and build
	4. Run `make install-dev` from the root of the repo. This will install `uv` and build

Uh oh!

Conversation

fpacifici commented Jun 27, 2025

Uh oh!

untitaker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fpacifici commented Jun 30, 2025

Uh oh!

untitaker commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

untitaker left a comment

Choose a reason for hiding this comment

Uh oh!

untitaker Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fpacifici commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evanh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

untitaker commented Jun 30, 2025 •

edited

Loading

untitaker Jun 30, 2025 •

edited

Loading

fpacifici commented Jun 30, 2025 •

edited

Loading