fix: explicitly set dev mode for data store by ascorbic · Pull Request #12947 · withastro/astro

ascorbic · 2025-01-09T13:04:08Z

Changes

The content layer data store is created in a different location according to whether it is running in dev or build, so that it is either cachable or watchable by the fs watcher. These are mutually exclusive for Reasons.
This needs to be the same locaiton for both writing (during sync) and reading (during runtime). Unfortunately knowing which of these we're running is not as simple as it could be, and there have been some bugs that have emerged due to conflicts.

We'd mostly fixed these cases in #12640, but it missed the case where dev was run programatically with NODE_ENV set to something other than development (mostly in a test environment).

Rather than playing whackamole for all of these scenarios, this PR changes the approach so that it never uses NODE_ENV, and always explicitly sets it. Either this is done via the command used (build and sync use production, dev uses development) or at runtime uses Vite's env.command (build uses production, serve uses development). This involved some refactoring around how files are handled by data stores (which was overdue, because it was needlessly complex as it was created when there were different requirements)

Fixes #12652

Testing

Added some tests to the astro mode suite, which helpfully messes around with these options.

Docs

changeset-bot · 2025-01-09T13:04:14Z

🦋 Changeset detected

Latest commit: 54dff26

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

ascorbic · 2025-01-09T13:09:33Z

!preview data-store-location

github-actions · 2025-01-09T13:11:12Z

Snapshots have been released for the following packages:

astro@experimental--data-store-location

Publish Log

🦋  warn ===============================IMPORTANT!===============================
🦋  warn Packages will be released under the experimental--data-store-location tag
🦋  warn ----------------------------------------------------------------------
🦋  info npm info astro
🦋  info npm info @astrojs/prism
🦋  info npm info @astrojs/rss
🦋  info npm info create-astro
🦋  info npm info @astrojs/db
🦋  info npm info @astrojs/alpinejs
🦋  info npm info @astrojs/markdoc
🦋  info npm info @astrojs/mdx
🦋  info npm info @astrojs/partytown
🦋  info npm info @astrojs/preact
🦋  info npm info @astrojs/react
🦋  info npm info @astrojs/sitemap
🦋  info npm info @astrojs/solid-js
🦋  info npm info @astrojs/svelte
🦋  info npm info @astrojs/tailwind
🦋  info npm info @astrojs/vue
🦋  info npm info @astrojs/web-vitals
🦋  info npm info @astrojs/internal-helpers
🦋  info npm info @astrojs/markdown-remark
🦋  info npm info @astrojs/studio
🦋  info npm info @astrojs/telemetry
🦋  info npm info @astrojs/underscore-redirects
🦋  info npm info @astrojs/upgrade
🦋  info astro is being published because our local version (0.0.0-data-store-location-20250109131010) has not been published on npm
🦋  warn @astrojs/prism is not being published because version 3.2.0 is already published on npm
🦋  warn @astrojs/rss is not being published because version 4.0.11 is already published on npm
🦋  warn create-astro is not being published because version 4.11.0 is already published on npm
🦋  warn @astrojs/db is not being published because version 0.14.5 is already published on npm
🦋  warn @astrojs/alpinejs is not being published because version 0.4.1 is already published on npm
🦋  warn @astrojs/markdoc is not being published because version 0.12.4 is already published on npm
🦋  warn @astrojs/mdx is not being published because version 4.0.4 is already published on npm
🦋  warn @astrojs/partytown is not being published because version 2.1.3 is already published on npm
🦋  warn @astrojs/preact is not being published because version 4.0.1 is already published on npm
🦋  warn @astrojs/react is not being published because version 4.1.2 is already published on npm
🦋  warn @astrojs/sitemap is not being published because version 3.2.1 is already published on npm
🦋  warn @astrojs/solid-js is not being published because version 5.0.1 is already published on npm
🦋  warn @astrojs/svelte is not being published because version 7.0.3 is already published on npm
🦋  warn @astrojs/tailwind is not being published because version 5.1.4 is already published on npm
🦋  warn @astrojs/vue is not being published because version 5.0.4 is already published on npm
🦋  warn @astrojs/web-vitals is not being published because version 3.0.1 is already published on npm
🦋  warn @astrojs/internal-helpers is not being published because version 0.4.2 is already published on npm
🦋  warn @astrojs/markdown-remark is not being published because version 6.0.1 is already published on npm
🦋  warn @astrojs/studio is not being published because version 0.1.3 is already published on npm
🦋  warn @astrojs/telemetry is not being published because version 3.2.0 is already published on npm
🦋  warn @astrojs/underscore-redirects is not being published because version 0.6.0 is already published on npm
🦋  warn @astrojs/upgrade is not being published because version 0.4.2 is already published on npm
🦋  info Publishing "astro" at "0.0.0-data-store-location-20250109131010"
🦋  success packages published successfully:
🦋  astro@0.0.0-data-store-location-20250109131010
🦋  Creating git tag...
🦋  New tag:  astro@0.0.0-data-store-location-20250109131010

Build Log


> root@0.0.0 build /home/runner/work/astro/astro
> turbo run build --filter=astro --filter=create-astro --filter="@astrojs/*" --filter="@benchmark/*"

• Packages in scope: @astrojs/alpinejs, @astrojs/cloudflare, @astrojs/db, @astrojs/internal-helpers, @astrojs/markdoc, @astrojs/markdown-remark, @astrojs/mdx, @astrojs/netlify, @astrojs/node, @astrojs/partytown, @astrojs/preact, @astrojs/prism, @astrojs/react, @astrojs/rss, @astrojs/sitemap, @astrojs/solid-js, @astrojs/studio, @astrojs/svelte, @astrojs/tailwind, @astrojs/telemetry, @astrojs/underscore-redirects, @astrojs/upgrade, @astrojs/vercel, @astrojs/vue, @astrojs/web-vitals, @benchmark/adapter, @benchmark/timer, astro, create-astro
• Running build in 29 packages
• Remote caching enabled
::group::@astrojs/internal-helpers:build
cache miss, executing ae274b7b445371c0

> @astrojs/internal-helpers@0.4.2 build /home/runner/work/astro/astro/packages/internal-helpers
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json

::endgroup::
::group::@astrojs/telemetry:build
cache miss, executing 9d09dcc39d80ac33

> @astrojs/telemetry@3.2.0 build /home/runner/work/astro/astro/packages/telemetry
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/prism:build
cache miss, executing e91e490b6c914b6e

> @astrojs/prism@3.2.0 build /home/runner/work/astro/astro/packages/astro-prism
> astro-scripts build "src/**/*.ts" && tsc -p ./tsconfig.json

::endgroup::
::group::@astrojs/upgrade:build
cache miss, executing 6b2a09f1fa053060

> @astrojs/upgrade@0.4.2 build /home/runner/work/astro/astro/packages/upgrade
> astro-scripts build "src/index.ts" --bundle && tsc

::endgroup::
::group::create-astro:build
cache miss, executing 4aca443843bf97c9

> create-astro@4.11.0 build /home/runner/work/astro/astro/packages/create-astro
> astro-scripts build "src/index.ts" --bundle && tsc

::endgroup::
::group::@astrojs/markdown-remark:build
cache miss, executing e709ae68f8fa2a4d

> @astrojs/markdown-remark@6.0.1 build /home/runner/work/astro/astro/packages/markdown/remark
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json

::endgroup::
::group::astro:build
cache miss, executing 43b7376e7b86c63f

> astro@0.0.0-data-store-location-20250109131010 build /home/runner/work/astro/astro/packages/astro
> pnpm run prebuild && astro-scripts build "src/**/*.{ts,js}" --copy-wasm && tsc


> astro@0.0.0-data-store-location-20250109131010 prebuild /home/runner/work/astro/astro/packages/astro
> astro-scripts prebuild --to-string "src/runtime/server/astro-island.ts" "src/runtime/client/{idle,load,media,only,visible}.ts"

::endgroup::
::group::@astrojs/studio:build
cache miss, executing 95432f1ea6b89142

> @astrojs/studio@0.1.3 build /home/runner/work/astro/astro/packages/studio
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/underscore-redirects:build
cache miss, executing f0750805821c48a4

> @astrojs/underscore-redirects@0.6.0 build /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json

::endgroup::
::group::@astrojs/rss:build
cache miss, executing 384aac7a1e8a3ee8

> @astrojs/rss@4.0.11 build /home/runner/work/astro/astro/packages/astro-rss
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@benchmark/adapter:build
cache miss, executing b963e82de471ef6b

> @benchmark/adapter@0.0.0 build /home/runner/work/astro/astro/benchmark/packages/adapter
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/partytown:build
cache miss, executing e19da7900ba3b7b8

> @astrojs/partytown@2.1.3 build /home/runner/work/astro/astro/packages/integrations/partytown
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/svelte:build
cache miss, executing 942ba4271a9a39aa

> @astrojs/svelte@7.0.3 build /home/runner/work/astro/astro/packages/integrations/svelte
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/tailwind:build
cache miss, executing 9559ed6410a0bb8a

> @astrojs/tailwind@5.1.4 build /home/runner/work/astro/astro/packages/integrations/tailwind
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdoc:build
cache miss, executing cbbac28dbcf3b9f5

> @astrojs/markdoc@0.12.4 build /home/runner/work/astro/astro/packages/integrations/markdoc
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/solid-js:build
cache miss, executing 2336de55e6051602

> @astrojs/solid-js@5.0.1 build /home/runner/work/astro/astro/packages/integrations/solid
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vue:build
cache miss, executing 4c8c7651841888d0

> @astrojs/vue@5.0.4 build /home/runner/work/astro/astro/packages/integrations/vue
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@benchmark/timer:build
cache miss, executing 551d34d2738807a8

> @benchmark/timer@0.0.0 build /home/runner/work/astro/astro/benchmark/packages/timer
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/sitemap:build
cache miss, executing c5c3cc7a96aa42da

> @astrojs/sitemap@3.2.1 build /home/runner/work/astro/astro/packages/integrations/sitemap
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/alpinejs:build
cache miss, executing 2e9734d16f6ba276

> @astrojs/alpinejs@0.4.1 build /home/runner/work/astro/astro/packages/integrations/alpinejs
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/react:build
cache miss, executing 06e002717241aefd

> @astrojs/react@4.1.2 build /home/runner/work/astro/astro/packages/integrations/react
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/mdx:build
cache miss, executing 940723abe835222c

> @astrojs/mdx@4.0.4 build /home/runner/work/astro/astro/packages/integrations/mdx
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/preact:build
cache miss, executing 19fa9ca50269ccfc

> @astrojs/preact@4.0.1 build /home/runner/work/astro/astro/packages/integrations/preact
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/db:build
cache miss, executing f7945b547993cfaa

> @astrojs/db@0.14.5 build /home/runner/work/astro/astro/packages/db
> astro-scripts build "src/**/*.ts" && tsc && pnpm types:virtual


> @astrojs/db@0.14.5 types:virtual /home/runner/work/astro/astro/packages/db
> tsc -p ./tsconfig.virtual.json

::endgroup::
::group::@astrojs/web-vitals:build
cache miss, executing 6304dc26f8b554b3

> @astrojs/web-vitals@3.0.1 build /home/runner/work/astro/astro/packages/integrations/web-vitals
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::

 Tasks:    25 successful, 25 total
Cached:    0 cached, 25 total
  Time:    45.628s

codspeed-hq · 2025-01-09T13:17:53Z

CodSpeed Performance Report

Merging #12947 will not alter performance

_{Comparing explicit-datastore (cd8ff83) with main (35d77b4)}

Summary

✅ 6 untouched benchmarks

delucis · 2025-01-09T13:51:29Z

I tested the experimental release for this PR against Starlight’s end-to-end tests (which fail with Astro 5.1.x currently) and it seems to resolve things for us: withastro/starlight#2776

ematipico · 2025-01-09T14:06:52Z

+			const dataStoreFile = getDataStoreFile(settings, isDev);
+			store = await MutableDataStore.fromFile(dataStoreFile);


existSync is now marked unused

ematipico · 2025-01-09T14:07:45Z


 	describe('build --mode testing --devOutput', () => {
 		before(async () => {
+			await deleteDataStoreFiles();


Any particular reason why we use this in before and not in after?

We're checking the creation of the files, so I wanted to be explicit in deleting any leftover ones before running the build.

fix: explicitly set dev mode for data store

efe3fac

github-actions Bot added the pkg: astro Related to the core `astro` package (scope) label Jan 9, 2025

Error handling

0383fce

ascorbic mentioned this pull request Jan 9, 2025

Update minimum supported Astro version to 5.1 withastro/starlight#2736

Merged

delucis mentioned this pull request Jan 9, 2025

Test Astro 5.1 experimental-release withastro/starlight#2776

Closed

ematipico approved these changes Jan 9, 2025

View reviewed changes

Add check for file before saving

8e56262

ascorbic force-pushed the explicit-datastore branch from 0e66162 to 8e56262 Compare January 9, 2025 14:57

rm unused import

c206eab

bluwy reviewed Jan 9, 2025

View reviewed changes

Comment thread packages/astro/src/core/sync/index.ts Outdated

bluwy reviewed Jan 9, 2025

View reviewed changes

Comment thread packages/astro/test/fixtures/content-layer/src/pages/missing.astro

ascorbic added 2 commits January 9, 2025 15:21

Use explicit command

54dff26

Ensure dir before writing

cd8ff83

ascorbic merged commit 3c2292f into main Jan 9, 2025

ascorbic deleted the explicit-datastore branch January 9, 2025 16:10

astrobot-houston mentioned this pull request Jan 9, 2025

[ci] release #12945

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: explicitly set dev mode for data store#12947

fix: explicitly set dev mode for data store#12947
ascorbic merged 6 commits into
mainfrom
explicit-datastore

ascorbic commented Jan 9, 2025

Uh oh!

changeset-bot Bot commented Jan 9, 2025 •

edited

Loading

Uh oh!

ascorbic commented Jan 9, 2025

Uh oh!

github-actions Bot commented Jan 9, 2025

Uh oh!

codspeed-hq Bot commented Jan 9, 2025 •

edited

Loading

Uh oh!

delucis commented Jan 9, 2025

Uh oh!

ematipico Jan 9, 2025

Uh oh!

ematipico Jan 9, 2025

Uh oh!

ascorbic Jan 9, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		const dataStoreFile = getDataStoreFile(settings, isDev);
		store = await MutableDataStore.fromFile(dataStoreFile);

Uh oh!

Conversation

ascorbic commented Jan 9, 2025

Changes

Testing

Docs

Uh oh!

changeset-bot Bot commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

ascorbic commented Jan 9, 2025

Uh oh!

github-actions Bot commented Jan 9, 2025

Uh oh!

codspeed-hq Bot commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #12947 will not alter performance

Summary

Uh oh!

delucis commented Jan 9, 2025

Uh oh!

ematipico Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

ematipico Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

ascorbic Jan 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changeset-bot Bot commented Jan 9, 2025 •

edited

Loading

codspeed-hq Bot commented Jan 9, 2025 •

edited

Loading