evals: add agent-041-draft-mode-use-cache#9
Draft
gaojude wants to merge 1 commit into
Draft
Conversation
Frames PR vercel#92794 (docs: draftMode alignment) as an agent eval. Tests whether a coding agent correctly wires draftMode() inside a "use cache" scope: - Uses "use cache" directive for caching - Reads draftMode().isEnabled inside the cache boundary (async) - Conditionally fetches draft vs published content - Does NOT call enable()/disable() inside cache scope - Does NOT call cookies()/headers() inside cache scope - Enables useCache or cacheComponents in next.config - Avoids deprecated caching patterns
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds a new agent eval (
agent-041-draft-mode-use-cache) that tests whether a coding agent correctly wiresdraftMode()inside a"use cache"scope — the behavior documented in PR vercel#92794.Why?
PR vercel#92794 adds documentation about how Draft Mode interacts with cache components. Agents trained on older Next.js patterns may not know that:
draftMode().isEnabledis readable inside"use cache"scopescookies()andheaders()are not allowed inside"use cache"(even when draft mode is active)enable()/disable()cannot be called inside cache scopesThis eval validates that agents reach for the correct modern pattern.
How?
Fixture: A CMS blog app with a
/blog/[slug]page that fetches posts viagetPost(slug)fromlib/cms. The starting state has no caching and no draft mode wiring — the agent must add both.EVAL.ts (9 assertions):
"use cache"directive is presentdraftMode().isEnabledis read inside the cache boundarydraftMode()is properlyawaited (Next.js 16 async API)getPostcall is conditional on draft mode statusenable()/disable()inside cache scopecookies()/headers()inside cache scopenext.configenablesuseCacheorcacheComponentsunstable_cache,fetch revalidate,dynamicIO)enable/disableVerified: 9/9 pass against a golden solution, 5/9 fail against the starting fixture.
Slack Thread