Improve build-test loop for AI agents by mokagio · Pull Request #25242 · wordpress-mobile/WordPress-iOS

mokagio · 2026-02-11T10:01:02Z

Summary

Refines CLAUDE.md and the project's test infrastructure so AI agents can build, lint, and run focused tests autonomously.

Add build & test instructions to CLAUDE.md (Fastlane test lane, xcodebuild build, rake lint, only_testing: syntax, available test targets)
Document source-to-test-target naming convention and the WordPressUIUnitTests exception
Register all module test targets in the default test plan (WordPressUnitTests.xctestplan) so agents can select any of them for focused runs
Remove the redundant AllModulesUnitTests test plan (subset of the default plan)
Add a local-only test Fastlane lane that skips CI prerequisites and reuses DerivedData/
Fix DesignSystem resource bundle resolution in the test runner (Bundle.designSystemBundle instead of .module)
Fix a timezone-sensitive assertion in MockStatsServiceTests
Move an orphan test file into the correct WordPressUIUnitTests target
Clean up unused rake tasks and overly broad Claude Code allow-list entries

Note

I haven't set up Xcode 26.3 yet, which promises to be faster than using Fastlane/xcodebuild. Still, I don't think it hurts to have this infra available even if it won't be used much on dev machines. It'd be interesting to see what can be done in CI without spinning up Xcode.

Testing instructions

No user-facing changes.
Verify the test plan and scheme changes work by running:

bundle exec fastlane test only_testing:DesignSystemTests
bundle exec fastlane test only_testing:WordPressCoreTests

Generated with Claude Opus 4.6

wpmobilebot · 2026-02-11T10:12:55Z

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Configuration	Release-Alpha
	Build Number	`30987`
	Version	`PR #25242`
	Bundle ID	`org.wordpress.alpha`
	Commit	`7cfc3ca`
	Installation URL	7hr8vm5s8a7kg

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

wpmobilebot · 2026-02-11T10:13:13Z

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Configuration	Release-Alpha
	Build Number	`30987`
	Version	`PR #25242`
	Bundle ID	`com.jetpack.alpha`
	Commit	`7cfc3ca`
	Installation URL	2b42rm39t477g

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

wpmobilebot · 2026-02-11T10:57:11Z

🤖 Build Failure Analysis

This build has failures. Claude has analyzed them - check the build annotations for details.

.claude/settings.json

mokagio · 2026-02-12T04:49:01Z

CLAUDE.md

+
+### Building and Testing
+
+Use the `test` Fastlane lane for local testing.


This might change soon with Xcode 26.3, I haven't had a chance to install the RC yet.

mokagio · 2026-02-12T04:50:26Z

Rakefile

-desc "Build #{XCODE_SCHEME}"
-task build: [:dependencies] do
-  xcodebuild(:build)
-end


Given we have Fastlane and/or the upcoming Xcode 26.3 MCP, these are redundant and confusing for the agent to have.

They were the first it picked up when I tried and wasted a bunch of time on them.

mokagio · 2026-02-12T09:24:27Z

@kean @crazytonyli sorry for the spam. This is not reayd for review yet. I mean, I'm always appreciative of your feedback, but I'm not done yet.

crazytonyli · 2026-02-13T05:21:27Z

fastlane/lanes/build.rb

+  #   bundle exec fastlane test clean:true
+  #
+  desc 'Run tests locally'
+  lane :test do |scheme: 'WordPress', device: 'iPhone 16', ios_version: nil, only_testing: nil, clean: false|


Using 'iPhone 17' is probably more inline with the Xcode version used in the project

crazytonyli · 2026-02-13T05:36:07Z

CLAUDE.md

+1. **Lint** — `rake lint` (fast, catches style and localization errors)
+2. **Build** — `xcodebuild build ...` (catches type errors without running tests)
+3. **Test** — `bundle exec fastlane test only_testing:TargetName/Class/method`
+


I don't think we need to be so prescriptive. LLM is already smart enough to know most of these: the xcodebuild build, the simctl command, and the test filter format. They may not know what rake tasks and fastlane lanes we have, so I guess it might be valuable to include them explicitly.

I think most of the time, LLM is smart enough to know which tools to use and how to use them during the job, which usually works out better than telling them what to do (via Claude.md in this case) before the job starts. In general, I'm leaning more towards cutting down the context, rather than adding more.

Please note, I'm not really against adding these instructions here. We can see how they work in practice after this PR is merged.

I don't like being prescriptive either, but every time I ask the model to generate this kind of stuff, it always opts for being verbose and prescriptive.

I don't like it, because it seems like a maintenance nightmare.

In fact, I had to push back on it enumerating all the module-tests pairs, and force it to track the naming convention instead.

I asked why and the reason it gave, which I know is not to be trusted but made sense to me, is that being prescriptive means it has less files to parse when needing to find the answer to which command to run for what.

crazytonyli · 2026-02-13T05:36:40Z

Modules/Sources/DesignSystem/Foundation/Bundle+DesignSystem.swift

+import Foundation
+
+extension Bundle {
+    public class var designSystemBundle: Bundle {


This function probably does not need to be public?

CI uses Fastlane exclusively; these local wrappers around `xcodebuild` were obsolete and hardcoded a stale simulator. --- Generated with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Lets agents run lint, build, test, and simulator queries without prompting. Other A8C repos (e.g. woocommerce.com) use the same pattern. --- Generated with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Skips CI prerequisites (env files, signing, toolkit checks) and reuses DerivedData for incremental builds. Supports only_testing, scheme, device, and clean options. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Ideally, this would build faster than the rest and allow focused testing. The blocker is that since the Modules don't build with `swift build` we can test with `swift test`. Plus, auto-generated schemes allow building but not testing!

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The file was in a `WordPressUITests` directory with no matching SPM target, so it was never compiled or run. Updated its import from `WordPress` to `WordPressUI`. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Applies the same test-bundle workaround already used elsewhere in WordPressUI so the storyboard loads in SPM test context. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The test calendar used `.eastern` but the service used `.current`, so date interval keys never aligned. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The test crashes when DesignSystem is built as a dynamic library, which happens because WordPressAuthenticator (also in the plan) depends on it transitively. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use `Bundle.designSystemBundle` instead of `.module` so resources resolve when Xcode builds the module as a dynamic library (same workaround as WordPressUI). --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Its targets are a subset of WordPressUnitTests.xctestplan, which is already the default test plan in the WordPress scheme. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

sonarqubecloud · 2026-02-19T05:32:24Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kean · 2026-02-19T21:15:17Z

Hey Gio, thanks for the PR!

I ran a few smoke tests. I wasn't completely sure what to expect, so I tried different types of targets.

Test 1

Try and test only WordPressData (Xcode target)

❯ Test WordPressData
bundle exec fastlane test only_testing:WordPressDataTests 2>&1

Took a lot of time to update Swift packages.

Expected Result

Should only compile and test WordPressData and take a couple of minutes.

Observed Result

Took about 6 minutes, seemingly compiling the app:

 [16:02:10]: ▸ [WordPress] Compiling DeleteSite.storyboard
     [16:02:10]: ▸ [WordPress] Compiling BloggingPromptsViewController.storyboard
     [16:02:10]: ▸ [WordPress] Compiling LaunchScreen.storyboard

Failed with an error:

It tried and failed again:

Test 2

Run tests for WordPressShared (SwiftPM module)

❯ Run tests for WordPressShared

⏺ Bash(bundle exec fastlane test only_testing:WordPressSharedTests)
  ⎿  Running…

 Bash command

   bundle exec fastlane test only_testing:WordPressSharedTests
   Run WordPressSharedTests via Fastlane

Observed Result

Had to re-resolve the package graph again and took a lot of time.

Test 3

Make a small change to WordPressShared that would result in an error.

Observed Result

Took about two minutes again. Based on the logs, it built WordPress.app and other unrelated modules. Failed with the following error:

It looks like it can't detect what the error is from the output.

Test 4

Observed Result

One of the tests in WordPressSharedTests was still supposed to fail.

It ran slow Intelligence module tests that are not designed to run on every change. Each tests takes around 5 seconds.

[16:19:28]: ▸     ⚠️  Test excerptGenerationNonEnglish(parameters:) recorded an issue with 1 argument(s)
[16:19:36]: ▸     ⚠️  Test excerptGenerationNonEnglish(parameters:) recorded an issue with 1 argument(s)
 [16:19:40]: ▸     ⚠️  Test excerptGenerationNonEnglish(parameters:) recorded an issue with 1 argument(s)

It nearly hit the 10 minutes timeout.

It was able to tell what tests failed, but I'm not sure if there are all actual failures.

I haven't set up Xcode 26.3 yet

It might be worth optimizing for 26.3 from the start.

kean · 2026-02-20T13:58:15Z

CLAUDE.md

@@ -1 +1,214 @@
+<<<<<<< HEAD


Looks like a merge conflict token.

mokagio commented Feb 12, 2026

View reviewed changes

.claude/settings.json Outdated Show resolved Hide resolved

mokagio commented Feb 12, 2026

View reviewed changes

mokagio changed the title ~~Update CLAUDE.md with info for the build-test loop~~ Update CLAUDE.md with info for the build-test loop Feb 12, 2026

mokagio marked this pull request as ready for review February 12, 2026 09:23

mokagio requested review from crazytonyli and kean and removed request for crazytonyli February 12, 2026 09:23

mokagio marked this pull request as draft February 12, 2026 09:23

mokagio mentioned this pull request Feb 13, 2026

Add agent instructions to CLAUDE.md #25246

Closed

1 task

mokagio force-pushed the ainfra-1963-refine-claudemd-in-wordpress-ios-with-build-test-loop branch from 66d7da1 to a903cbf Compare February 13, 2026 02:24

mokagio changed the title ~~Update CLAUDE.md with info for the build-test loop~~ Improve build-test loop for AI agents Feb 13, 2026

mokagio added this to the 26.8 milestone Feb 13, 2026

mokagio added the Tooling Build, Release, and Validation Tools label Feb 13, 2026

mokagio self-assigned this Feb 13, 2026

mokagio marked this pull request as ready for review February 13, 2026 04:26

mokagio force-pushed the ainfra-1963-refine-claudemd-in-wordpress-ios-with-build-test-loop branch from a903cbf to ae6d1ed Compare February 13, 2026 04:27

crazytonyli approved these changes Feb 13, 2026

View reviewed changes

mokagio and others added 7 commits February 19, 2026 11:29

Update CLAUDE.md with info for the build-test loop

c183465

Remove unused rake build/test tasks

5201ff0

CI uses Fastlane exclusively; these local wrappers around `xcodebuild` were obsolete and hardcoded a stale simulator. --- Generated with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add test plan to run all unit tests in the modules

5b571f6

Ideally, this would build faster than the rest and allow focused testing. The blocker is that since the Modules don't build with `swift build` we can test with `swift test`. Plus, auto-generated schemes allow building but not testing!

Add missing module tests to default plan

494071f

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Document test targets and plan convention

40a4354

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mokagio and others added 10 commits February 19, 2026 11:36

Warn against parallel fastlane runs

de78218

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Document source-to-test-target convention

72f79a8

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Document WordPressUIUnitTests naming exception

a6e2fee

--- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove overly generous Claude allow-list entries

fc6ca07

Fix timezone mismatch in MockStatsServiceTests

95278f8

The test calendar used `.eastern` but the service used `.current`, so date interval keys never aligned. --- Generate with the help of Claude Code, https://code.claude.com Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mokagio force-pushed the ainfra-1963-refine-claudemd-in-wordpress-ios-with-build-test-loop branch from ae6d1ed to 3d2d225 Compare February 19, 2026 00:36

mokagio added 2 commits February 19, 2026 15:15

Use iPhone 17 as the default Simulator for local Fastlane tests

b5a7ab6

Push a class back from public to internal

7cfc3ca

kean reviewed Feb 20, 2026

View reviewed changes

CLAUDE.md

@@ -1 +1,214 @@

<<<<<<< HEAD

Copy link

Contributor

kean Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a merge conflict token.


		### Building and Testing

		Use the `test` Fastlane lane for local testing.

Conversation

mokagio commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing instructions

Uh oh!

wpmobilebot commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpmobilebot commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Build Failure Analysis

Uh oh!

Uh oh!

mokagio Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio commented Feb 12, 2026

Uh oh!

crazytonyli Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

crazytonyli Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

mokagio Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

crazytonyli Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 19, 2026

Quality Gate passed

Uh oh!

kean commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test 1

Test 2

Test 3

Test 4

Uh oh!

kean Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mokagio commented Feb 11, 2026 •

edited

Loading

wpmobilebot commented Feb 11, 2026 •

edited

Loading

wpmobilebot commented Feb 11, 2026 •

edited

Loading

wpmobilebot commented Feb 11, 2026 •

edited

Loading

kean commented Feb 19, 2026 •

edited

Loading