Skip to content

⚡ Bolt: Make NodeModulesScanner truly concurrent and reduce syscalls #216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
acebytes merged 1 commit into from
May 5, 2026
+13 −11
Merged

⚡ Bolt: Make NodeModulesScanner truly concurrent and reduce syscalls #216

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## 2024-05-05 - TaskGroup parallelism and URL.resourceValues
**Learning:** In Swift Concurrency, when an actor creates a `TaskGroup` and its subtasks call an `isolated` method on the same actor, execution is serialized on the actor's executor, destroying parallelism. Also, `URL.resourceValues(forKeys:)` can perform a single optimized syscall instead of separate `FileManager.fileExists` and `FileManager.attributesOfItem` calls.
**Action:** Mark computationally intensive or state-independent methods as `static` or `nonisolated` so they run on the global concurrent pool, and pass dependencies (like `FileManager` and `skipDirs`) as arguments. Use `URL.resourceValues(forKeys:)` for checking directory existence and attributes simultaneously.
21 changes: 10 additions & 11 deletions Sources/Cacheout/Scanner/NodeModulesScanner.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,14 +55,15 @@ actor NodeModulesScanner {
func scan(maxDepth: Int = 6) async -> [NodeModulesItem] {
let home = fileManager.homeDirectoryForCurrentUser
var allItems: [NodeModulesItem] = []
let currentFileManager = self.fileManager

// Scan each search root in parallel
await withTaskGroup(of: [NodeModulesItem].self) { group in
for root in Self.searchRoots {
let rootURL = home.appendingPathComponent(root)
guard fileManager.fileExists(atPath: rootURL.path) else { continue }
guard currentFileManager.fileExists(atPath: rootURL.path) else { continue }
group.addTask {
await self.findNodeModules(in: rootURL, maxDepth: maxDepth)
await Self.findNodeModules(in: rootURL, fileManager: currentFileManager, skipDirs: Self.skipDirs, maxDepth: maxDepth)
}
}
for await items in group {
Expand All @@ -77,25 +78,23 @@ actor NodeModulesScanner {
.sorted { $0.sizeBytes > $1.sizeBytes }
}

private func findNodeModules(in directory: URL, maxDepth: Int, currentDepth: Int = 0) async -> [NodeModulesItem] {
private static func findNodeModules(in directory: URL, fileManager: FileManager, skipDirs: Set<String>, maxDepth: Int, currentDepth: Int = 0) async -> [NodeModulesItem] {
guard currentDepth < maxDepth else { return [] }

var results: [NodeModulesItem] = []
let nodeModulesURL = directory.appendingPathComponent("node_modules")

// Check if this directory contains node_modules
var isDir: ObjCBool = false
if fileManager.fileExists(atPath: nodeModulesURL.path, isDirectory: &isDir), isDir.boolValue {
let size = directorySize(at: nodeModulesURL)
if let values = try? nodeModulesURL.resourceValues(forKeys: [.isDirectoryKey, .contentModificationDateKey]), values.isDirectory == true {
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot May 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat symlinked node_modules as directories

Using resourceValues(...).isDirectory == true stops detecting node_modules when that path is a symbolic link to a directory. FileManager.fileExists(atPath:isDirectory:) (the previous logic) reports such symlinks as directories, but isDirectoryKey on the symlink URL is false, so these projects are now skipped entirely and their cache size is never reported. This affects setups that symlink node_modules (common in some workspace/tooling layouts).

Useful? React with 👍 / 👎.

let size = directorySize(at: nodeModulesURL, fileManager: fileManager)
if size > 0 {
let lastMod = try? fileManager.attributesOfItem(atPath: nodeModulesURL.path)[.modificationDate] as? Date
let projectName = directory.lastPathComponent
results.append(NodeModulesItem(
projectName: projectName,
projectPath: directory,
nodeModulesPath: nodeModulesURL,
sizeBytes: size,
lastModified: lastMod
lastModified: values.contentModificationDate
))
}
// Don't recurse into projects that have node_modules — they won't have nested projects
Expand All @@ -111,16 +110,16 @@ actor NodeModulesScanner {

for item in contents {
let name = item.lastPathComponent
guard !Self.skipDirs.contains(name) else { continue }
guard !skipDirs.contains(name) else { continue }
guard (try? item.resourceValues(forKeys: [.isDirectoryKey]).isDirectory) == true else { continue }
let subResults = await findNodeModules(in: item, maxDepth: maxDepth, currentDepth: currentDepth + 1)
let subResults = await findNodeModules(in: item, fileManager: fileManager, skipDirs: skipDirs, maxDepth: maxDepth, currentDepth: currentDepth + 1)
results.append(contentsOf: subResults)
}

return results
}

private func directorySize(at url: URL) -> Int64 {
private static func directorySize(at url: URL, fileManager: FileManager) -> Int64 {
var total: Int64 = 0
guard let enumerator = fileManager.enumerator(
at: url,
Expand Down