[Perf] Add performance monitor tool & optimize log output

[0oshowero0]

Summary by CodeRabbit

Release Notes

• Chores
  • Introduced performance monitoring instrumentation across key system operations and request handling pathways with timing measurements.
  • Enhanced internal logging and tracing throughout the request processing lifecycle for improved operational visibility.
  • Updated internal identifiers for consistency across storage unit communications and response messages.
  • Added performance measurement utilities with configurable flush intervals to track per-operation success counts and execution time statistics.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

A new performance monitoring utility (IntervalPerfMonitor) is introduced to track operation metrics and periodically flush summarized statistics. It is integrated across the controller and storage backend to instrument GET_META, PUT_DATA, GET_DATA, and CLEAR_DATA pathways. Logging is enhanced with additional processing-start entries, and server-id references are replaced with storage-unit-id for consistency.

Changes

Cohort / File(s)	Change Summary
Performance Monitoring Utility transfer_queue/utils/perf_utils.py	New file introducing IntervalPerfMonitor class that tracks success counts and per-operation execution times. Provides measure() context manager to wrap operations, computes timing statistics (total, average, min, max, rate), and periodically flushes logs via _flush_logs(). Configurable via environment variables for logging level and flush interval.
Controller Instrumentation transfer_queue/controller.py	Imports and applies IntervalPerfMonitor wrapping around key request/metadata pathways (GET_META, GET_CLEAR_META, CLEAR_META, CHECK_CONSUMPTION, NOTIFY_DATA_UPDATE). Enhances logging with "start-processing" entries in _process_request and _update_data_status. Updates log message prefixes from "Dynamic Controller" to "Controller". Adds runtime timing and tracing in thread lifecycle.
Storage Backend Instrumentation transfer_queue/storage/simple_backend.py	Imports IntervalPerfMonitor and initializes per-storage-unit perf monitor in main processing loop. Wraps PUT_DATA, GET_DATA, and CLEAR_DATA operations with perf measurement contexts. Replaces zmq_server_info.id with storage_unit_id across log messages, error messages, and ZMQ response metadata. Adds informational log when starting put/get request processing.
Storage Manager Logging transfer_queue/storage/managers/simple_backend_manager.py	Adds INFO-level logging statements in AsyncSimpleStorageManager.put_data and AsyncSimpleStorageManager.get_data to record the number of samples being processed. No control flow changes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• perf_utils.py: New utility class with metric tracking and periodic logging; verify environment variable configuration defaults and timer accuracy.
• controller.py: Multiple perf monitoring blocks added across different code paths; ensure consistent context manager usage and verify logging message clarity.
• simple_backend.py: ID replacement across many references; carefully audit all occurrences of zmq_server_info.id → storage_unit_id to ensure no regressions, and verify perf monitor initialization.
• Consistent patterns across files reduce cognitive load, but the spread and ID replacement warrant careful cross-file verification.

Poem

🐰 Hopping through operations with a timer in paw,
Measuring each pathway without a single flaw,
From meta to data, performance now shines bright,
Queue instrumentation makes everything right! ⚡

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[0oshowero0]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Copilot]

Pull request overview

This PR introduces a performance monitoring utility and optimizes logging output across the transfer queue system. The changes add interval-based performance tracking for key operations and clean up log messages by removing redundant "Dynamic" prefixes and improving consistency in identifier usage.

Key Changes

• New performance monitoring utility: Added IntervalPerfMonitor class to track operation metrics (count, timing statistics) with configurable flush intervals
• Integrated performance tracking: Instrumented PUT/GET/CLEAR operations in storage backend and GET_META/CLEAR_META/CHECK_CONSUMPTION operations in controller
• Log output optimization: Replaced zmq_server_info.id with storage_unit_id for consistency, removed "Dynamic" prefix from controller log messages, and simplified partition creation logs

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
transfer_queue/utils/perf_utils.py	New performance monitoring utility with interval-based metrics aggregation and logging
transfer_queue/storage/simple_backend.py	Integrated performance monitoring for storage operations and updated logging to use storage_unit_id consistently
transfer_queue/storage/managers/simple_backend_manager.py	Added informational logging for put_data and get_data requests with sample counts
transfer_queue/controller.py	Added performance monitoring to controller request processing threads and cleaned up log messages by removing "Dynamic" prefix and updating thread names

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In the measure context manager, successful operations are counted even when exceptions occur. The finally block increments success_counts and records process_time regardless of whether the measured operation succeeded or failed. This could lead to misleading performance metrics that include failed operations. Consider moving the metric recording to the try block or adding error tracking.

Suggested change

	finally:
	cost = time.perf_counter() - start_time
	self.success_counts[op_type] += 1
	self.process_time[op_type].append(cost)
	except Exception:
	# Optionally, add error tracking here
	raise
	else:
	cost = time.perf_counter() - start_time
	self.success_counts[op_type] += 1
	self.process_time[op_type].append(cost)
	finally: