[PD1-588] Allow access to data marketplace data #6
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @amy-nihao, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enables access to data from the data marketplace, specifically for modeling metrics. It provides new client-side capabilities to fetch this data, including an option to retrieve it in the highly efficient Apache Arrow format. The changes ensure that the new data access adheres to existing permission models, requiring the same METRICS_READ permission as normal metrics.
Highlights
- New Data Marketplace Access: Introduced new capabilities to retrieve data from the data marketplace, specifically focusing on "modeling metrics" data.
- Expanded API Endpoints: Added three new client methods: get_modeling_metrics, get_modeling_metrics_data, and get_modeling_metrics_data_arrow, to interact with the new modeling metrics API endpoints.
- Apache Arrow Integration: Implemented support for fetching modeling metrics data in Apache Arrow IPC format, which provides a more efficient way to transfer large datasets.
- Updated Documentation and Examples: The README.md has been updated with documentation for the new methods, and dedicated example scripts have been added to demonstrate their usage.
- Comprehensive Testing: New unit tests have been added to ensure the correctness and reliability of the newly introduced modeling metrics retrieval functionalities.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces capabilities to retrieve data from the data marketplace, adding new methods for fetching modeling metrics and their data, including support for Apache Arrow format. The changes are well-supported with new example scripts, comprehensive unit tests, and updated documentation. My review focuses on improving documentation consistency for clarity, suggesting a refactor to reduce code duplication for better long-term maintainability, and enhancing the robustness of type checking in the new methods.
There was a problem hiding this comment.
I think that "marketplace" would be a better name than "modeling" for these endpoints. I'm not sure that it's worth the effort to change the name, since I think that we might rather merge all metrics into one namespace (I mean: use the same endpoint for all types of metrics: local, marketplace, and predictive model.)
I verified that both examples run, using https://github.com/CVector-Energy/cvec-backend/pull/76
(.venv) joshuanapoli@mac cvec-python % CVEC_API_KEY=cva_6DSH9oHkQ5b157bpKc9hbGMGJJ3vdj49cuqe CVEC_HOST=https://sandbox.cvector.dev python examples/get_modeling_metrics_data_arrow_example.py
Getting modeling metrics data as Arrow...
Arrow table shape: 21840 rows
First few rows:
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 34.06 at 2025-06-17 21:25:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 33.49 at 2025-06-17 21:30:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 33.01 at 2025-06-17 21:35:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 33.39 at 2025-06-17 21:40:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 37.03 at 2025-06-17 21:50:00+00:00
(.venv) joshuanapoli@mac cvec-python % CVEC_API_KEY=cva_6DSH9oHkQ5b157bpKc9hbGMGJJ3vdj49cuqe CVEC_HOST=https://sandbox.cvector.dev python examples/get_modeling_metrics_data_example.py
Getting modeling metrics data...
Found 36 data points
First few data points:
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 17.48 at 2025-09-05 05:30:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 14.17 at 2025-09-05 05:35:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 11.52 at 2025-09-05 05:40:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 6.67 at 2025-09-05 05:45:00+00:00
- Data_Marketplace/PROD/Miso_5min/ILLINOIS_HUB_lmp: 6.61 at 2025-09-05 05:50:00+00:00
... and 31 more data points
|
Will come back for a larger refractor |
joshuanapoli
changed the title
Description
Add capabilities to retrieve data marketplace data. Requires same
METRICS_READpermission as normal metrics. The behavior is the same as normal metrics but there is no way to add to the marketplace database since that is controlled in a different repo. Also supports returning the data in arrow format.Testing
Test with https://github.com/CVector-Energy/cvec-backend/pull/76
The
cvec-backendendpoints required is already deployed to sandbox. Running any of the new modeling example scripts likeget_modeling_metrics_data_exampleretrieves the data from the modeling database.