-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Swap CI to CFS
#45995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Swap CI to CFS
#45995
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
6b3ca6a
simplify the location of auth-dev-feed so that it works for both play…
scbedd f58b1da
make build test echo out the feed being utilized
scbedd cf28c2f
didn't actually uncomment the UV index setting
scbedd adbbb3c
Update eng/pipelines/templates/steps/build-test.yml
scbedd 77e9ee9
update step name, add a requirement that is NOT YET in our dev feed t…
scbedd 0e824b6
more manipulation
scbedd e39fe81
UV won't pull stuff automatically into the feed? need to add pip extr…
scbedd d0c6272
revert attempt so far. add explicit check for installing ci_tools.txt
scbedd 017ef3c
accidentally disabled UV
scbedd ce01280
test a pip install
scbedd 2f48306
ensure that new packages can be pulled in
scbedd 333736d
install pip and call
scbedd 11f0d58
log properly
scbedd c892017
UV_INDEX_URL -> UV_DEFAULT_INDEX
scbedd 90d6c47
use environment variables only. name the index
scbedd 8d60b83
try again with the unauthed string + UV_INDX value
scbedd 3bd9af3
target a package that ISNT in the feed, is UV able to retrieve it pro…
scbedd f1b5007
-vv install! WHAT IS HAPPENING
scbedd 3ffc9e4
remove some extra changes that were added for debugging
scbedd 5504fde
verbose output so we can see why this thing is blowing up
scbedd 84505b1
pin older version of cibuildwheel during save package properties
scbedd e6e4ed9
cibuildwheel needs to be pinned as it's installing a new version of p…
scbedd eecf6ab
on newer macs, we're seeing the port resolve to an IPV6 address by de…
scbedd d582481
revert the changes
scbedd acc6758
fix the remaining references to hitting pypi directly
scbedd 4838828
update project_release and test_pypi_client.py
scbedd 6741df5
ensure that we auth the dev feed on generating job matrix and regress…
scbedd 2b15c69
account for requests maximum on 2.32.5
scbedd 2f7ac38
apply formatting
scbedd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| import base64 | ||
| import json | ||
| import logging | ||
| import os | ||
| import re | ||
| from dataclasses import dataclass | ||
| from typing import Any, Dict, List, Optional | ||
| from urllib.parse import urlparse | ||
|
|
||
| from packaging.version import Version, InvalidVersion, parse | ||
| from urllib3 import PoolManager, Retry | ||
|
|
||
|
|
||
| def pep503_normalize(name: str) -> str: | ||
| return re.sub(r"[-_.]+", "-", name).lower() | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class AzureArtifactsFeedConfig: | ||
| organization: str | ||
| project: Optional[str] # None if the feed is organization-scoped | ||
| feed: str # feed name or GUID | ||
| api_version: str = "7.1" | ||
|
|
||
| bearer_token: Optional[str] = None | ||
| pat: Optional[str] = None | ||
|
|
||
|
|
||
| # Pattern: https://pkgs.dev.azure.com/{org}/{project}/_packaging/{feed}/pypi/simple/ | ||
| # or org-scoped: https://pkgs.dev.azure.com/{org}/_packaging/{feed}/pypi/simple/ | ||
| _AZDO_FEED_RE = re.compile( | ||
| r"/(?P<org>[^/]+)/(?:(?P<project>[^/_][^/]*)/)?" r"_packaging/(?P<feed>[^/]+)/pypi/simple/?$" | ||
| ) | ||
|
|
||
|
|
||
| def parse_pip_index_url(url: str) -> Optional[AzureArtifactsFeedConfig]: | ||
| """If *url* points to an Azure Artifacts PyPI feed, return a config; else None.""" | ||
| parsed = urlparse(url) | ||
| if "pkgs.dev.azure.com" not in parsed.hostname: | ||
| return None | ||
|
|
||
| m = _AZDO_FEED_RE.search(parsed.path) | ||
| if not m: | ||
| return None | ||
|
|
||
| # Embedded credentials from PipAuthenticate@1 | ||
| pat = None | ||
| if parsed.password: | ||
| pat = parsed.password | ||
|
|
||
| return AzureArtifactsFeedConfig( | ||
| organization=m.group("org"), | ||
| project=m.group("project"), | ||
| feed=m.group("feed"), | ||
| pat=pat or os.environ.get("AZDO_PAT"), | ||
| ) | ||
|
|
||
|
|
||
| class AzureArtifactsClient: | ||
| """ | ||
| Minimal client to list package versions from an Azure Artifacts feed | ||
| via Azure DevOps Artifacts REST API. | ||
| """ | ||
|
|
||
| def __init__(self, cfg: AzureArtifactsFeedConfig, base_url: str = "https://feeds.dev.azure.com"): | ||
| self._cfg = cfg | ||
| self._base_url = base_url.rstrip("/") | ||
| self._http = PoolManager( | ||
| retries=Retry(total=3, raise_on_status=True), | ||
| ca_certs=os.getenv("REQUESTS_CA_BUNDLE", None), | ||
| ) | ||
|
|
||
| def _auth_header(self) -> Dict[str, str]: | ||
| if self._cfg.bearer_token: | ||
| return {"Authorization": f"Bearer {self._cfg.bearer_token}"} | ||
|
|
||
| if self._cfg.pat: | ||
| # Azure DevOps PATs can be used via HTTP Basic by base64-encoding ":<PAT>". | ||
| token = base64.b64encode(f":{self._cfg.pat}".encode("utf-8")).decode("ascii") | ||
| return {"Authorization": f"Basic {token}"} | ||
|
|
||
| return {} | ||
|
|
||
| def _path_prefix(self) -> str: | ||
| # If project-scoped feed: /{org}/{project}/... | ||
| # If org-scoped feed: /{org}/... | ||
| if self._cfg.project: | ||
| return f"{self._cfg.organization}/{self._cfg.project}" | ||
| return self._cfg.organization | ||
|
|
||
| def _get_json(self, url: str, params: Dict[str, Any]) -> Any: | ||
| headers = {"Accept": "application/json", **self._auth_header()} | ||
| r = self._http.request("GET", url, fields=params, headers=headers) | ||
| return json.loads(r.data.decode("utf-8")) | ||
|
|
||
| def list_feeds(self) -> List[Dict[str, Any]]: | ||
| url = f"{self._base_url}/{self._path_prefix()}/_apis/packaging/feeds" | ||
| data = self._get_json(url, {"api-version": self._cfg.api_version}) | ||
| # Many Azure DevOps APIs return {"count": n, "value": [...]}; be tolerant. | ||
| return data["value"] if isinstance(data, dict) and "value" in data else data | ||
|
|
||
| def resolve_feed_id(self) -> str: | ||
| feed = self._cfg.feed | ||
| if re.fullmatch(r"[0-9a-fA-F-]{36}", feed): | ||
| return feed | ||
|
|
||
| for f in self.list_feeds(): | ||
| if f.get("name") == feed: | ||
| return f["id"] | ||
|
|
||
| raise KeyError(f"Feed not found: {feed!r}") | ||
|
|
||
| def get_package_record(self, package_name: str, include_deleted: bool = False) -> Dict[str, Any]: | ||
| feed_id = self.resolve_feed_id() | ||
| url = f"{self._base_url}/{self._path_prefix()}/_apis/packaging/Feeds/{feed_id}/packages" | ||
|
|
||
| params = { | ||
| "api-version": self._cfg.api_version, | ||
| "protocolType": "pypi", | ||
| "packageNameQuery": package_name, | ||
| "includeAllVersions": "true", | ||
| "includeDeleted": "true" if include_deleted else "false", | ||
| } | ||
|
|
||
| data = self._get_json(url, params) | ||
| packages = data["value"] if isinstance(data, dict) and "value" in data else data | ||
|
|
||
| # packageNameQuery is "contains string", so choose best match. | ||
| target = pep503_normalize(package_name) | ||
| for pkg in packages: | ||
| if pep503_normalize(pkg.get("normalizedName", pkg.get("name", ""))) == target: | ||
| return pkg | ||
| for pkg in packages: | ||
| if pep503_normalize(pkg.get("name", "")) == target: | ||
| return pkg | ||
|
|
||
| raise KeyError(f"Package not found in feed: {package_name!r}") | ||
|
|
||
| def get_ordered_versions(self, package_name: str, include_deleted: bool = False) -> List[Version]: | ||
| pkg = self.get_package_record(package_name, include_deleted=include_deleted) | ||
|
|
||
| out: List[Version] = [] | ||
| for v in pkg.get("versions", []): | ||
| if (not include_deleted) and v.get("isDeleted", False): | ||
| continue | ||
|
|
||
| raw = v.get("version") | ||
| if not raw: | ||
| continue | ||
|
|
||
| try: | ||
| out.append(parse(raw)) | ||
| except InvalidVersion: | ||
| logging.warning("Invalid version %r for package %s (feed=%s)", raw, package_name, self._cfg.feed) | ||
|
|
||
| out.sort() | ||
| return out |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.