Allow all external download URLs to be aliased

[rarkins]

Some users need to build their own containerbase-derived images in their product environments which do not have direct internet access. Instead, they need to set up generic proxies on Artifactory for each distinct external host which is approved, such as github.com, nodejs.org, etc.

Here's some examples used by containerbase:

Host	Tool(s)
https://nodejs.org	node
https://api.adoptium.net	java
https://downloads.lightbend.com	scala
https://github.com	sbt, python, php,
https://dot.net	dotnet
https://cache.ruby-lang.org/	ruby

We can assume for now that such users can already redirect common registries like apt, npm, pip, etc - the primary concern is on arbitrary URLs like the above.

The best way to test this would be to have a build environment with external URLs blocked by default and then try to build a "full" image with all tools and find the missing URLs one by one.

The next thing we'd want is a simple way of defining alternative URLs using env.

One possibility would be to have a syntax where the full from/to is specified in separate, related variables. e.g.

URL_REPLACE_0_FROM=https://node.org
URL_REPLACE_0_TO=https://artifactory.company.com/something/nested

Such a syntax is verbose but then at least should cover every case.

A higher level approach would be like JAVA_BASE_URL=https://artifactory.company.com/something/java but has the downsides:

• One tool might try multiple hosts, and
• Multiple tools might use the same host (e.g. github.com)

---

Missing Tools

• bundler (gem) feat(ruby): convert gem tools and add install-gem command #1355
• cocoapods (gem) feat(ruby): convert gem tools and add install-gem command #1355
• dotnet feat(cli): convert dotnet #1347
• hashin (pip) 6b8670c
• pdm (pip) 6b8670c
• pip-tool (pip) 6b8670c
• pipenv (pip) 6b8670c
• poetry (pip) 6b8670c

[viceice]

will do this after conversation to higher level, as it's much easier to implement in typescript.

[rarkins]

I need this within maximum 4 weeks :-/

I think longer term we should take all these URLs and proxy them ourselves through our own CDN/host so that the user only needs to alias one or a few URLs and not 10+

[viceice]

blocked by

• Switch from bash scripts to higher language #771

[viceice]

working

• npm can be configured via npm_config_registry env or ~/.npmrc
• pip can be configured via PIP_INDEX_URL env or ~/.config/pip/pip.conf
• gem can be configured via RUBYGEMS_HOST env or ~/.gemrc ^{1 2}

Footnotes

1. https://docs.inedo.com/docs/proget-feeds-rubygem ↩

2. https://guides.rubygems.org/command-reference/#gem-environment ↩

[viceice]

related:

• Log hosts during build #7
• Convert tools to ts #1074

[randygeyer-ws]

Is it sufficient to replace just base urls or will the entire url path need to be set, perhaps minus the artifact file name/ext?

An alternate/simple way to test this is to add /etc/hosts entries for each default hostname to point back to localhost, vs actually blocking the outbound traffic.

[rarkins]

Is it sufficient to replace just base urls or will the entire url path need to be set, perhaps minus the artifact file name/ext?

By base URL I'm not meaning only the host, I mean it could be like https://host.com/some/path too. Is that what you're worried about?