Skip to content

Implement new library loading approach #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pombredanne merged 43 commits into from
May 31, 2021
Merged

Implement new library loading approach #20

pombredanne merged 43 commits into from
May 31, 2021

Conversation

@pombredanne
Copy link
Member

@pombredanne pombredanne commented May 27, 2021 •
edited
Loading

We now load the libmagic native library and its database from paths found in:

  1. environment variables,
  2. OR a location provider plugin,
  3. OR the system PATH,
  4. OR we fail with an informative error message.

Based on original code contributed by @priv-kweihmann to scancode-plugins
in aboutcode-org/scancode-plugins#9 and
moved here and adapted for use in the core code rather than in a plugin.

This also includes the merge of @tardyp #17 to search for common locations as a failover

Signed-off-by: Philippe Ombredanne pombredanne@nexb.com

Signed-off-by: Philippe Ombredanne pombredanne@nexb.com

pombredanne and others added 30 commits January 25, 2021 12:54
@pombredanne
Use wheels embedded in virtualenv.pyz
182532f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Do not force an upgrade on virtualenv.pyz embeds
cd4e87b 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@steven-esser
Fix .gitattributes
51510cb 
Signed-off-by: Steven Esser <sesser@nexb.com>
@steven-esser
Merge pull request #21 from nexB/fix-gitattributes
386bb90 
Fix .gitattributes
@tardyp
failover to python-magic detection if vendored library not present
e7efce4 
This allows scancode to work on platforms not supported by typecode-libmagic
e.g. M1 Macs or windows WSL

Most of this code is from https://github.com/ahupp/python-magic/blob/e0ccc6d/magic/loader.py
by Adam Hupp, MIT License, compatible with typecode's Apache

Signed-off-by: Pierre Tardy <pierre.tardy@renault.com>
@pombredanne
Merge pull request #24 from nexB/use-venv-embeds
1c84e9e 
Use virtualenv-embedded libraries
@pombredanne
Update markers syntax for pytest
d6fe59f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add fallback version for setuptools_scm
ca6ab21 
This will work even from a git archive or when git is not installed.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add note for setuptools_scam fallback version
1364bbb 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Use azure-posix.yml for linux and macOS
be851b0 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new configure script derived from ScanCode
4f0aecf 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add notes on customization
aa04429 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adopt new configure --dev convention
56ada8f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Clarify CHANGELOG to be Rst
0dbcdc9 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add skeleton release notes to README.rst
d21aef3 
This was they do not end up in the template CHANGELOG.rst

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Implement new envt. variables approach
ae78d43 
We now load native libraries and executables from:
1. an envt. variable path
2. OR a locatin provider plugin
3. OR the PATH
or we fail

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge pull request #25 from nexB/new-configure
a75737d 
New configure
@pombredanne
Require latest commoncode
d77ff06 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Update CHANGELOG and README
7e9287a 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add note to vendor Pygments
a2f1535 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge latest skeleton
79388fb 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add vendy to testing/dev extras
9dfe6d4 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Bump vendored pygments to 2.9.0
f4d0960 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Make new libmagic loading work
50e1abb 
Its was skipping plugin loading. Also add tracing.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Detect file type content only if no extension
0f5029a 
Only do content-based detection for programing language if there is
no file extension as this avoids a large number of detection issues.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Adapt tests to latest Pygments
b480cf3 
Aslo ensure we use assertion in the correct order.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Use pip extras with the correct syntax
e5b373f 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Update CHANGELOG
5bbd8cb 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add CI links and format ReST
ca73e37 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Implement new library loading approach
001c831 
We now load the libmagic native library and its database from paths found in:
1. environment variables,
2. OR a location provider plugin,
3. OR the system PATH,
4. OR we fail with an informative error message.

Based on original code contributed by @priv-kweihmann to
scancode-plugins in aboutcode-org/scancode-plugins#9 and
moved here and adapted for use in the core code rather than in a plugin.

Contributed-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge branch 'failover' from @tardyp
1df5b73 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Rename test file
9963b44 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Remove unused import
b7f7337 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Format code and streamline failover support #17
68a2b7d 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Copy link
Member Author

pombredanne commented May 30, 2021

@tardyp I merged 17 in this branch which also carries over in a single place the code originally contributed by @priv-kweihmann for the typecode_libmagic_system_provided plugins aboutcode-org/scancode-plugins#9

Getting both of your reviews would be great before I merge

@pombredanne pombredanne mentioned this pull request May 30, 2021
Closed
@pombredanne pombredanne changed the title Implement new envt. variables approach Implement new library loding approach May 30, 2021
@pombredanne pombredanne changed the title Implement new library loding approach Implement new library loading approach May 30, 2021
@pombredanne
Format code and streamline license headers
237569d 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
tardyp
tardyp reviewed May 31, 2021
View reviewed changes
Copy link
Contributor

@tardyp tardyp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello,
Thanks for pinging me.

I am having hard time reviewing this 272 files PR.
Franckly I am not a huge fan of vendoring for opensource projects, although I do understand the need to limit dependency variablility for support reason.

I didn't file too much issue though, but I cannot claim a deep review. :-)

configure.bat Outdated
set PYTHON_EXECUTABLE=%PROVIDED_PYTHON%
goto run
if "%1" EQU "--python" (
echo "The --python is now DEPRECATED. Use the PYTHON_EXECUTABLE environment
Copy link
Contributor

@tardyp tardyp May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is an extra CR here.

Copy link
Member Author

@pombredanne pombredanne May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed in https://github.com/nexB/skeleton and merged here

src/typecode/magic2.py
logger_debug('get_magicdb_location:', 'got path magicdb location:', magicdb_loc)

if not magicdb_loc:
warnings.warn(
Copy link
Contributor

@tardyp tardyp May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this not needed as libmagic will (probably) already complain if it does not find its db in standard location.

Copy link
Member Author

@pombredanne pombredanne May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point... I like keeping the warning on our side too though as the message would be more explicit

@pombredanne
Update format
3aeb2ec 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add Python 3.9 to Travis
2c412e8 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Format and remove spurious spaces
69eec23 
From #20

Reported-by: Pierre Tardy <pierre.tardy@renault.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge latest skeleton
475c6ac 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Copy link
Member Author

pombredanne commented May 31, 2021

@tardyp you wrote:

I am having hard time reviewing this 272 files PR.
Franckly I am not a huge fan of vendoring for opensource projects, although I do understand the need to limit dependency variablility for support reason

I agree yet the Pygments vendoring here is a must here as we have dependencies on specific versions and vendoring is the sane approach to avoid conflicts (that popped up often in the past) when this fairly common Pygments library is installed otherwise.

@pombredanne
Copy link
Member Author

pombredanne commented May 31, 2021

@tardyp note also that we should be able to support Apple ARM chips as soon as I can find some CI supporting it.

tardyp
tardyp reviewed May 31, 2021
View reviewed changes
@pombredanne
Bump to more modern version of setuptools_scm
0e09ad9 
And remove v prefix from fallback version

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add space for correct syntax
e339a70 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge remote-tracking branch 'skeleton/main' into use-env-var
6395a4a
@pombredanne
Copy link
Member Author

pombredanne commented May 31, 2021

This looks decent now. Merging and releasing!

@pombredanne pombredanne merged commit d5a26bb into main May 31, 2021
@pombredanne pombredanne deleted the use-env-var branch May 31, 2021 17:38
JonoYang pushed a commit to aboutcode-org/purldb that referenced this pull request Apr 14, 2025
@pombredanne
Format and remove spurious spaces
94c382c 
From aboutcode-org/typecode#20

Reported-by: Pierre Tardy <pierre.tardy@renault.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
JonoYang added a commit to aboutcode-org/purl-validator that referenced this pull request Oct 24, 2025
@JonoYang @pombredanne
@chinyeungli @TG1999 @keshav-space @AyanSinhaMahapatra @swastkk @arnav-mandal1234 @arijitde92 @OmkarPh @johnmhoran
Add initial code and update skeleton files (#26)
9bc3e43 
* Add skeleton release notes to README.rst

This was they do not end up in the template CHANGELOG.rst

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update format

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add Python 3.9 to Travis

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format and remove spurious spaces

From aboutcode-org/typecode#20

Reported-by: Pierre Tardy <pierre.tardy@renault.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Bump to more modern version of setuptools_scm

And remove v prefix from fallback version

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add space for correct syntax

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Create junction from Scripts to bin

 * This is handy for windows to have the same path as linux

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Check for deps in local thirdparty directory #31

Signed-off-by: Jono Yang <jyang@nexb.com>

* Enforce use of requirements.txt #34

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add scripts from scancode-toolkit/etc/release/ #33

Signed-off-by: Jono Yang <jyang@nexb.com>

* Normalize license in load_pkginfo_data #33

    * Create copyright statement from holder information

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add --init option to configure #33

    * This is used for the case where we are starting off a project and have not yet generated requirements files

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update README.rst #33

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use venv as virtual environment directory name #37

    * Replace all references to `tmp` with `venv`

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update configure.bat #33

    * Add --init option to configure.bat
    * Update help text in configure and configure.bat

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add placeholder requirements.txt files #33

Signed-off-by: Jono Yang <jyang@nexb.com>

* Handle multiple options in configure #33

Signed-off-by: Jono Yang <jyang@nexb.com>

* Fix path to aboutcode in utils_thirdparty.py #33

    * Update README.rst

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update release notes in README.rst

Signed-off-by: Jono Yang <jyang@nexb.com>

* Handle ExpressionParseError #33

    * Update README.rst with instructions for post-initialization usage

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update README.rst

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Address review comments #33

    * Replace references to scancode-toolkit repo with links to the skeleton repo

    * Remove --python option from configure.bat

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update READMEs

Signed-off-by: Jono Yang <jyang@nexb.com>

* Fixed #41 - Handled encoding issue when generating ABOUT files

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Treat text files as text

And not a possible binaries
Also Ensure that we craft a minimally parsable license expression, even
if not correct.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add helper to publish files in GH releases

The upload is otherwise shaky.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add code to use curl if wget is not installed

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Do not issue warning if thirdparty dir is missing

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Handle as_text correctly in cache

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Handle as_text correctly in cache

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add support for Python 3.10

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Adopt black style

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Drop Ubuntu 16 add Python 3.10

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Disable Python 3.10 tests on macOS 10.14

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add code to remove the symlink before creating

 * It will prompts "Cannot create a file when that file already exists."
if the symlink already exist.

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Improve handling licenses without scancode

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add support for deb and rpm containers

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Support  licenses when ScanCode is not installed

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Improve wheel build

Allow to launch builds and then fetch built wheels later
Improve support for newer Pythons and OS versions.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Aligne with latest ScanCode TK updates

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not ignore setup.py

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update scripts

Make fetch_built_wheel work
Add new strip classifiers option to fix_thirdparty
Improve simple requirements parsing to get the latest versions

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not use pytest 7.0.0 which is buggy

Reference: pytest-dev/pytest#9608
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update `configure` to work with space in installation path

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Remove echo statement

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* automate pypi release on a tag

Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>

* Refine GH Action definition

* Do not use explicit references to Python version and project name in
  descriptions
* Use Python 3.8 as a base
* Use only plain ASCII

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format GH action yaml

The indentations were not correct

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Use verbose name for job

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add black codestyle test for skeleton

- see aboutcode-org/skeleton#54

Signed-off-by: keshav-space <keshav6240@gmail.com>

* Update configure.bat

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Deprecate windows-2016 images for azure CI

Modifies the azure CI for `vs2017-win2016` to `windows-2022` as 
the former will be deprecated very soon.

* Remove macos 10.14 job from azure-pipelines.yml

Signed-off-by: Jono Yang <jyang@nexb.com>

* Do not use Python 3.6 on Windows 2022 jobs

    * Python 3.6 is not available on Windows 2022 images

Signed-off-by: Jono Yang <jyang@nexb.com>

* Deprecate windows-2016 images for azure CI

* Modifies the azure CI for `vs2017-win2016` to `windows-2022` as the former will be deprecated very soon.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Remove macos 10.14 job from azure-pipelines.yml

Signed-off-by: Jono Yang <jyang@nexb.com>

* Do not use Python 3.6 on Windows 2022 jobs

    * Python 3.6 is not available on Windows 2022 images

Signed-off-by: Jono Yang <jyang@nexb.com>

* Run tests on macOS 11

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Align configuration scripts on POSIX and Windows

Doing so helps with keeping them in sync

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update README

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refactor and streamline thirdparty utilities

There is now a single primary script "fetch_thirdparty.py" that handles
everything in a simplified way. There is no utility to build wheels
anymore: these MUST be available before hand on PyPI or built
manually and uploaded in our self-hosted PyPI repository.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Cleanup whitespaces

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add usage instructions to README.rst

Signed-off-by: Jono Yang <jyang@nexb.com>

* Format code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Reorg setup sections

This is now organized with more important data first.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not depend on click. Use argparse.

These boostrap scripts cannot depend on click.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Correct configure scripts

These were no longer working

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Remove remnants of configure --init

This does not exists anymore

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Pytyon 3.6 is not available on Windows 2022

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add long_description_content_type

Twine and PyPI prefer having it.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Run fewer Azure jobs

This new configuration means that all the Python versions are tested
in a single CI job. This allows doing fewer checkouts and improves
CI throughput overall.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add RTD css templates

Adds changes to conf.py and html template theme_overrides.css
created by @johnmhoran

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Add GitHub action for doc build tests

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Fix conf.py to fix doc build

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Add docs option in configure

Adds a --docs option to the configure script to also install requirements
for the documentation builds.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Add documentation contribute page

Adds documentation page on contributing to the docs, and
also modifies directory structure to avoid having the
skeleton directory in docs merged in projects.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Properly check for existance of thirdparty dir

Signed-off-by: Jono Yang <jyang@nexb.com>

* Improve GH action documentation

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Use Python 3.9 as a base for actions

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Remove variable from string in fetch_thirdparty.py

    * The variable `environment` is not used when fetching sdists

Signed-off-by: Jono Yang <jyang@nexb.com>

* Improve thirdparty scripts

Ensure that site-package dir exists.
Other minor adjustments from a scancode-toolkit release

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update thirdparty fetching utilities

These were buggy in some corner cases.
They have been updated such that:
* --latest-version works.
* we can reliable fetch combinations of wheels and sdists for multiple
  OS combos at once
* we now support macOS universal wheels (for ARM CPUs)

Caching is now simpler: we have essentially a single file-based cache
under .cache. PyPI indexes are fetched and not cached, unless the new
--use-cached-index is used which can be useful when fetching many
thirdparty in a short timeframe.

The first PyPI repository in a list has precendence and we never fetch
from other repositories if we find wheels and sdsists there. This avoid
pounding too much on the self-hosted repo.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format code with black

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Only use PyPI for downloads

This is much faster

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Enable automatic release on tag

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Remove Travis config

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add ScanCode Code of Conduct

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Improve detection of thirdparty dir

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add comment

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Remove the thirdparty directory

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Update the azure-piprlines

The following images are deprecated in GitHub actions and Azure DevOps:

* `ubuntu-18.04` : actions/runner-images#6002
* `macos-10.15` : actions/runner-images#5583

Due to this there was failing tests due to planned brownouts.

Updated ubuntu18 to ubuntu22
Updated macos-1015 to macos12

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Add missing os import in utils_requirements.py

    * Rename references to etc/release with etc/scripts

Signed-off-by: Jono Yang <jyang@nexb.com>

* Reinstate Ubuntu 18 and drop Python 3.6 and 3.7

3.7 is not available anymore on newer OSes and is retired in 2023.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Replace packaging with packvers

* import update in src/scripts/utils_dejacode

* Packvers replacing packaging in other src/scripts

* Added packvers in src/scripts/requirments.txt

Signed-off-by: swastik <swastkk@gmail.com>

* Ignore egginfo

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Drop Python 3.7 add Python 3.11

Also test on latest Ubuntu and macOS

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Clean .cache and .eggs

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add COC to redistributed license-like files

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add new Makefile

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Align scripts with latest ScanCode Toolkit

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add pycodestyle in setup.cfg

Signed-off-by: Arnav Mandal <arnav.mandal1234@gmail.com>

* Remove deprecated github-actions/azure images

- remove deprecated `ubuntu-18.04` image
- remove deprecated `macos-10.15` image

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Pin Sphinx version to 6.2.1

Signed-off-by: Jono Yang <jyang@nexb.com>

* Publish PDF version of RTD documentation

Made changes in the .readthedocs.yaml to enable format for downloading pdf and epub versions of the documentation and added latex_elements in the conf.py file which generates the pdf without blank pages. The minimum version requirement for sphinx was 6.2.1 which was causing build failure in read the docs, hence changing it 3.3.1 as written in setup.cfg of nexB/aboutcode

Signed-off-by: Arijit De <arijitde2050@gmail.com>

* Bump github actions versions #75

Update the following actions:

* actions/checkout
* actions/setup-python

Reference: aboutcode-org/skeleton#75
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Update github actions

Reference: aboutcode-org/skeleton#75
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Update RTD buil

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Fix unordered lists issue

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Add redirects for docs

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Create script to update repo skeleton #80

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add macOS-13 job in azure-pipelines.yml

Signed-off-by: Jono Yang <jyang@nexb.com>

* Added docs server script, dark mode & copybutton for docs

Signed-off-by: Omkar Phansopkar <omkarphansopkar@gmail.com>

* Update CSS to widen page and handle mobile #84

Reference: aboutcode-org/skeleton#84

Signed-off-by: John M. Horan <johnmhoran@gmail.com>

* Delete theme_overrides_SUPERSEDED.css as no longer needed #84

Reference: aboutcode-org/skeleton#84

Signed-off-by: John M. Horan <johnmhoran@gmail.com>

* Fix top padding for rst content

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Update CI runners and python version

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Replace deprecated macos CI runners

Replace macos-11 runners with macos-14 runners.

Reference: https://github.com/actions/runner-images?tab=readme-ov-file#available-images
Reference: aboutcode-org/skeleton#89
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Update minimum required python version to 3.8

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Update link references of ownership from nexB to aboutcode-org

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Drop deprecated macos-12 runner

* Update CI/Actions runners

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Replace black and isort with ruff

    * Use ruff config and Make commands from scancode.io

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use org standard 100 line length

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Lint all common code directories

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Remove unused targets

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Improve import sorting

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Apply small code updates

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Apply cosmetic refactorings

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Reformat test code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine ruff configuration

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format doc

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Run doc8 on all rst files

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Enable doc style checks

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add support for new OS versions

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update scripts aboutcode references

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not format more test data

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not treat rst as Python

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Combine testing and docs extra for simplicity

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine checking of docs with doc8

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine doc handling

* remove CI scripts and use Makefile targets instead
* ensure doc8 runs quiet
* add new docs-check make target to run documentation and links checks
* update oudated doc for docs contribution

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add twine check to release publication

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine doc contribution docs

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Replace black and isort with ruff

    * Use ruff config and Make commands from scancode.io

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use org standard 100 line length

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Lint all common code directories

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Remove unused targets

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Improve import sorting

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Apply small code updates

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Apply cosmetic refactorings

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Reformat test code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine ruff configuration

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format doc

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Run doc8 on all rst files

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Enable doc style checks

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not format more test data

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not treat rst as Python

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Combine testing and docs extra for simplicity

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine checking of docs with doc8

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine doc handling

* remove CI scripts and use Makefile targets instead
* ensure doc8 runs quiet
* add new docs-check make target to run documentation and links checks
* update oudated doc for docs contribution

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add twine check to release publication

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Refine doc contribution docs

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update codestyle command

    * Remove trailing whitespace

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update README.rst

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update BUILDDIR envvar in docs/Makefile

Signed-off-by: Jono Yang <jyang@nexb.com>

* Fix doc line length

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Format code

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Correct supported runner on Azure

See for details:
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml

macOS ARM images do not seem to be supported there

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add code checks to CI

Remove running "make check" as a test

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Revert support for Python 3.13

This is not yet supported everywhere

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Ignore local .env file

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add correct extras for documentation

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Improve MANIFEST

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Improve cleaning on POSIX

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Rename dev extra to "dev"

Instead of testing ... and update references accordingly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add more excludes from tests

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Do not lint django migrations

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Add README.rst to list of "license files"

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Use Python 3.9 as lowest suupported version

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Drop pycodestyle

Not used anymore

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Bump pytest minimal version

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

* Update CI runners and scripts

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Add missing wheel builds on release CI

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Drop python3.9 support and add python 3.14

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Support trusted-publishing for package releases

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Update RTD build python version

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Add initial code

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add initial purl map

Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove functions to create PurlValidator from purl list

    * This was done for dev purposes

Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove unnecessary code

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update comments

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update AUTHORS.rst

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update requirements.txt

Signed-off-by: Jono Yang <jyang@nexb.com>

* Add tests

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update formatting

Signed-off-by: Jono Yang <jyang@nexb.com>

---------

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Chin Yeung Li <tli@nexb.com>
Signed-off-by: Jono Yang <jyang@nexb.com>
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
Signed-off-by: keshav-space <keshav6240@gmail.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: swastik <swastkk@gmail.com>
Signed-off-by: Arnav Mandal <arnav.mandal1234@gmail.com>
Signed-off-by: Arijit De <arijitde2050@gmail.com>
Signed-off-by: Omkar Phansopkar <omkarphansopkar@gmail.com>
Signed-off-by: John M. Horan <johnmhoran@gmail.com>
Co-authored-by: Philippe Ombredanne <pombredanne@nexb.com>
Co-authored-by: Chin Yeung Li <tli@nexb.com>
Co-authored-by: Tushar Goel <tushar.goel.dav@gmail.com>
Co-authored-by: Philippe Ombredanne <pombredanne@gmail.com>
Co-authored-by: keshav-space <keshav6240@gmail.com>
Co-authored-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Co-authored-by: swastik <swastkk@gmail.com>
Co-authored-by: Arnav Mandal <arnav.mandal1234@gmail.com>
Co-authored-by: Arijit De <arijitde2050@gmail.com>
Co-authored-by: Omkar Phansopkar <omkarphansopkar@gmail.com>
Co-authored-by: John M. Horan <johnmhoran@gmail.com>
Co-authored-by: Keshav Priyadarshi <git@keshav.space>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@tardyp tardyp tardyp left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pombredanne @tardyp @steven-esser