Collect data from Yocto/bitbake .bb manifest files

[pombredanne]

The bitbake manifests commonly contain decent license and origin information.

[pombredanne]

See also https://gist.github.com/craigez/df6bcd275440ee0047d020511a3d4532#bitbake-recipes-and-yocto-distributions by @craigez

[craigez]

I have some little scripts that dump declared licenses, origins, etc... let me know if that would be useful.

[pombredanne]

@craigez that would be great! Even if this is not in Python that will be useful to create a new manifest parser in https://github.com/nexB/scancode-toolkit/tree/develop/src/packagedcode

[craigez]

I actually insert it into the bitbake process to dump the information. This requires updating the configuration to include this build step.

[pombredanne]

@craigez that's fine I guess, especially since this is Python code. I am also curious on how you plugin some code in the bitbake process BTW ... as this can be useful for tracing, licensing scanning, etc.

[craigez]

@pombredanne So, here's an example of the type of script I've used before to extract data for one-off investigations. It was based on the existing spdx.bbclass that uses fossology, but this just dumps out information without scanning: https://gist.github.com/craigez/607ec343a7a08f3b465c7307377ae84a

And here's a hacky way to get it to run: https://gist.github.com/craigez/066285753b69139b9d98b0fdd17ae56a

You can add other bitbake fields like 'license', etc... to get that information into the CSV

Mark and I were also looking at how to incorporate scanning, etc... https://github.com/mcharleb/meta-srcmap (this is also how you properly add a layer...)

[pombredanne]

@craigez Thanks you++ for taking the time to put these together ... But this means this is used while actually running a build, correct?

That would be more of a job for TraceCode then IMHO.

ScanCode only does static analysis and manifest parsing... so I was more thinking about something that can collect information from .bb files found in a codebase at rest.

[craigez]

@pombredanne I wanted to do something like that too, but I couldn't find a pre-existing python library to parse the files. It really needs to interpret them though anyway, as files include other files and there are variables defined during the build, etc... So it was easier/quicker for my use case to just add a build step and then use bitbake to run it (without compilation, linking steps, etc...)

[pombredanne]

Some extra pointers for some minimal implementation of a static analysis:

• https://www.openembedded.org/wiki/Upgrading_packages helps to understand that a package .bb filename is "name_version.bb" and so does this doc: https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#locating-and-parsing-recipes
• https://www.openembedded.org/wiki/Versioning_Policy documents versioning and PN, PV and PR fields (e.g name, version and release
• https://www.openembedded.org/wiki/Recipe_License_Fields documents the license fields of a .bb file
• On licenses see also Consider collecting Yocto and OE license checksums #2211 about license checksums
• .bb file routinely include other files. We may need to handle that too: see http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/dropbear for an example
• https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#ref-bb-variables-glos provides an index of the most common variables
• https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#variable-expansion explains variable expansion ${} which is something we also do for Maven POMs
• https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#dependencies explains dependency managenent
• this BSD-license linter tool has a list of variables https://github.com/priv-kweihmann/oelint-adv/blob/3740ed478477877949ef2160b53f323cc8cdd1ea/oelint_adv/const_vars.py#L155 and a parser of .bb files https://github.com/priv-kweihmann/oelint-adv/blob/master/oelint_adv/parser.py which is integration-tested at https://github.com/priv-kweihmann/oelint-adv/blob/master/test.sh#L9
  The author ( @priv-kweihmann ) posted this interesting article about licenses and bitbake at https://bitbakesoda.blogspot.com/2020/03/the-in-compliance-stands-for-automation.html and does not seem to like scancode-toolkit :P for having too many dependencies (but still uses our license-expression)
• https://github.com/openembedded/meta-openembedded/blob/master/contrib/oe-stylize.py is another linter that may contain some useful bits
• https://github.com/meta-rust/cargo-bitbake provides an example of Cargo to bb mappings
• Bitbake's own conf has key fields defaults https://github.com/openembedded/bitbake/blob/master/conf/bitbake.conf
• Bitbake's own parser is likely too complex for our use case and is GPL-licensed https://github.com/openembedded/bitbake/tree/master/lib/bb/parse

[pombredanne]

@priv-kweihmann FWIW, since I have your attention this ticket is about adding a parser for .bb file metadata so that we can collect them as part of a scancode "package" scan e.g. collect and normalize the bitbake recipe data found in a .bb file to the models we use here: https://github.com/nexB/scancode-toolkit/tree/develop/src/packagedcode and https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/models.py

Your linter likely provides a decent base for parsing (though I shall say that I feel it may have too many dependencies if I use only the parser ;) https://github.com/priv-kweihmann/oelint-adv/blob/master/requirements.txt )

[pombredanne]

@chinyeungli ping... Would this be something you can look into?

[priv-kweihmann]

@priv-kweihmann FWIW, since I have your attention this ticket is about adding a parser for .bb file metadata so that we can collect them as part of a scancode "package" scan e.g. collect and normalize the bitbake recipe data found in a .bb file to the models we use here: https://github.com/nexB/scancode-toolkit/tree/develop/src/packagedcode and https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/models.py

Your linter likely provides a decent base for parsing (though I shall say that I feel it may have too many dependencies if I use only the parser ;) https://github.com/priv-kweihmann/oelint-adv/blob/master/requirements.txt )

FWIW if you're not obligated by licensing I would actually recommend to fork the original bitbake parser (which is GPL2) - but sure I can life with using mine too - btw the dependencies shouldn't apply to the parser itself - they are just used for the rules.