CBS script enhancements to manage packages lifecycle and ability to fetch external dependencies by dennis-behm · Pull Request #299 · IBM/dbb

dennis-behm · 2025-01-31T15:25:57Z

This is extending the existing common backend scripts dbbBuild.sh and zBuilder.sh to automatically fetch external dependencies from an artifact repository (instead of any concatenation).

It updates the structure how tar files are store in the Artifact repository.

wazideploy-generate.sh script to enhanced to re-compute the url of the package, to be able to download the the package with the Wazi Deploy provided mechanics.

High-level capabilities for fetch capability:

Read build dependency configuration defined in the Application Descriptor.
Load external dependencies based on naming conventions of the CBS from an artifact repository (Nexus, Artifactory)
Implement a central cache of external packages
Expand the external dependencies in the build workspace for consumption by the build framework

Deployment:

Enable wazideploy-generate to automatically pull the tar from the Artifact repository

Additional changes:

Common backend scripts no longer carry configuration parameters to point to existing scripts that sit in the same repository. It references the scripts within this repo.
ArtifactRepositoryHelpers methods upload and download are enhanced to return their return code

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB

A few comments.

M-DLB · 2025-02-05T08:42:29Z


 # Path to main build script build.groovy.
 zAppBuild="$HOME/dbb-zappbuild"
+# DBEHM: MyCustomization for Path to build.groovy.


Would it make sense to remove this line and the next one?

This will be just for testing for the moment

M-DLB · 2025-02-05T08:43:03Z


 # (Optional) Additional property files to customize the build framework
 zAppBuildPropFiles=""            
+zAppBuildPropFiles="/var/dbb/dbb-zappbuild-config/build.properties,/var/dbb/dbb-zappbuild-config/datasets.properties,/var/dbb/dbb-zappbuild-config/buildFolder.properties"


Would it make sense to remove this line?

M-DLB · 2025-02-05T08:46:46Z

+importFolder.mkdirs()
+
+// setup package cache
+cacheFolder = new File("/tmp/.pkg")


Can we externalize this cache folder location with a property in the pipelineBackend.config?

I have updated the pipelineBackend.config with new settings and implemented the logic for the fetch

M-DLB · 2025-02-05T08:53:43Z

+
+				println("** Downloading application package '$artifactUrl' from Artifact Repository into ${cacheFolder}/${artifactRelPath}.")
+				def rc = artifactRepositoryHelpers.download(artifactUrl, tarFile as String, user, password, true)
+				println "download complete $rc" // TODO: Error handling in helper


Could we have a better info message, and a handling of rc != 0?

M-DLB · 2025-02-05T08:56:32Z

+	}
+}
+
+// Do we actually need this or can this just be obtained from the dependencies?


The Baseline section of the AD file is meant to describe "dependency" on itself for a given application. It can serve two purposes: the first one is downloading the packages from earlier version that can be used as a base for artifacts that are not rebuilt (i.e., private object decks, generated include files from BMS, ...). The second one is to detect the baselineReference tag during an impact build.

I think the next section is necessary. Maybe it can be optimized to be turned into a method, to minimize the common code with the previous section.

I still disagree to use the dependencies section to load our own baseline, we should use the baseline section of the AD file to retrieve the baseline archive from the artifactory repository.

M-DLB · 2025-02-05T08:57:22Z

+	yamlBuilder externalDependencies
+
+	externalDependencyFile = new File(props.externalDependenciesFilePath)
+	externalDependencyFile.withWriter("IBM-1047") { extDepWriter ->


For the sake of clarity, tagging is missing, even if it might not be useful.

Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Co-authored-by: Lauren Li <45975633+lauren-li@users.noreply.github.com>

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB

A couple of changes. I think we're close, but we could simplify it a bit more, especially with not printing "packageUrl=" when calling computePackageUrl. By the way can we rename it to computeArchiveUrl? :)
I've tried to remove occurrences of Package as much as I can and i know I missed a few, if you can have a look, please.

M-DLB · 2025-04-01T11:50:07Z

        dependency.name = applicationDependency
-        dependency.version = version
-        dependency.type = type
+		dependency.type = type


There is a mix a tabs and spaces in the indentation.

Format applied

M-DLB · 2025-04-01T11:50:51Z

- * If an existing baseline for a given branch already exists, the method replaces it
- */
-
-def addBaseline(ApplicationDescriptor applicationDescriptor, String branch, String baseline) {


Why is this method dropped?

I think this is derived from the MM. We don't need it here - @M-DLB do you want me to add it back in? Then we can share the entire utils interchangeability?

Yes please. For easy management, we should have the exact same version than in the MM. You then might want to open a PR on the MM, to update it as well there :)

M-DLB · 2025-04-01T12:56:30Z

-    if [[ ! ${PackageInputFile:0:1} == "/" ]] ; then 
-        checkWorkspace
-        PackageInputFile="$(getLogDir)/${PackageInputFile}"
+    if [ ! ${PackageInputFile:0:1} == "/" ] && [ -z "${usePackageUrl}" ]; then


ArchiveInputFile? I know WD uses packageInputFile...
No need to change...

With introducing the Archive - we already start with a new terminology. We should apply the Wazi Deploy terms here.

Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB

Thanks @dennis-behm for these great enhancements!

…etch external dependencies (IBM#299) * (CBS) packageBuildOutputs.sh to follow implemented naming conventions how archives are published to Artifact repository * (CBS) dbbBuild.sh and zBuilder.sh to process build dependency configuration defined in the Application Descriptor * (CBS) Download external dependencies based on naming conventions from the artifact repository (Nexus, Artifactory) using the ArtifactRepositoryHelpers * (CBS) Expanding the external dependencies in the build workspace for consumption by the build framework * (CBS) Central cache of external archives * (CBS) Enable wazideploy-generate to automatically download the tar from the Artifact repository * (CBS) Scripts no longer carry configuration parameters to point to existing scripts. It references the scripts directly within this repo. --------- Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com> Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com> Co-authored-by: Lauren Li <45975633+lauren-li@users.noreply.github.com>

dennis-behm added 8 commits January 30, 2025 10:47

Fetch build dependencies based on AD

c8ca5db

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Missing import of Utilities

82a3809

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Align fetchDependencies script with Application Descriptor

e9eab81

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

initial documentation

8de7794

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

initial documentation

f584cfc

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Consolidate fetch script for both dbbBuild.sh and zBuilder.sh

9e98a02

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Set execute bits for zBuilder.sh

7b49b4e

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Update release info and add page break

ca23c00

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB reviewed Feb 5, 2025

View reviewed changes

dennis-behm and others added 21 commits February 5, 2025 16:47

Doc updates from code review

98e9de0

Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>

Configure Cache directory

b87255e

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Configure Cache directory

9a1f583

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Configure Cache directory

e923fd2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Configure Cache directory

8499816

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Add dbbCommunityRepoRoot

4a32af5

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Read Application Descriptor via utils

27587cf

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Read Application Descriptor via utils

7e1ac6d

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Read Application Descriptor via utils

88f2df3

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Update Packaging process

2881037

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Update Packaging notes

d4bd7dd

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

packageUtils

4aba150

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Add eyecatcher to build frameworks

43797c7

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

why on earth again

4b25d17

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

bla

2d9da76

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

add buildIdentifier and releaseIdentifier

333c0f7

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Compute publishing parameters for packaging

2eea9b1

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

fix duplicate variable usage

068cc8b

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

introduce packageBuildIdentifier

fd398d2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Refactored PackageBuildOutputs to return url

d09cdac

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

reference PackagingScript

86a5b61

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

dennis-behm mentioned this pull request Mar 25, 2025

Reference cleanup scripts in generateCleanupCommand.sh #307

Merged

README updates

b1bf88e

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

lauren-li reviewed Mar 26, 2025

View reviewed changes

dennis-behm and others added 5 commits March 28, 2025 14:15

Apply suggestions from code review

858591b

Co-authored-by: Lauren Li <45975633+lauren-li@users.noreply.github.com>

numbering in readme

15504fe

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Switch to UTF-8 output

1391d58

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Fix path in ucdPackaging

deeb9bb

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Remove configuration

e4e41f2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB reviewed Apr 1, 2025

View reviewed changes

dennis-behm and others added 12 commits April 1, 2025 17:15

Apply suggestions from code review

275a526

Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>

Use Archive instead of Package

51a0da2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Fix renamed object names

9de0df9

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

sync with latest AD schema

eaca963

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

code formatting

5da0453

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

Updating messages for more consistency

f03c412

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

rename to archiveCache

3739497

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

disable new feature

6134ce3

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

rename to just url

be55b7a

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

manage missing AD definitions

5d73f63

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

fix syntax error

28b88c4

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

fix syntax error

837e847

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB approved these changes Apr 7, 2025

View reviewed changes

dennis-behm merged commit 28eb9a5 into IBM:main Apr 7, 2025
1 check failed

This was referenced Apr 9, 2025

Align Application Descriptor Schema with latest structural updates IBM/dbb-git-migration-modeler#29

Closed

Enhance CBS scripts to pull configured application dependencies #297

Closed

Update pipeline templates to use updated package lifecycle #311

Open

This was referenced Apr 16, 2025

Update application descriptor with latest structural updates IBM/dbb-git-migration-modeler#30

Merged

Enable upload of object decks from build workspace into target libraries IBM/dbb-zappbuild#609

Closed

dennis-behm mentioned this pull request Apr 24, 2025

Resolve and upload object decks from build workspace into target libraries IBM/dbb-zappbuild#610

Merged

Conversation

dennis-behm commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

M-DLB left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

M-DLB left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

M-DLB left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dennis-behm commented Jan 31, 2025 •

edited

Loading