OCPQE-29415:add testing framework based on OTE and the first tests-private case by kuiwang02 · Pull Request #1113 · openshift/operator-framework-olm

kuiwang02 · 2025-09-18T04:29:20Z

Add OLMv0 Testing Framework Based on OTE

Summary

This PR introduces a comprehensive testing framework for OLMv0 (Operator Lifecycle Manager v0) based on OTE (OpenShift Tests Extension). The implementation includes:

Complete testing infrastructure with utility libraries for OLM operations
Test data and fixtures for various OLM scenarios including catalog sources, subscriptions, and operator groups
Comprehensive test utilities for container management, database operations, and cluster interactions
First migrated test case from tests-private to demonstrate the framework functionality

Key Components

🧪 Test Framework Infrastructure

Test utilities (test/qe/util/) providing helpers for:
- OLM operations (catalog sources, subscriptions, operator groups, etc.)
- Container management (Podman, Quay clients)
- Database operations (SQLite support)
- Cluster and node management
- Template processing and YAML handling

📋 Test Data & Fixtures

Extensive test data (test/qe/testdata/olm/) with YAML files covering:
- Catalog source configurations (image, configmap, legacy variants)
- Subscription templates (manual, proxy, environment configurations)
- Operator group definitions
- RBAC configurations and service accounts
- ConfigMaps and custom resources

🏗️ Framework Architecture

OTE integration following OpenShift Tests Extension patterns
Suite definitions for different test execution contexts:
- Parallel/Serial/Slow suites for openshift-tests
- Extended suites for QE-specific testing
Automatic labeling system to distinguish tests from different sources

📚 Documentation

Comprehensive README with migration guidelines and best practices
Test case organization strategy for origin vs tests-private cases
Suite selection logic for different CI environments

Test Result

./openshift-tests run olmv0/extended --monitor watch-namespaces                                                                                                           
  I0918 12:11:06.364645   27240 i18n.go:119] Couldn't find the LC_ALL, LC_MESSAGES or LANG environment variables, defaulting to en_US
  I0918 12:11:06.364799   27240 i18n.go:157] Setting language to default
openshift-tests v4.1.0-9884-g87d58ca
...
INFO[0225] Found 0 must-gather tests                    
started: 0/1/2 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

passed: (0s) 2025-09-18T04:14:55 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

started: 0/2/2 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

passed: (55s) 2025-09-18T04:15:53 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

Shutting down the monitor
Collecting data.
INFO[0287] Starting CollectData for all monitor tests   
INFO[0287]   Starting CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287]   Finished CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287] Finished CollectData for all monitor tests   
Computing intervals.
Evaluating tests.
Cleaning up.
INFO[0287] beginning cleanup                             monitorTest=watch-namespaces
Serializing results.
Writing to storage.
  m.startTime = 2025-09-18 12:14:51.32438 +0800 CST m=+225.047897417
  m.stopTime  = 2025-09-18 12:15:53.424133 +0800 CST m=+287.148314084
Processing monitorTest: watch-namespaces
  finalIntervals size = 4
  first interval time: From = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876; To = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876
  last interval time: From = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751; To = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751
Writing junits.
Writing JUnit report to e2e-monitor-tests__20250918-041150.xml
2 pass, 0 skip (4m3s)
INFO[0290] Successfully removed tmpDir /var/folders/wl/t8n9cyz511q96jx5xvjfwkn40000gn/T/openshift-tests1513432006

Usage
Please refer to README

Assisted-by: Claude Code

openshift-ci-robot · 2025-09-18T04:29:24Z

@kuiwang02: This pull request references OCPQE-29415 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Add OLMv0 Testing Framework Based on OTE

Summary

This PR introduces a comprehensive testing framework for OLMv0 (Operator Lifecycle Manager v0) based on OTE (OpenShift Tests Extension). The implementation includes:

Complete testing infrastructure with utility libraries for OLM operations

Test data and fixtures for various OLM scenarios including catalog sources, subscriptions, and operator groups

Comprehensive test utilities for container management, database operations, and cluster interactions

First migrated test case from tests-private to demonstrate the framework functionality

Key Components

🧪 Test Framework Infrastructure

Test utilities (test/qe/util/) providing helpers for:

OLM operations (catalog sources, subscriptions, operator groups, etc.)

Container management (Podman, Quay clients)

Database operations (SQLite support)

Cluster and node management

Template processing and YAML handling

📋 Test Data & Fixtures

Extensive test data (test/qe/testdata/olm/) with YAML files covering:

Catalog source configurations (image, configmap, legacy variants)

Subscription templates (manual, proxy, environment configurations)

Operator group definitions

RBAC configurations and service accounts

ConfigMaps and custom resources

🏗️ Framework Architecture

OTE integration following OpenShift Tests Extension patterns

Suite definitions for different test execution contexts:

Parallel/Serial/Slow suites for openshift-tests

Extended suites for QE-specific testing

Automatic labeling system to distinguish tests from different sources

📚 Documentation

Comprehensive README with migration guidelines and best practices

Test case organization strategy for origin vs tests-private cases

Suite selection logic for different CI environments

Test Result
./openshift-tests run olmv0/extended --monitor watch-namespaces                                                                                                           
 I0918 12:11:06.364645   27240 i18n.go:119] Couldn't find the LC_ALL, LC_MESSAGES or LANG environment variables, defaulting to en_US
 I0918 12:11:06.364799   27240 i18n.go:157] Setting language to default
openshift-tests v4.1.0-9884-g87d58ca
INFO[0000] Including only extension binaries with image tags: [tests olm-operator-controller] 
...
INFO[0225] Found 0 must-gather tests                    
started: 0/1/2 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

passed: (0s) 2025-09-18T04:14:55 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

started: 0/2/2 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

passed: (55s) 2025-09-18T04:15:53 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

Shutting down the monitor
Collecting data.
INFO[0287] Starting CollectData for all monitor tests   
INFO[0287]   Starting CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287]   Finished CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287] Finished CollectData for all monitor tests   
Computing intervals.
Evaluating tests.
Cleaning up.
INFO[0287] beginning cleanup                             monitorTest=watch-namespaces
Serializing results.
Writing to storage.
 m.startTime = 2025-09-18 12:14:51.32438 +0800 CST m=+225.047897417
 m.stopTime  = 2025-09-18 12:15:53.424133 +0800 CST m=+287.148314084
Processing monitorTest: watch-namespaces
 finalIntervals size = 4
 first interval time: From = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876; To = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876
 last interval time: From = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751; To = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751
Writing junits.
Writing JUnit report to e2e-monitor-tests__20250918-041150.xml
2 pass, 0 skip (4m3s)
INFO[0290] Successfully removed tmpDir /var/folders/wl/t8n9cyz511q96jx5xvjfwkn40000gn/T/openshift-tests1513432006 
Usage

Assisted-by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-09-18T04:30:40Z

@kuiwang02: This pull request references OCPQE-29415 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Add OLMv0 Testing Framework Based on OTE

Summary

This PR introduces a comprehensive testing framework for OLMv0 (Operator Lifecycle Manager v0) based on OTE (OpenShift Tests Extension). The implementation includes:

Complete testing infrastructure with utility libraries for OLM operations

Test data and fixtures for various OLM scenarios including catalog sources, subscriptions, and operator groups

Comprehensive test utilities for container management, database operations, and cluster interactions

First migrated test case from tests-private to demonstrate the framework functionality

Key Components

🧪 Test Framework Infrastructure

Test utilities (test/qe/util/) providing helpers for:

OLM operations (catalog sources, subscriptions, operator groups, etc.)

Container management (Podman, Quay clients)

Database operations (SQLite support)

Cluster and node management

Template processing and YAML handling

📋 Test Data & Fixtures

Extensive test data (test/qe/testdata/olm/) with YAML files covering:

Catalog source configurations (image, configmap, legacy variants)

Subscription templates (manual, proxy, environment configurations)

Operator group definitions

RBAC configurations and service accounts

ConfigMaps and custom resources

🏗️ Framework Architecture

OTE integration following OpenShift Tests Extension patterns

Suite definitions for different test execution contexts:

Parallel/Serial/Slow suites for openshift-tests

Extended suites for QE-specific testing

Automatic labeling system to distinguish tests from different sources

📚 Documentation

Comprehensive README with migration guidelines and best practices

Test case organization strategy for origin vs tests-private cases

Suite selection logic for different CI environments

Test Result
./openshift-tests run olmv0/extended --monitor watch-namespaces                                                                                                           
 I0918 12:11:06.364645   27240 i18n.go:119] Couldn't find the LC_ALL, LC_MESSAGES or LANG environment variables, defaulting to en_US
 I0918 12:11:06.364799   27240 i18n.go:157] Setting language to default
openshift-tests v4.1.0-9884-g87d58ca
INFO[0000] Including only extension binaries with image tags: [tests olm-operator-controller] 
...
INFO[0225] Found 0 must-gather tests                    
started: 0/1/2 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

passed: (0s) 2025-09-18T04:14:55 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

started: 0/2/2 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

passed: (55s) 2025-09-18T04:15:53 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

Shutting down the monitor
Collecting data.
INFO[0287] Starting CollectData for all monitor tests   
INFO[0287]   Starting CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287]   Finished CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287] Finished CollectData for all monitor tests   
Computing intervals.
Evaluating tests.
Cleaning up.
INFO[0287] beginning cleanup                             monitorTest=watch-namespaces
Serializing results.
Writing to storage.
 m.startTime = 2025-09-18 12:14:51.32438 +0800 CST m=+225.047897417
 m.stopTime  = 2025-09-18 12:15:53.424133 +0800 CST m=+287.148314084
Processing monitorTest: watch-namespaces
 finalIntervals size = 4
 first interval time: From = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876; To = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876
 last interval time: From = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751; To = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751
Writing junits.
Writing JUnit report to e2e-monitor-tests__20250918-041150.xml
2 pass, 0 skip (4m3s)
INFO[0290] Successfully removed tmpDir /var/folders/wl/t8n9cyz511q96jx5xvjfwkn40000gn/T/openshift-tests1513432006 
Usage
Please refer to README

Assisted-by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-09-18T06:00:34Z

@kuiwang02: This pull request references OCPQE-29415 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Add OLMv0 Testing Framework Based on OTE

Summary

This PR introduces a comprehensive testing framework for OLMv0 (Operator Lifecycle Manager v0) based on OTE (OpenShift Tests Extension). The implementation includes:

Complete testing infrastructure with utility libraries for OLM operations

Test data and fixtures for various OLM scenarios including catalog sources, subscriptions, and operator groups

Comprehensive test utilities for container management, database operations, and cluster interactions

First migrated test case from tests-private to demonstrate the framework functionality

Key Components

🧪 Test Framework Infrastructure

Test utilities (test/qe/util/) providing helpers for:

OLM operations (catalog sources, subscriptions, operator groups, etc.)

Container management (Podman, Quay clients)

Database operations (SQLite support)

Cluster and node management

Template processing and YAML handling

📋 Test Data & Fixtures

Extensive test data (test/qe/testdata/olm/) with YAML files covering:

Catalog source configurations (image, configmap, legacy variants)

Subscription templates (manual, proxy, environment configurations)

Operator group definitions

RBAC configurations and service accounts

ConfigMaps and custom resources

🏗️ Framework Architecture

OTE integration following OpenShift Tests Extension patterns

Suite definitions for different test execution contexts:

Parallel/Serial/Slow suites for openshift-tests

Extended suites for QE-specific testing

Automatic labeling system to distinguish tests from different sources

📚 Documentation

Comprehensive README with migration guidelines and best practices

Test case organization strategy for origin vs tests-private cases

Suite selection logic for different CI environments

Test Result
./openshift-tests run olmv0/extended --monitor watch-namespaces                                                                                                           
 I0918 12:11:06.364645   27240 i18n.go:119] Couldn't find the LC_ALL, LC_MESSAGES or LANG environment variables, defaulting to en_US
 I0918 12:11:06.364799   27240 i18n.go:157] Setting language to default
openshift-tests v4.1.0-9884-g87d58ca
...
INFO[0225] Found 0 must-gather tests                    
started: 0/1/2 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

passed: (0s) 2025-09-18T04:14:55 "[sig-operator][Jira:OLM] OLMv0 should pass a trivial sanity check"

started: 0/2/2 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

passed: (55s) 2025-09-18T04:15:53 "[sig-operator][Jira:OLM] OLMv0 should PolarionID:22259-[Skipped:Disconnected]marketplace operator CR status on a running cluster[Serial]"

Shutting down the monitor
Collecting data.
INFO[0287] Starting CollectData for all monitor tests   
INFO[0287]   Starting CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287]   Finished CollectData for [Jira:"Test Framework"] monitor test watch-namespaces collection 
INFO[0287] Finished CollectData for all monitor tests   
Computing intervals.
Evaluating tests.
Cleaning up.
INFO[0287] beginning cleanup                             monitorTest=watch-namespaces
Serializing results.
Writing to storage.
 m.startTime = 2025-09-18 12:14:51.32438 +0800 CST m=+225.047897417
 m.stopTime  = 2025-09-18 12:15:53.424133 +0800 CST m=+287.148314084
Processing monitorTest: watch-namespaces
 finalIntervals size = 4
 first interval time: From = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876; To = 2025-09-18 12:14:51.327796 +0800 CST m=+225.051312876
 last interval time: From = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751; To = 2025-09-18 12:15:53.423517 +0800 CST m=+287.147697751
Writing junits.
Writing JUnit report to e2e-monitor-tests__20250918-041150.xml
2 pass, 0 skip (4m3s)
INFO[0290] Successfully removed tmpDir /var/folders/wl/t8n9cyz511q96jx5xvjfwkn40000gn/T/openshift-tests1513432006 
Usage
Please refer to README

Assisted-by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

kuiwang02 · 2025-09-18T07:18:05Z

/hold
to wait #1114

kuiwang02 · 2025-09-18T08:14:14Z

/test okd-scos-e2e-aws-ovn

kuiwang02 · 2025-09-18T08:14:27Z

/test e2e-gcp-ovn

kuiwang02 · 2025-09-19T00:20:07Z

/unhold
because #1114 is merged

kuiwang02 · 2025-09-19T00:22:30Z

/verified by @kuiwang02

openshift-ci-robot · 2025-09-19T00:22:41Z

@kuiwang02: This PR has been marked as verified by @kuiwang02.

Details

In response to this:

/verified by @kuiwang02

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jianzhangbjz · 2025-09-19T00:30:52Z

/approve
/lgtm

openshift-ci · 2025-09-19T00:31:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianzhangbjz, kuiwang02

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~DOWNSTREAM_OWNERS~~ [jianzhangbjz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-09-19T05:03:38Z

/retest-required

Remaining retests: 0 against base HEAD 44c9931 and 2 for PR HEAD c126c41 in total

openshift-ci · 2025-09-19T08:36:21Z

@kuiwang02: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 18, 2025

openshift-ci Bot requested review from ankitathomas and rashmigottipati September 18, 2025 04:31

add testing framework based on OTE and the first tests-private case

c126c41

kuiwang02 force-pushed the initial branch from 162657b to c126c41 Compare September 18, 2025 04:38

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 18, 2025

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 19, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Sep 19, 2025

openshift-ci Bot assigned jianzhangbjz Sep 19, 2025

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 19, 2025

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 19, 2025

openshift-merge-bot Bot merged commit 36ad306 into openshift:main Sep 19, 2025
16 checks passed

kuiwang02 deleted the initial branch September 22, 2025 00:45

Conversation

kuiwang02 commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 18, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 18, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 18, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuiwang02 commented Sep 18, 2025

Uh oh!

kuiwang02 commented Sep 18, 2025

Uh oh!

kuiwang02 commented Sep 18, 2025

Uh oh!

kuiwang02 commented Sep 19, 2025

Uh oh!

kuiwang02 commented Sep 19, 2025

Uh oh!

openshift-ci-robot commented Sep 19, 2025

Uh oh!

jianzhangbjz commented Sep 19, 2025

Uh oh!

openshift-ci Bot commented Sep 19, 2025

Uh oh!

openshift-ci-robot commented Sep 19, 2025

Uh oh!

openshift-ci Bot commented Sep 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kuiwang02 commented Sep 18, 2025 •

edited

Loading

openshift-ci-robot commented Sep 18, 2025 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Sep 18, 2025 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Sep 18, 2025 •

edited by openshift-ci Bot

Loading