sfp orchestrator:validate with Code Coverage Validation

[Schuchie]

Describe the bug
If the tests are not run completely in parallel mode and then retested in a retry run and run successfully, sfp ignores the Second Coverage Report. This leads to an error in the validation process.

To Reproduce

1. Not sure how a class could not run in parallel mode, but somehow it occurs in standard controllers
2. run the sfp orchestrator:validate command
3. Two Junit Report files are generated.
4. SFP reads only one and throws an error, that the needed code coverage is not reached.

Expected behavior
SFP reads all unit reports and merges the results.

Screenshots
As shown in the attached screenshot, the retried test class is not listed in the Test-Coverage classes.

Platform Details (please complete the following information):

• OS: ghcr.io/dxatscale/sfpowerscripts:latest
• Version: sfpowerscripts -- The DX@Scale CI/CD Orchestrator -Version:22.6.1 -Release:June 23
• Salesforce CLI(sfdx cli) Version: Same as in the latest sfpowerscripts container
• CI Platform: GitHub

Additional context
Log:

-------------------------------------------------------------------------------------------
  Triggering Apex tests for src-core
  -------------------------------------------------------------------------------------------
  Test Mode: parallel
  Successfully updated apex testing setting
  Tests to be executed: VdmcCustomCodesSelectDTOTest,vdmcCustomCodesSelectTest,VdmcEmailAttachmentTest,vdmcLVPremixPDFControllerTest,VdmcPerformancePackageOverviewDTOTest,VdmcPerformancePackageOverviewTest,VdmcRefundConfirmationControllerTest
  Test Status: Completed:5/7 Queued(0) Failed(0)  
  Test Status: Completed:6/7 Queued(0) Failed(0)  
  Test Status: Completed:6/7 Queued(0) Failed(0)  
  Test Status: Completed:6/7 Queued(0) Failed(0)  
  Test Status: Completed:6/7 Queued(0) Failed(0)  
  Test Status: Completed:6/7 Queued(0) Failed(0)  
  Failed Tests while triggered in parallel: VdmcRefundConfirmationControllerTest
  Triggering tests synchronously: VdmcRefundConfirmationControllerTest
  Successfully updated apex testing setting
  Test Status: Completed:0/1 Queued(0) Failed(0)  
  Test Status: Completed:1/1 Queued(0) Failed(0)  
  Junit Report file available at .testresults/test-result-7075t00001Y0z3A-junit.xml
  Junit Report file available at .testresults/test-result-7075t00001Y0yXa_7075t00001Y0z3A-junit.xml
  === Test Results
                                                                                                  
   Test Name                                                   Outcome   Message   Runtime (ms) 
   VdmcCustomCodesSelectDTOTest.testCodeCoverageDTO            Pass                24           
   vdmcCustomCodesSelectTest.testCustomCodesSelectbackend      Pass                270          
   VdmcEmailAttachmentTest.executeTestWithDoc                  Pass                8478         
   VdmcRefundConfirmationControllerTest.setupTestData          Pass                6362         
   VdmcPerformancePackageOverviewDTOTest.testCodeCoverageDTO   Pass                31           
   VdmcPerformancePackageOverviewTest.testPerfPackageBackend   Pass                4475         
   VdmcRefundConfirmationControllerTest.setupTestData          Pass                6362         
                                                                                                  
  Package type is Source. Validating individual class coverage
  Validating individual classes for code coverage greater than 75 percent
  
  
  === Test Coverage
                                                                 
   Class                                      Coverage Percent 
   VdmcCustomCodesSelectRepository            82               
   VdmcPerformancePackageOverviewRepository   100              
   VdmcCustomCodesSelectController            100              
   VdmcCustomCodesSelectBusiness              88               
   VdmcLVPremixPDFBusiness                    100              
   VdmcLVPremixPDFFacade                      100              
   VdmcLVPremixPDFController                  100              
   VdmcPerformancePackageOverviewDTO          100              
   VdmcCustomCodesSelectFacade                100              
   VdmcPerformancePackageOverviewController   100              
   VdmcLVPremixPDFPersistence                 100              
   VdmcPerformancePackageOverviewBusiness     91               
   VdmcCustomCodesSelectDTO                   100              
   VdmcPerformancePackageOverviewFacade       100              
   VdmcEmailAttachmentVisualforcePage         80               
   VdmcEmailAttachmentRepository              80               
   VdmcEmailAttachmentFacade                  100              
   VdmcEmailAttachmentAction                  100              
   VdmcEmailAttachmentBusiness                88               
   VdmcRefundConfirmationBusiness             0                
   VdmcRefundConfirmationController           0                
   VdmcRefundConfirmationFacade               0                
   VdmcRefundConfirmationPersistence          0                
                                                                 
  The following classes do not satisfy the 75% code coverage requirement:
                                                          
   Class                               Coverage Percent 
   VdmcRefundConfirmationBusiness      0                
   VdmcRefundConfirmationController    0                
   VdmcRefundConfirmationFacade        0                
   VdmcRefundConfirmationPersistence   0                
                                                          
  
  
  
  === Test Summary
                                                                            
   Name                Value                                              
   outcome             Passed                                             
   testsRan            7                                                  
   passing             7                                                  
   failing                                                                
   skipped                                                                
   passRate            100%                                               
   failRate            0%                                                 
   testStartTime       Mon Jul 17 2023 1:43:40 PM                         
   testExecutionTime   37656 ms                                           
   testTotalTime       37656 ms                                           
   commandTime         1173 ms                                            
   hostname            https://.scratch.my.salesforce.com/ 
   orgId               00D5t000000CytJEAS                                 
   username            test@example.com                      
   testRunId           7075t00001Y0yXa_7075t00001Y0z3A                    
   userId              0055t000003cPBAAA2                                 
   packageCoverage     90                                                 
                                                                            
  Error: There are classes that do not satisfy the minimum code coverage of 75%

[azlam-abdulsalam]

@Schuchie it does indeed use the second coverage report. See the code here

sfpowerscripts/packages/core/src/apextest/TriggerApexTests.ts

Line 419 in c8e05ca

if (isCoverageToBeFetched) {

I have seen issues where salesforce sometimes return 0 , could that be the case here?

[Schuchie]

@azlam-abdulsalam as I can see the tests are not returned with a 0, its more like the coverage is splitted into two files. I can see the missing coverage reports of the "failing" classes in the other report with a 100% coverage.

[Schuchie]

So the problem is that the second run replaces only exsiting values and not adding new classes. Problem code is the following. I will try to fix it :)

//`sfpowerscripts/packages/core/src/apextest/TriggerApexTests.ts:413`
 //Replace original test result
 modifiedTestResult.tests = modifiedTestResult.tests.map(
    (obj) => secondTestResult.tests.find((o) => o.methodName === obj.methodName) || obj
);

//Replace original code coverage
if (isCoverageToBeFetched) {
    modifiedTestResult.codecoverage = modifiedTestResult.codecoverage.map(
        (obj) => secondTestResult.codecoverage.find((o) => o.name === obj.name) || obj
    );
}