8.4.0+ - Issue with jscpd

[VictorRos]

Describe the bug

Hello!
In my team we have a bug with jscpd when migrating from MegaLinter 8.3.0 to 8.4.1.

We enabled debug logs with LOG_LEVEL: DEBUG.

We run the pipeline with the following versions:

• MegaLinter 8.3.0 : ✅
• MegaLinter 8.4.0 : ❌
• MegaLinter 8.4.1 : ❌
• MegaLinter 8.4.2 : ❌

We found an old bug #2450.

But it does not seem to be related as jscpd is in version 4.0.5 and the command run is the same for both MegaLinter 8.3.0 and 8.4.0.

We do not add command options for jscpd.

Here is our configuration file:

{
	"threshold": 10,
	"reporters": ["html", "markdown"],
	"absolute": true,
	"gitignore": true,
	"ignore": [
		"**/node_modules/**",
		"**/.git/**",
		"**/.rbenv/**",
		"**/.venv'/**",
		"**/report*/**",
		"**/*cache*/**",
		"**/*.json",
		"**/*.yaml",
		"**/*.yml",
		"**/*.md",
		"**/*.html",
		"**/*.xml",
		"scripts/pwsh/**",
		"scripts/sh/**"
	]
}

To Reproduce

Steps to reproduce the behavior:

1. Migrate MegaLinter to 8.4.0 or 8.4.1
2. Run pipeline with LOG_LEVEL: DEBUG
3. Scroll down to -COPYPASTE_JSCPD.log
4. See error

Expected behavior

No error with jscpd if threshold is not reached.

It looks like the expected behavior but but for unknown reason it was working 😂

Screenshots

Instead of screenshots, I'm providing jscpd output from MegaLinter

MegaLinter 8.3.0

2025-01-31T15:57:56.3350553Z [Text Reporter] Generated TEXT report: /tmp/lint/megalinter-reports/linters_logs/SUCCESS-COPYPASTE_JSCPD.log
2025-01-31T15:57:56.3477723Z ##[group]�[32m✅ Linted [COPYPASTE] files with [jscpd] successfully - (1.22s)�[0m (expand for details)
2025-01-31T15:57:56.3507970Z - Using [jscpd v4.0.5] https://megalinter.io/8.3.0/descriptors/copypaste_jscpd
2025-01-31T15:57:56.3509195Z - MegaLinter key: [COPYPASTE_JSCPD]
2025-01-31T15:57:56.3511999Z - Rules config: [/.jscpd.json]
2025-01-31T15:57:56.3512416Z - Command: [jscpd --gitignore --reporters console,html --exitCode 1 --output /tmp/lint/megalinter-reports/copy-paste/ -c /tmp/lint/.jscpd.json .]
2025-01-31T15:57:56.3585299Z ##[endgroup]

MegaLinter 8.4.0

2025-01-31T15:47:37.2532288Z [Text Reporter] Generated TEXT report: /tmp/lint/megalinter-reports/linters_logs/ERROR-COPYPASTE_JSCPD.log
2025-01-31T15:47:37.2547104Z ##[group]�[31m❌ Linted [COPYPASTE] files with [jscpd]: Found 2 error(s) - (2.92s)�[0m (expand for details)
2025-01-31T15:47:37.2547608Z - Using [jscpd v4.0.5] https://megalinter.io/8.4.1/descriptors/copypaste_jscpd
2025-01-31T15:47:37.2547999Z - MegaLinter key: [COPYPASTE_JSCPD]
2025-01-31T15:47:37.2548401Z - Rules config: [/.jscpd.json]
2025-01-31T15:47:37.2548807Z - Command: [jscpd --gitignore --reporters console,html --exitCode 1 --output /tmp/lint/megalinter-reports/copy-paste/ -c /tmp/lint/.jscpd.json .]
2025-01-31T15:47:37.2549205Z --Error detail:
2025-01-31T15:47:37.2549488Z Clone found (javascript):
2025-01-31T15:47:37.2549826Z  - /tmp/lint/scripts/js/database-prepare/app.js [90:1 - 99:2] (9 lines, 87 tokens)
2025-01-31T15:47:37.2550160Z    /tmp/lint/scripts/js/elfsight/elfsight.js [24:1 - 35:41]
2025-01-31T15:47:37.2550318Z 
2025-01-31T15:47:37.2550560Z Clone found (javascript):
2025-01-31T15:47:37.2551908Z  - /tmp/lint/scripts/js/database-prepare/app.js [573:3 - 604:19] (31 lines, 164 tokens)
2025-01-31T15:47:37.2552906Z    /tmp/lint/scripts/js/elfsight/elfsight.js [65:3 - 95:2]
2025-01-31T15:47:37.2553427Z 
2025-01-31T15:47:37.2554839Z ┌────────────┬────────────────┬─────────────┬──────────────┬──────────────┬──────────────────┬───────────────────┐
2025-01-31T15:47:37.2556080Z │ Format     │ Files analyzed │ Total lines │ Total tokens │ Clones found │ Duplicated lines │ Duplicated tokens │
2025-01-31T15:47:37.2557265Z ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
2025-01-31T15:47:37.2558423Z │ typescript │ 56             │ 1474        │ 11174        │ 0            │ 0 (0%)           │ 0 (0%)            │
2025-01-31T15:47:37.2560093Z ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
2025-01-31T15:47:37.2561939Z │ javascript │ 9              │ 1151        │ 9253         │ 2            │ 40 (3.48%)       │ 251 (2.71%)       │
2025-01-31T15:47:37.2562749Z ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
2025-01-31T15:47:37.2563401Z │ sql        │ 12             │ 636         │ 3206         │ 0            │ 0 (0%)           │ 0 (0%)            │
2025-01-31T15:47:37.2564213Z ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
2025-01-31T15:47:37.2565741Z │ python     │ 1              │ 52          │ 291          │ 0            │ 0 (0%)           │ 0 (0%)            │
2025-01-31T15:47:37.2566437Z ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
2025-01-31T15:47:37.2567009Z │ powershell │ 2              │ 278         │ 1965         │ 0            │ 0 (0%)           │ 0 (0%)            │
2025-01-31T15:47:37.2567722Z ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
2025-01-31T15:47:37.2568286Z │ Total:     │ 80             │ 3591        │ 25889        │ 2            │ 40 (1.11%)       │ 251 (0.97%)       │
2025-01-31T15:47:37.2569043Z └────────────┴────────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────┘
2025-01-31T15:47:37.2569428Z Found 2 clones.
2025-01-31T15:47:37.2569790Z HTML report saved to /tmp/lint/megalinter-reports/copy-paste/html/
2025-01-31T15:47:37.2570142Z Detection time:: 1.284s
2025-01-31T15:47:37.2570275Z 
2025-01-31T15:47:37.2570507Z ##[endgroup]

Additional context

I run locally jscpd in version 4.0.5 and I have the exact same behavior as MegaLinter.

npx -y jscpd@4.0.5 --gitignore --reporters console,html --exitCode 1 --config .jscpd.json .
echo $?
# 1

EDIT: Tested with MegaLinter 8.4.2, linter jscpd still failed.

[VictorRos]

FYI @snaquekiller

[holtkampjs]

I'm curious if MegaLinter isn't picking up your config file for some reason @VictorRos. What happens if you explicitly supply your config file by setting COPYPASTE_JSCPD_CONFIG_FILE in your .mega-linter.yml file?

EDIT:
TBH, I'm expecting the same output.

[VictorRos]

Hello @holtkampjs
I explicitly added in .mega-linter.yml the following line and it's still failed.

COPYPASTE_JSCPD_CONFIG_FILE: .jscpd.json

Output

[Text Reporter] Generated TEXT report: /tmp/lint/megalinter-reports/linters_logs/ERROR-COPYPASTE_JSCPD.log
❌ Linted [COPYPASTE] files with [jscpd]: Found 2 error(s) - (1.97s)
- Using [jscpd v4.0.5] https://megalinter.io/8.4.2/descriptors/copypaste_jscpd
- MegaLinter key: [COPYPASTE_JSCPD]
- Rules config: [/.jscpd.json]
- Command: [jscpd --gitignore --reporters console,html --exitCode 1 --output /tmp/lint/megalinter-reports/copy-paste/ -c /tmp/lint/.jscpd.json .]
--Error detail:
Clone found (javascript):
 - /tmp/lint/scripts/js/database-prepare/app.js [90:1 - 99:2] (9 lines, 87 tokens)
   /tmp/lint/scripts/js/elfsight/elfsight.js [24:1 - 35:41]

Clone found (javascript):
 - /tmp/lint/scripts/js/database-prepare/app.js [573:3 - 604:19] (31 lines, 164 tokens)
   /tmp/lint/scripts/js/elfsight/elfsight.js [65:3 - 95:2]

┌────────────┬────────────────┬─────────────┬──────────────┬──────────────┬──────────────────┬───────────────────┐
│ Format     │ Files analyzed │ Total lines │ Total tokens │ Clones found │ Duplicated lines │ Duplicated tokens │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ typescript │ 56             │ 1474        │ 11174        │ 0            │ 0 (0%)           │ 0 (0%)            │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ javascript │ 9              │ 1151        │ 9253         │ 2            │ 40 (3.48%)       │ 251 (2.71%)       │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ sql        │ 12             │ 636         │ 3206         │ 0            │ 0 (0%)           │ 0 (0%)            │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ python     │ 1              │ 52          │ 291          │ 0            │ 0 (0%)           │ 0 (0%)            │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ powershell │ 2              │ 278         │ 1965         │ 0            │ 0 (0%)           │ 0 (0%)            │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ bash       │ 1              │ 8           │ 9            │ 0            │ 0 (0%)           │ 0 (0%)            │
├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ Total:     │ 81             │ 3599        │ 25898        │ 2            │ 40 (1.11%)       │ 251 (0.97%)       │
└────────────┴────────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────┘

[VictorRos]

From jscpd CLI options:

npx -y jscpd@4.0.5 --help
...
--exitCode [number]        exit code to use when code duplications are detected
...

IMO, it looks like that jscpd is now working correctly, as intendend.
And that was not the case until now? Who knows? 😂 😂

If what I say turns out to be correct, then I propose to remove from the command line run in the Docker image the hardcoded --exitCode 1.
Because as soon as we have at least a duplicate detected, linter will exit with code 1 even if threshold is not reached.
This is what I see.

Unless --exitCode 1 is removed, I need to override it by adding in .mega-linter.yml the following line:

COPYPASTE_JSCPD_ARGUMENTS: --exitCode 0

It looks like by design, the options provided are added after the hardcoded ones so my --exitCode 0 is last and therefore is the one used by the CLI.

What do you think? @holtkampjs , @nvuillam

[Perdiga]

I have the same issue here, but what I found I that when you add --exitCode 0 mega lint will not output anything, but if you run it locally it will show the results

This is the envs that I am using

   ENABLE_LINTERS: COPYPASTE_JSCPD 
   COPYPASTE_JSCPD_ARGUMENTS: "--threshold 20 --exitCode 0 "
   COPYPASTE_JSCPD_COMMAND_REMOVE_ARGUMENTS: "-c /action/lib/.automation/.jscpd.json --exitCode 1"

I was expecting that in both situations I would get the correct summary. So I think that is more than change the exitCode

[nvuillam]

Thank you all for your analysis :)

It seems that the behavior change is from jscpd itself, not MegaLinter

I'll try to make a PR without --exitCode forced argument to see if there are regressions with MegaLinter internal test cases