Error cloning with ssh when manually verifying host key and workspace has a space character

[llbonet14]

Jenkins and plugins versions report

Environment

Jenkins: 2.528.3
OS: Windows Server 2022 - 10.0
Java: 21.0.3 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM)
---
allure-jenkins-plugin:2.34.0
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-170.v023de017ccd7
asm-api:9.9-185.va_6c6b_3348b_c3
atlassian-jira-software-cloud:2.0.15
authentication-tokens:1.144.v5ff4a_5ec5c33
blueocean:1.27.25
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-config:1.27.25
blueocean-core-js:1.27.25
blueocean-dashboard:1.27.25
blueocean-display-url:2.4.4
blueocean-events:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-i18n:1.27.25
blueocean-jwt:1.27.25
blueocean-personalization:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-editor:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1268.v044a_87612da_8
build-timeout:1.38
build-token-root:151.va_e52fe3215fc
built-on-column:1.5
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:373.vfe7645102093
cloudbees-bitbucket-branch-source:937.2.2
cloudbees-folder:6.1073.va_7888eb_dd514
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-2
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-210.v7480a_da_70b_9e
conditional-buildstep:1.5.0
config-file-provider:1006.vc7366c201f57
credentials:1453.v9b_a_29777a_b_fd
credentials-binding:702.vfe613e537e88
data-tables-api:2.3.5-1482.vb_47a_3c19f78c
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:457.v0f62a_94f11a_3
docker-workflow:634.vedc7242b_eda_7
durable-task:639.vefb_3d8372f6d
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-19.vc432d923e5ee
envinject:2.926.v69c9b_3896a_96
envinject-api:1.236.v35fd4d7eb_515
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.253.v9b_413168133b_
font-awesome-api:7.1.0-882.v1dfb_771e3278
git:5.8.1
git-client:6.4.0
git-server:137.ve0060b_432302
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1917.v9ee8a_39b_3d0d
global-slack-notifier:1.5
groovy:497.v7b_061a_a_de65d
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427
http_request:1.23
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-10.v9b_7e1d1fc40b_
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jersey2-api:2.47-165.ve7809a_3e87e0
jira:3.20
jjwt-api:0.11.5-120.v0268cf544b_89
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jquery:1.12.4-3
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-173.v596efb_962a_31
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.21.2-66.v6ea_38164b_8a_2
junit:1369.v15da_00283f06
ldap:793.v754d6b_41b_ea_4
locale:614.va_6a_5a_1a_f2b_38
lockable-resources:1438.v3c0f8c9e2060
mailer:525.v2458b_d8a_1a_71
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
mercurial:1323.ve69d2a_db_8a_b_d
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
msbuild:1.37
nodejs:1.6.6
okhttp-api:4.12.0-195.vc02552c04ffd
oss-symbols-api:424.ved751e062911
pam-auth:1.12
parameterized-trigger:873.v8b_e37dd8418f
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-groovy-lib:787.ve2fef0efdca_6
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.38
pipeline-utility-steps:2.20.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
powershell:2.3
prism-api:1.30.0-630.va_e19d17f83b_0
pubsub-light:1.19
resource-disposer:0.25
role-strategy:848.va_a_ea_673cf0b_c
run-condition:243.v3c3f94e46a_8b_
scm-api:724.v7d839074eb_5c
script-security:1385.v7d2d9ec4d909
scriptler:432.v872ed6cd6d5b_
slack:795.v4b_9705b_e6d47
snakeyaml-api:2.5-143.v93b_c004f89de
sse-gateway:1.28
ssh-credentials:361.vb_f6760818e8c
sshd:3.374.v19b_d59ce6610
structs:362.va_b_695ef4fdf9
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
versioncolumn:320.v6b_b_814ca_01f7
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4238.va_6fb_65c1f699
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1559.va_a_533730b_ea_d
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1010.vb_b_39488a_9841
ws-cleanup:0.49

What Operating System are you using (both controller, and any agents involved in the problem)?

Windows Server 2022 - 10.0

Reproduction steps

Step 1: upgrade Git credential plugin to version 6.4.1
Step 2: Build a job that gets the jenkins file from an azure devops repo using ssh

Expected Results

The job starts

Actual Results

There's an error in the jenkins console and the job does not even start

10:18:11  Started by user admin
10:18:12  hudson.plugins.git.GitException: Command "git.exe fetch --tags --force --progress --prune -- origin +refs/heads/develop:refs/remotes/origin/develop" returned status code 128:
10:18:12  stdout: 
10:18:12  stderr: hostname contains invalid characters
10:18:12  fatal: Could not read from remote repository.
10:18:12  
10:18:12  Please make sure you have the correct access rights
10:18:12  and the repository exists.
10:18:12  
10:18:12  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2861)
10:18:12  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2201)
10:18:12  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:638)
10:18:12  	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:408)
10:18:12  Caused: java.io.IOException
10:18:12  	at PluginClassLoader for git//jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:413)
10:18:12  	at PluginClassLoader for scm-api//jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:219)
10:18:12  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:123)
10:18:12  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70)
10:18:12  	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:317)
10:18:12  	at hudson.model.ResourceController.execute(ResourceController.java:101)
10:18:12  	at hudson.model.Executor.run(Executor.java:460)
10:18:12  Finished: FAILURE

Anything else?

Returning to previous version of Jenkins with the upgraded plugin does not solve the issue, but downgrading the plugin to version 6.4.0 solves the issue. Nothing else has changed

Are you interested in contributing a fix?

No response

[MarkEWaite]

I'm unable to duplicate the issue as reported.

I'm running plugins as listed in my plugins.txt. I created 6 jobs to test credentials with Azure DevOps. They all use my private tasks repository and test the following configurations:

• SSH private key with command line git (as indicated by your error message)
• SSH private key with JGit
• SSH private key with JGitApache
• HTTPS with command line git
• HTTPS with JGit
• HTTPS with JGitApache

All of those configurations work as expected. You'll need to provide enough information so that others can duplicate the error.

[MarkEWaite]

Closing as not planned, since I cannot duplicate the issue. I suspect it is a local configuration error. If more details are provided, you are welcome to reopen the issue.

[amainwaring]

I have also noticed this issue when trying to run jobs that have spaces in the filename or folder.

[MarkEWaite]

I have also noticed this issue when trying to run jobs that have spaces in the filename or folder.

Can you confirm that you are seeing the message "hostname contains invalid characters" when you try to run jobs that have spaces in the filename or folder?

I can't duplicate the issue with spaces in the job name. Please provide detailed steps that will allow me to duplicate the issue from a fresh Jenkins installation.

I created a Pipeline job named "git-client-1703 with spaces in job name" and successfully ran the following Pipeline script:

pipeline {
    agent {
        label 'windows'
    }
    stages {
        stage('checkout') {
            steps {
                git credentialsId: 'MarkEWaite-github-username-password',
                    url: 'https://github.com/MarkEWaite/tasks.git'
            }
        }
    }
}

[amainwaring]

Yeah, this looks like what we have. The folder in Jenkins was named like Test folder but we also noticed jobs just with spaces in their names failing too with that same error.

Most of our jobs are passing but there are a few that are failing like this and we're trying to find commonalities. The jobs in the test folder all failed with the trace like above. They are configured for lightweight checkout and appear to failing once the pipeline starts running. In this job we don't even appear to be using any git commands..

12:05:39  Started by user ....
12:05:40  Obtained SqlTools/Build-SqlInstance_AMI.jenkinsfile from git git@github.com:reponame.git
12:05:40  [Pipeline] Start of Pipeline
12:05:40  [Pipeline] node
12:05:40  Running on AWS Modularized i-instanced in D:\jenkins\workspace\DBA\AMI Build\AMI Build
12:05:40  [Pipeline] {
12:05:40  [Pipeline] stage
12:05:40  [Pipeline] { (Declarative: Checkout SCM)
12:05:40  [Pipeline] checkout
12:05:40  The recommended git tool is: git
12:05:40  using credential credential
12:05:40  Cloning the remote Git repository
12:05:40  Cloning repository git@github.com:reponame.git
12:05:40   > git init D:\jenkins\workspace\DBA\AMI Build\AMI Build - SQL 2019 - OS 2019 # timeout=10
12:05:40  Fetching upstream changes from git@github.com:reponame.git
12:05:40   > git --version # timeout=10
12:05:40   > git --version # 'git version 2.52.0.windows.1'
12:05:40  using GIT_SSH to set credentials credential deploy key
12:05:40  Verifying host key using manually-configured host key entries
12:05:40   > git fetch --tags --force --progress -- git@github.com:reponame.git +refs/heads/*:refs/remotes/origin/* # timeout=10
12:05:42  ERROR: Error cloning remote repo 'origin'
12:05:42  hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@github.com:reponame.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
12:05:42  stdout: 
12:05:42  stderr: hostname contains invalid characters
12:05:42  fatal: Could not read from remote repository.
12:05:42  
12:05:42  Please make sure you have the correct access rights
12:05:42  and the repository exists.
12:05:42  
12:05:42  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2861)
12:05:42  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2201)
12:05:42  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:638)
12:05:42  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:880)
12:05:42  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:173)
12:05:42  	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:164)
12:05:42  	at hudson.remoting.UserRequest.perform(UserRequest.java:225)
12:05:42  	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
12:05:42  	at hudson.remoting.Request$2.run(Request.java:391)
12:05:42  	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
12:05:42  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:05:42  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
12:05:42  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
12:05:42  	at java.base/java.lang.Thread.run(Thread.java:840)

[MarkEWaite]

Some questions based on the logged statements

12:05:40 using credential credential
12:05:40 Cloning the remote Git repository
12:05:40 Cloning repository git@github.com:reponame.git

I assume that you replaced the name of your organization and repository with "reponame". Can you confirm that and also confirm that your GitHub organization name does not include spaces and the repository name does not include spaces?

12:05:40 > git init D:\jenkins\workspace\DBA\AMI Build\AMI Build - SQL 2019 - OS 2019 # timeout=10

Can you confirm that Jenkins has a folder named "DBA", another folder named "AMI Build", and a job named "AMI Build - SQL 2019 - OS 2019"?

Can you confirm that none of those folders are Pipelines or organization folders? If they are multibranch Pipelines are organization folders, please describe which of the folders is a Pipeline and which is and organization folder?

You mentioned lightweight checkout. What types of jobs are you using at each of the levels?

12:05:40 Fetching upstream changes from git@github.com:reponame.git

That syntax indicates that you're using SSH to access the repository. Is a private key credential defined in the controller?

12:05:40 > git --version # timeout=10
12:05:40 > git --version # 'git version 2.52.0.windows.1'
12:05:40 using GIT_SSH to set credentials credential deploy key
12:05:40 Verifying host key using manually-configured host key entries

Since you're using manually configured host key entries, can you check that your host key entry for github.com include the correct SSH host keys and that there are no invalid entries in your manually configured host keysd?

12:05:40 > git fetch --tags --force --progress -- git@github.com:reponame.git +refs/heads/:refs/remotes/origin/ # timeout=10
12:05:42 ERROR: Error cloning remote repo 'origin'
12:05:42 hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@github.com:reponame.git +refs/heads/:refs/remotes/origin/" returned status code 128:
12:05:42 stdout:
12:05:42 stderr: hostname contains invalid characters
12:05:42 fatal: Could not read from remote repository.

Is a stack trace logged on the Jenkins controller that provides more information?

[amainwaring]

Thanks! Yeah, i removed some of the key and repo names. Our GitHub repo does not include a space.

Can you confirm that Jenkins has a folder named "DBA", another folder named "AMI Build", and a job named "AMI Build - SQL 2019 - OS 2019"?

yes, that's correct

Can you confirm that none of those folders are Pipelines or organization folders? If they are multibranch Pipelines are organization folders, please describe which of the folders is a Pipeline and which is and organization folder?

They are not multibranch pipelines or organization folders. DBA and AMI Build are both folders, and AMI Build - SQL 2019 - OS 2019 is the name of the Pipeline job.

You mentioned lightweight checkout. What types of jobs are you using at each of the levels?

They are all pipeline jobs, defined in the same repo. The only commonalities we could find is the Jenkins job or folder has a space in it. There are jobs within the DBA folder with spaces in them and those fail. The jobs without spaces succeed. We only have a handful of jobs/folders in this specific sub-folder that have spaces but everything else seems to be functioning properly.

That syntax indicates that you're using SSH to access the repository. Is a private key credential defined in the controller?

Yes, via the AWS Secrets Manager plugin.

Since you're using manually configured host key entries, can you check that your host key entry for github.com include the correct SSH host keys and that there are no invalid entries in your manually configured host keysd?

I've just verified the keys we have with GitHub and have confirmed them. We have them set via CasC so our Jenkins controllers all have the same values.

[MarkEWaite]

Thanks. I'm able to duplicate it in at least one case. Critical parts of duplicating include:

• Must clone with an SSH private key and an SSH protocol git repository (not a problem for HTTPS repositories)
• Must run on a Windows agent (not a problem for non-Windows agents)
• Must have manually provided host keys enabled (does not fail with the default "Accept first connection")
• Must have spaces in the full path to the workspace
• Must have Git Client plugin 6.4.1 (not a problem for Git Client plugin 6.4.0)

[MarkEWaite]

@mikecirioli are you available to help with the investigation of this Git Client plugin 6.4.1 regression on Windows agents with remote host checking with manually provided host keys, spaces in the workspace path, and SSH access to a private repository? I'll spend some time investigating with a debugger on a Windows controller to see if I can identify the root of the failure as well.

In the interim, those affected by the bug have several alternatives:

• Switch Jenkins security from "Manually verified host keys" to "Accept first connection"
• Switch back to git client plugin 6.4.0

[amainwaring]

Thanks. I'm able to duplicate it in at least one case. Critical parts of duplicating include:

• Must clone with an SSH private key and an SSH protocol git repository (not a problem for HTTPS repositories)
• Must run on a Windows agent (not a problem for non-Windows agents)
• Must have manually provided host keys enabled (does not fail with the default "Accept first connection")
• Must have spaces in the full path to the workspace
• Must have Git Client plugin 6.4.1 (not a problem for Git Client plugin 6.4.0)

That's pretty specific! Thanks so much for tracking that down!