Add couchbase database plugin to spring-cloud-vault by fhitchen · Pull Request #408 · spring-cloud/spring-cloud-vault

fhitchen · 2020-06-22T22:51:53Z

Hi,

I have added a couchbase database plugin, Hashicorp are reviewing it and will make it available at https://github.com/hashicorp/vault-plugin-database-couchbase when ready and I also added it to this package.

Note: this plugin is a database plugin, not a secrets plugin.

To run the two tests, you will need a 6.5.0 or newer Couchbase DB with an additional username "vault-static" added for the second static role test.

Use this command to start Couchbase container

docker run -d --name db -p 18091-18096:18091-18096 -p 11207:11207 -p 8091-8094:8091-8094 -p 11210:11210 fhitchen/vault-couchbase

This container already has the Administrator user configured and a test "bucket" added. The tests do not make use of the bucket.

The plugin is available from my own repo https://github.com/fhitche/vault-plugin-database-couchbase and needs to be build and registered in the sys/plugins/catalog/database/ before the tests will pass.

I had to disable the checkstyle plugin or I got these errors...

INFO] --- maven-checkstyle-plugin:3.1.0:check (checkstyle-validation) @ spring-cloud-vault-config ---
[INFO] Starting audit...
[ERROR] /home/fhitchen/git/spring-cloud-vault/spring-cloud-vault-config/src/test/java/org/springframework/cloud/vault/util/Settings.java:31:1: Utility classes should not have a public or default constructor. [HideUtilityClassConstructor]
[ERROR] /home/fhitchen/git/spring-cloud-vault/spring-cloud-vault-config/src/test/java/org/springframework/cloud/vault/util/CanConnect.java:29:1: Utility classes should not have a public or default constructor. [HideUtilityClassConstructor]
[ERROR] /home/fhitchen/git/spring-cloud-vault/spring-cloud-vault-config/src/test/java/org/springframework/cloud/vault/util/TestRestTemplateFactory.java:40:1: Utility classes should not have a public or default constructor. [HideUtilityClassConstructor]
Audit done.

I could not regenerate the README.adoc as I was unable to locate ./docs/src/main/ruby/generate_readme.sh?

I hope you will be able to approve this pull request.

mp911de

Thanks for your pull request. This PR requires some cleanup and should not touch the buildfile at all. Please check out the comments.

…r the couchbase plugin.

fhitchen · 2020-07-13T18:53:27Z

Hi Mark, I made the changes you requested and asked for a new review. I have not seen any activity so I wondered if I had not done something else that needed to be done? regards, Francis.

…

On Tue, Jun 23, 2020 at 1:55 AM Mark Paluch ***@***.***> wrote: ***@***.**** requested changes on this pull request. Thanks for your pull request. This PR requires some cleanup and should not touch the buildfile at all. Please check out the comments. ------------------------------ In pom.xml <#408 (comment)> : > @@ -235,7 +235,8 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> - <version>2.21.0</version> +  + <version>2.21.0</version> Please refrain from pom.xml changes. This issue is about the Couchbase database plugin. We can sort out any checkstyle warnings later on. ------------------------------ In spring-cloud-vault-config-databases/pom.xml <#408 (comment)> : > @@ -95,6 +95,11 @@ <artifactId>spring-boot-starter-jdbc</artifactId> <scope>test</scope> </dependency> + <dependency> + <groupId>com.couchbase.client</groupId> + <artifactId>java-client</artifactId> + <version>3.0.5</version> + </dependency> This dependency goes into test scope. The version should be inherited from Spring Boot. ------------------------------ In spring-cloud-vault-config-databases/src/main/java/org/springframework/cloud/vault/config/databases/VaultCouchbaseProperties.java <#408 (comment)> : > +public class VaultCouchbaseProperties implements DatabaseSecretProperties { + + /** + * Enable couchbase backend usage. + */ + private boolean enabled = false; + + /** + * Role name for credentials. + */ + private String role; + + /** + * Enable static role usage. + * + * @SInCE 2.2 Note: @SInCE should go into the class documentation. ------------------------------ In spring-cloud-vault-config-databases/src/test/java/org/springframework/cloud/vault/config/databases/CouchbaseSecretIntegrationTests.avaj <#408 (comment)> : > @@ -0,0 +1,113 @@ +/* + * Copyright 2016-2020 the original author or authors. The file suffix is avaj. ------------------------------ In spring-cloud-vault-config-databases/src/test/java/org/springframework/cloud/vault/config/databases/VaultConfigCouchbaseDatabaseStaticTests.java <#408 (comment)> : > + @value("${spring.data.couchbase.password}") + String password; + + Cluster cluster; + + /** + * Initialize the couchbase secret backend. + */ + @BeforeClass + public static void beforeClass() { + + VaultRule vaultRule = new VaultRule(); + vaultRule.before(); + + assumeTrue(CanConnect.to(new InetSocketAddress(COUCHBASE_HOST, COUCHBASE_PORT))); + assumeTrue(vaultRule.prepare().getVersion() In which version did Vault add Couchbase support? These guards are in place as we run tests against multiple Vault versions. ------------------------------ In spring-cloud-vault-config-databases/src/test/java/org/springframework/cloud/vault/config/databases/VaultConfigCouchbaseDatabaseTests.java <#408 (comment)> : > + @value("${spring.data.couchbase.password}") + String password; + + Cluster cluster; + + /** + * Initialize the couchbase secret backend. + */ + @BeforeClass + public static void beforeClass() { + + VaultRule vaultRule = new VaultRule(); + vaultRule.before(); + + assumeTrue(CanConnect.to(new InetSocketAddress(COUCHBASE_HOST, COUCHBASE_PORT))); + assumeTrue(vaultRule.prepare().getVersion() See the previous file's version note for further details. ------------------------------ In src/test/bash/create_certificates.sh <#408 (comment)> : > @@ -67,7 +67,7 @@ chmod 400 ${CA_DIR}/private/localhost.decrypted.key.pem echo "[INFO] Generating server certificate request" openssl req -config <(cat ${DIR}/openssl.cnf \ - <(printf "\n[SAN]\nsubjectAltName=DNS:localhost,IP:127.0.0.1")) \ + <(printf "\n[SAN]\nsubjectAltName=DNS:localhost,IP:127.0.0.1,DNS:chanchan2")) \ Care to elaborate on why this change is? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#408 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABML24DAX6F2ZCKZL6GRGFLRYBGWDANCNFSM4OFDGUXA> .

mp911de · 2020-07-14T07:18:06Z

I was waiting for the signal from your side that you're done. It's not obvious from occasional commits flying into the PR.

fhitchen · 2020-07-14T14:46:26Z

Yes, well I'm learning about spring cloud. i just realized I should use the ... @Autowired private CouchbaseTemplate couchbaseTemplate; Instead of directly coding to the Couchbase SDK in the tests. I will fix that as soon as possible. Thanks for your patience and "stay safe" as they say here. regards, Francis.

…

On Tue, Jul 14, 2020 at 2:18 AM Mark Paluch ***@***.***> wrote: I was waiting for the signal from your side that you're done. It's not obvious from occasional commits flying into the PR. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABML24CHDO3WWEWOOUQIXBTR3QBDZANCNFSM4OFDGUXA> .

fhitchen · 2020-07-15T23:14:47Z

Hi Mark, I'm stuck using the spring-data-couchbase stuff instead of the Couchbase SDK. I have to add the @SpringBootApplication annotation to get the dependency injection to work, but then I hit this error... [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.031 s <<< FAILURE! - in org.springframework.cloud.vault.config.databases. Vault ConfigCouchbaseDatabaseStaticTests [ERROR] initializationError(org.springframework.cloud.vault.config.databases.VaultConfigCouchbaseDatabaseStaticTests) Time elapsed: 0.005 s <<< ERROR! java.lang.IllegalStateException: Test classes cannot include @bean methods Can you help me, I'm stuck. I tried to create my own bean in an inner class but to be honest, I don't really know what I am doing. Can you point me in the correct direction? regards, Francis.

…

On Tue, Jul 14, 2020 at 9:46 AM Francis Hitchens ***@***.***> wrote: Yes, well I'm learning about spring cloud. i just realized I should use the ... @Autowired private CouchbaseTemplate couchbaseTemplate; Instead of directly coding to the Couchbase SDK in the tests. I will fix that as soon as possible. Thanks for your patience and "stay safe" as they say here. regards, Francis. On Tue, Jul 14, 2020 at 2:18 AM Mark Paluch ***@***.***> wrote: > I was waiting for the signal from your side that you're done. It's not > obvious from occasional commits flying into the PR. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#408 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABML24CHDO3WWEWOOUQIXBTR3QBDZANCNFSM4OFDGUXA> > . >

mp911de · 2020-07-16T07:06:48Z

Generally speaking, all @Bean methods should be defined in @Configuration classes. Check out the failure message: java.lang.IllegalStateException: Test classes cannot include @Bean methods

fhitchen · 2020-07-31T15:52:33Z

Hi, I still have not managed to get the tests to work with the @Autowired dependencies. A simple app does mind you so it is just my lack of knowledge about Spring Boot Annotations etc. that is a fault. I have been busy working on getting the Couchbase database plugin into the Hashicorp Git repo. It is finally there if you want to take a look at it here.

However I am truly stuck now as the build process is not working any more. Once Spencer Gibb has fixed it I will start working on it again.

To fix broken build

mp911de · 2020-08-11T10:25:41Z

Any update on this pull request?

fhitchen · 2020-08-11T20:29:37Z

Hi, yes, I just got the maven build to work again (have been busy with Hashicorp getting the plugin into their Git repo) and will try and write a proper test as soon as i figure out the issue i am having with injected dependencies.

fhitchen · 2020-08-13T21:06:09Z

Hi Mark,
Could you do another review now? I decided to stick with the native couchbase test as the Spring wrapping did not let me test that the credentials were correct as all that layer is abstracted away. You need to be able to call the WaitUntilReady method.

I am seeing this in the Couchbase surefire test logs and have no idea why it would be happening.

com.mongodb.MongoSocketOpenException: Exception opening socket
        at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-4.1.0.jar:na]
        at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143) ~[mongodb-driver-co>
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMoni>
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144) ~[mong>
        at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.net.ConnectException: Connection refused (Connection refused)

mp911de

I left a comment about the dependencies. Please also squash your changes into a single commit.

fhitchen · 2020-08-15T14:13:02Z

Hi,
A question before I do my first attempt at squashing. I have had to go back to the upstream/master branch a couple of times so I am a bit confused about which commits I should include in the squash.

Should I squash from f7573ac through 6f2bcab ?

* 6f2bcab (HEAD -> master, origin/master, origin/HEAD) fixed pom.xml
* 9e95a4a Switched to use Couchbase sandbox image.
*   50fc519 Merge commit '895df4456a6d61e3f2afb3679c787c5f29fc86b7' To fix broken build
|\  
| * 895df44 Uses new spring-cloud-starter-bootstrap.
| * 053c054 Ignores .sdkmanrc
| * d65368c Updates to use repo.spring.io/snapshot
| * d1ecef4 Going back to snapshots
| * b8a7610 Update SNAPSHOT to 3.0.0-M3
* | 27cde25 Fixed checkstyle import issues.
* | 1a35a2b Updated dynamic role test to use new role+group creation statement for the couchbase plugin.
* | 922b379 Fixed issues from the PR review.
* | 66bdef6 Fixed issues from the PR review.
* | 2a03cbc Fixed issues from the PR review.
* | 077e525 Fixed issues from the PR review.
* | c1d3edd Fixed issues from the PR review.
* | 7ed535e Added couchbase.
* | 6d587d1 Added note.
* | 511564c Added static role test for Couchbase.
* | 1c94f7f added documentation.
* | 4447b47 Working couchbase solution.
* | 5065b3c Added correct couchbase-database-plugin test file.
* |   7e72257 Merge remote-tracking branch 'upstream/master'
|\ \  
| |/  
| * ef59dcd Adds junit-vintage-engine as a test dependency
| * 5ba60c3 Changed packaging to jar
| * 970c648 Uploading sources for docs
| * bf890e0 Bumping versions
* | 454fbc7 Initial fumblings
* | 17d082b Worked.
* | f7573ac Start adding couchbase db to spring cloud vault
|/  
* 89dd6ab Going back to snapshots

spencergibb · 2020-08-15T14:15:18Z

No need to squash, we can do it when it is merged

fhitchen · 2020-08-15T14:41:54Z

Ok thanks. To test it against an official couchbase db, after you have installed the plugin, instructions here, spin up this docker container using the following command...

$ docker run -d --name db -p 18091-18096:18091-18096 -p 11207:11207 -p 8091-8094:8091-8094 -p 11210:11210 couchbase/server-sandbox:6.5.0

fhitchen · 2020-08-24T14:57:25Z

Are you waiting on anything from me here?

spencergibb · 2020-08-24T15:04:29Z

Mark is on leave right now

fhitchen · 2020-08-24T15:09:08Z

Ok, thanks.

…

On Mon, Aug 24, 2020 at 10:04 AM Spencer Gibb ***@***.***> wrote: Mark is on leave right now — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABML24AXF2Z2YN3KXIBUSDTSCJ6QZANCNFSM4OFDGUXA> .

mp911de · 2020-09-21T08:13:07Z

I'm going to take this PR from here and merge it for 3.0-M4

mp911de · 2020-09-21T10:18:18Z

This pull request is pretty much a mess. The Couchbase integration with Spring Boot never worked as the property names don't match and VaultConfigCouchbaseTests looks like an unfinished copy with references to Cassandra. It took quite some hours to get the tests working. Once this PR is merged, please take a look at the polishing commit.

Original pull request: gh-408.

mp911de · 2020-09-21T11:54:44Z

That's merged and polished now.

Original pull request: gh-408.

Fix property names to spring.couchbase instead of spring.data.couchbase. Fix VaultConfigCouchbaseTests to make it work with Couchbase instead of using Cassandra. Add CouchbaseSecretIntegrationTests to verify interaction through VaultConfigTemplate. Remove property overrides for username/password so that Vault PropertySources are used. Use unique role name to avoid clashes with other tests. Enable VaultConfigCouchbaseDatabaseTests for ConfigData API by importing vault://. Update reference documentation. Resolves gh-408.

fhitchen and others added 10 commits June 1, 2020 16:47

Start adding couchbase db to spring cloud vault

f7573ac

Worked.

17d082b

Initial fumblings

454fbc7

Merge remote-tracking branch 'upstream/master'

7e72257

Added correct couchbase-database-plugin test file.

5065b3c

Working couchbase solution.

4447b47

added documentation.

1c94f7f

Added static role test for Couchbase.

511564c

Added note.

6d587d1

Added couchbase.

7ed535e

mp911de requested changes Jun 23, 2020

View reviewed changes

fhitchen added 5 commits June 23, 2020 11:43

Fixed issues from the PR review.

c1d3edd

Fixed issues from the PR review.

077e525

Fixed issues from the PR review.

2a03cbc

Fixed issues from the PR review.

66bdef6

Fixed issues from the PR review.

922b379

fhitchen requested a review from mp911de June 24, 2020 14:33

Updated dynamic role test to use new role+group creation statement fo…

1a35a2b

…r the couchbase plugin.

fhitchen mentioned this pull request Jul 10, 2020

Adding a Couchbase database plugin hashicorp/vault#9034

Closed

Fixed checkstyle import issues.

27cde25

Merge commit '895df4456a6d61e3f2afb3679c787c5f29fc86b7'

50fc519

To fix broken build

Switched to use Couchbase sandbox image.

9e95a4a

mp911de requested changes Aug 14, 2020

View reviewed changes

Comment thread spring-cloud-vault-config-databases/pom.xml

fixed pom.xml

6f2bcab

mp911de added this to the 3.0.0-M4 milestone Sep 21, 2020

mp911de added the type: enhancement label Sep 21, 2020

mp911de pushed a commit that referenced this pull request Sep 21, 2020

Add support for Couchbase through vault-plugin-database-couchbase

7a2742c

Original pull request: gh-408.

mp911de closed this in a60c519 Sep 21, 2020

spencergibb pushed a commit that referenced this pull request Sep 14, 2023

Add support for Couchbase through vault-plugin-database-couchbase

9161d58

Original pull request: gh-408.

Conversation

fhitchen commented Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mp911de left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fhitchen commented Jul 13, 2020 via email

Uh oh!

mp911de commented Jul 14, 2020

Uh oh!

fhitchen commented Jul 14, 2020 via email

Uh oh!

fhitchen commented Jul 15, 2020 via email

Uh oh!

mp911de commented Jul 16, 2020

Uh oh!

fhitchen commented Jul 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mp911de commented Aug 11, 2020

Uh oh!

fhitchen commented Aug 11, 2020

Uh oh!

fhitchen commented Aug 13, 2020

Uh oh!

mp911de left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fhitchen commented Aug 15, 2020

Uh oh!

spencergibb commented Aug 15, 2020

Uh oh!

fhitchen commented Aug 15, 2020

Uh oh!

fhitchen commented Aug 24, 2020

Uh oh!

spencergibb commented Aug 24, 2020

Uh oh!

fhitchen commented Aug 24, 2020 via email

Uh oh!

mp911de commented Sep 21, 2020

Uh oh!

mp911de commented Sep 21, 2020

Uh oh!

mp911de commented Sep 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fhitchen commented Jun 22, 2020 •

edited

Loading

fhitchen commented Jul 31, 2020 •

edited

Loading