Implement generate signature plugin workflow

[qmuntal]

This PR contains the following changes:

• Add support for describe-key command
• Expand generate-signature request to include keySpec, hashAlgorithm and configMap
• Implement plugin signer with generate-signature workflow support
• Unit tests for everything
• Edit: Create spec/v1/plugin and spec/v1/signature package. These contains the Go types as defined in the notary spec. Having this in its own package will help decoupling the spec itself with the crypto-related code and also plugin authors who want to implement the generate-envelope workflow.

This PR still doesn't implement:

• Setting the right contract version in all plugin command
• generate-signature workflow

@SteveLasker @shizhMSFT @gokarnm @jaredpar

[qmuntal]

Plugin spec updated with latest changes in notaryproject/specifications#155

[shizhMSFT]

I think Descriptor should be maintained at the notation level, regardless how plugin protocol works.

[qmuntal]

Any particular reason? The descriptor is part of the signing spec, so IMO it should be in the spec package.

[shizhMSFT]

Then we have the Signer interface to be spec specific, and probably we need to put the Signer into the specs/v1 folder. LoL.

[qmuntal]

Signer is not part of the spec, it's just an interface defined by notation-go, so I would put it in /spec. As we have removed the version subfolders from /spec, then we have no conflict here anymore.

[shizhMSFT]

There are two protocols here. One is at the Signer interface level, another is at the spec level. We probably need a design doc here.

[gokarnm]

IMO types which are inputs and outputs of Sign/Verify methods in top level package should also be in the same package, Descriptor should remain at notation level.

I think spec package is not required, Notation implements one major version of Notary V2 specifications, which can introduce non breaking changes. Breaking changes can be introduced in a new major version of specification and corresponding new major version of Notation.
An exception is the plugin contract spec versioning, which is independent of other versioning, but we need not reflect this in package naming, consumers do not directly interact with plugin contract version.
We should also put most types under internal package, and selectively move then out as publicly visible types as appropriate for Notation CLI and other consumers . Are we planning to use pkg folder for public library types? I was looking at Go lang project layout standard, but not sure how widely it's adopted.

[gokarnm]

There are two protocols here. One is at the Signer interface level, another is at the spec level. We probably need a design doc here.

Can you clarify @shizhMSFT? I didn't understand why two interfaces are required, are you referring to the low level Signer interface to be added in notation-core-go.

[qmuntal]

I think spec package is not required, Notation implements one major version of Notary V2 specifications, which can introduce non breaking changes. Breaking changes can be introduced in a new major version of specification and corresponding new major version of Notation.

I will not tire of repeating that mixing the notation-go module versioning with the Notary versioning is an artificial restriction that will limit notation-go capacity to improve its API. What if in a couple of months, or even years, we find that part of the API is too easy to misuse or has security vulnerabilities? What if we think there is a much better way to implement the Notary spec? We won't be able to react properly if we can't bump the major version of nottion-go.

About having the spec package or putting everything at the root of the module, I think there is value on having a clear separation for types which are defined in the Notary spec and types that are invented by notation-go. I'm not a maintainer of this library, so if you don't see it then I can remove it.

An exception is the plugin contract spec versioning, which is independent of other versioning, but we need not reflect this in package naming, consumers do not directly interact with plugin contract version.
We should also put most types under internal package, and selectively move then out as publicly visible types as appropriate for Notation CLI and other consumers

spec/plugin package is intended for plugin authors consumption, if we move those types into internal they will have to reimplement the whole plugin spec. I can assure you that all types and methods in spec are relevant for consumers, as I'm implementing the notation-azure-kv plugin and notation cli in parallel and I'm just exporting what's necessary.

Are we planning to use pkg folder for public library types? I was looking at Go lang project layout standard, but not sure how widely it's adopted.

Having a package named pkg is not idiomatic, it used to be long time ago, and the current best practice is to place packages at the module root, or stack them with care.

[gokarnm]

I will not tire of repeating that mixing the notation-go module versioning with the Notary versioning is an artificial restriction that will limit notation-go capacity to improve its API. What if in a couple of months, or even years, we find that part of the API is too easy to misuse or has security vulnerabilities?

I agree with the concern, I don't think we have thought enough about versioning of the multiple things (plugins, signature, trust policy) and which ones may reflect in package structure. For example, trust policy document has its own version property, so that policy language in trustPolicy.json can be independently revised without impacting anything else. We'll be making a round of refactoring to get it in line with intended separation between notation, notation-go and notation-core-go, we'll also wait for the rest of the specs to be completed. I'll track this as one of the items in a refactoring issue, and an independent issue to clarify versioning. Do you have few examples of projects that expose a spec and spec version to consumers as part of package structure, /image-spec was one, but that's quite stable.

[qmuntal]

Do you have few examples of projects that expose a spec and spec version to consumers as part of package structure, /image-spec was one, but that's quite stable.

To name a few big ones:

• azure-go-sdk use it all around.
• aws-sdk-go is more stable but also uses it.
• google/go.

[gokarnm]

Added some comments, will cover the rest of changes in another pass.

[shizhMSFT]

LGTM

[shizhMSFT]

There are two protocols here. One is at the Signer interface level, another is at the spec level. We probably need a design doc here.

[SteveLasker]

@gokarnm , is your requested change resolved?

[dtzar]

@SteveLasker on the call yesterday @gokarnm said the changes he made are fine with the code he reviewed so far but wants a couple changes after the PR is merged and he will file issues to track. He only reviewed 1/2 of the PR and will review the rest today.

[gokarnm]

Mostly superficial comments around naming and error messages.

For organizing packages and types, IMO we should not have a spec top level package. I'd keep public methods and corresponding input/output types (e.g. Descriptor) in the same package. Any private types should be moved to internal package, any public plugin types should move to plugin package.

[qmuntal]

Thanks everyone for the feedback. I've finally removed the spec package, as there is no consensus on how to manage the versioning of the spec itself and this discussion is out of the scope of this PR.

[gokarnm]

LGTM. Thanks @qmuntal for implementing this and improving the spec along the way!

[SteveLasker]

LGTM
Great work @qmuntal