Support raw signing of hashes and aggregated verification

[codablock]

This PR adds support for signHash/verifyHash and verifyAggregatedHashes. The code for aggregated verification is based on https://github.com/herumi/mcl/blob/265b4649f4b456e3fe5fcdce4ca436167adc78e4/include/mcl/aggregate_sig.hpp#L109.

The reason to allow signing of raw hashes is that we don't want to rely on the internals of the BLS library when it comes to hashing. We use internal wrappers around the BLS library which forces us to provide a public key every time we sign something. This public key is than hashed together with the actual message to prevent rogue public key attacks.

verifyAggregatedHashes is useful for us as we do batched verification as much as possible to reduce the number of required pairings.

I also added tests for verifyAggregatedHashes, but noticed that they fail for some reason on BLS12_381. The last check in the test modifies the first input hash and asserts that verification fails. For some reason, this works well on all the other curves but fails on the BLS12_381. Expected behavior would be that verification fails.

I'm not sure if this is related to the verifyAggregatedHashes function or if there is some general error in the BLS12_381 implementation and might need some advise here.

[herumi]

I'll merge this patch, please wait a moment.

[herumi]

@codablock
I have some notifications and questions.

1. There is a possibility that blsSignHash fails for special values because the algorithm of mapToG1.
   af2e557#diff-386ad04c9b4bebb18128e42623ad79b8R75
2. I split verifyAggregatedHashes into blsG1SetHash + verifyAggregation to avoid dynamic allocation in C-api. Do you use only C++ inteface?
   48a7320
3. Why do you check duplicated hashes in blsVerifyAggregatedHashes?

[codablock]

@herumi

1. Can you describe why there are these special values and why they fail to map to G1? And are only these 5 hash values going to fail and every other value will never fail? Do these (or other special values) also exist for other curves?
2. I only use the C interface and have my own C++ wrappers around the C API
3. The check for duplicate hashes is copied from https://github.com/herumi/mcl/blob/265b4649f4b456e3fe5fcdce4ca436167adc78e4/include/mcl/aggregate_sig.hpp#L120. And as of https://crypto.stanford.edu/~dabo/pubs/papers/BLSmultisig.html, under the section "Batch Verification", duplicate hashes are not allowed. My wrappers have checks for this and split batched verification into multiple verifications if required.

[herumi]

@codablock

Can you describe why there are these special values and why they fail to map to G1? And are only these 5 hash values going to fail and every other value will never fail?

There are three special values, i.e., zero, sqrt(-3), -sqrt(-3) in Fp.
The reason of it is by the algorithm of map to G1 function.
zero means n-bit length of \x00\x00...
See the comment of https://github.com/herumi/mcl/blob/master/include/mcl/bn.hpp#L345-L352

hashAndMapToG1 functions ignores the possibility that hashed value becomes the special values.
https://github.com/herumi/mcl/blob/master/include/mcl/bn.hpp#L2071

I only use the C interface and have my own C++ wrappers around the C API

I see.

My wrappers have checks for this and split batched verification into multiple verifications if required.

My aggregate signature implementation uses the hashed value for given message,
then we do not need to worry about modification of the hashed value such as the above the special values,
and we can be sure that there are no same values if we check the hashed values.

But if hashed values are given, we can be sure that there are no same values.
For example, h != h' := (h | (1 << 256)), but Fp::setArrayMask() returns same value for h and h'.
If you check duplication of hashed values before calling blsVerifyAggregation then blsVerifyAggregation
does not need to check it again.

The reason to allow signing of raw hashes is that we don't want to rely on the internals of the BLS library when it comes to hashing.

Internally, the hash function of mcl can be changed.
https://github.com/herumi/mcl/blob/master/include/mcl/fp.hpp#L477-L480
https://github.com/herumi/mcl/blob/master/include/mcl/fp.hpp#L339-L344

There is no C api for bls, is it better to append not verifyHash but the api?

I also added tests for verifyAggregatedHashes, but noticed that they fail for some reason on BLS12_381.
The last check in the test modifies the first input hash and asserts that verification fails.

The difference between BN curve and BLS12 is according to the mapTo point algorithm.
I do not know the good algorithm for BLS12 like for BN, so I use naive algorithm.
https://github.com/herumi/mcl/blob/master/include/mcl/bn.hpp#L418-L432

see https://github.com/herumi/bls/blob/dev/test/bls_test.hpp#L459-L463
The one way to avoid this, see

bad0d52

The probability that squareRoot success per one loop is about 1/2, so if one byte is appended at the bottom of hashed value,
The probability that naiveMapTo(h << 8) != naiveMapTo((h+1) << 8) is about 1-(1/2)^256.

[codablock]

The probability that squareRoot success per one loop is about 1/2, so if one byte is appended at the bottom of hashed value,
The probability that naiveMapTo(h << 8) != naiveMapTo((h+1) << 8) is about 1-(1/2)^256.

Ok I think I understand this solution, it basically shifts the FP element to the left by 8 bits to give the mapping algorithm room for searching y.

While looking into the details I was also wondering why Fp is implemented by masking input values instead of calculating x % order? I assume "masking" means it's basically doing x & ((1<<bitSize) - 1), which does not work for prime orders if I understand it correctly. That's also the reason why your shifting solution works, but it would not work if Fp was implemented using x % order with a prime order (If I understand all this correctly...).

I'm comparing your implementation with another library (https://github.com/Chia-Network/bls-signatures, which is based on https://github.com/relic-toolkit/relic) and try to figure out were these differ and incompatibilities might occur. In relic, the finite fields are of prime order, so I assume all the field arithmetic differs, meaning that produced signatures and public keys very likely would differ as well.

I also figured out that mapping to G1 and G2 for BN curves would differ between your libraries and the relic based ones. Relic does not implement the technique from "Indifferentiable hashing to Barreto Naehrig curves" and only uses the naive implementation. Or does the use of naive mapping instead of the mentioned technique result in the same points on the curves (I assume they would differ)?

[herumi]

While looking into the details I was also wondering why Fp is implemented by masking input values instead of calculating x % order?

I wondered which way to choose (%p or mask), and I chose mask for simplicity.

%p requires long integer division for large data.

I think that it is inevitable that specifications differ according to implementation.

I also figured out that mapping to G1 and G2 for BN curves would differ
between your libraries and the relic based ones.

Yes, the map-to-point algorithm for BN-curve is faster, then I chose it.
Considering various condition between BN381 and BLS12-381,
it is a difficult problem to select which curve for BLS signature.

[codablock]

I assume I can close this PR as you're doing your own implementation now?

FYI: Chia-Network/bls-signatures#19 seems to be an implementation of the Fouque tibouchi mapping but for BLS curves instead of BN curves. Not sure why you choose the naive mapping for the BLS curve, but maybe you can consider using the Fouque tibouchi for BLS as well?

[herumi]

I assume I can close this PR as you're doing your own implementation now?

Does not blsVerifyAggregation verify duplication of hashes?
If you check them before calling this function, then I can append verifyAggregatedHashes for C api.

FYI: Chia-Network/bls-signatures#19 seems to be an implementation of the Fouque tibouchi mapping but for BLS curves instead of BN curves.

I did not know how to implement it for BLS, I'll read it, thank you.

[codablock]

@herumi I'm not sure I understand your question. I see that you have not implemented blsVerifyAggregation with the duplication check. In my code I ensure that I never call this function with duplicated hashes.

[herumi]

In my code I ensure that I never call this function with duplicated hashes.

I see. I was planning to implement blsVerifyAggregatedHashes after hearing your reply.
How about e68b2a3 ?

[codablock]

Looks good :)

[herumi]

I pushed it into master.

[herumi]

I did not know how to implement it for BLS, I'll read it, thank you.

I modified mcl to use calcBN for BLS12 (herumi/mcl@76b13b0) so also changed a test for BLS(1c36a87).

And I'm sorry that I found naiveMapTo is 1.2~1.5 times faster than calcBN.