Handle IPFS Objects with links

[nginnever]

Currently the README states that this verifier is limited to IPFS objects with no links. This is because the protobuf byte definition will change if an object has defined a links array in the protobuf message.

We may be stuck on being able to verify IPFS objects with links since an IPFS chunk is currently hard coded to 256k, meaning the file is already pretty large if it has links and the input data will cause large overhead in the sha256 hashing. I am interested in seeing what the max input data size is for this verifier, perhaps at peak optimization and only doing sha256 hashing and comparison without conversions.

However if we reduce the fixed size of an IPFS chunk and found the hard limit on data input we could perhaps handle the protobuf definition for an IPFS object with links.

Proposal:

• test for upper bound on data input size (total with links)
  • document gas cost bottleneck(s) (probably sha256)
• create a fixed sized chunker in verifyIPFS.sol (not immediatly obvious how to do this)
  • ipfs fsc: 256k through stream to chunk
  • if data input > fixed chunk size, create links
  • find max length data the contract can handle as a chunk size
• create a method for concatenation of the links protobuf hex

These are just quick thoughts but I would love to take a stab a PR on this if we hammer this out a bit more.

[MrChico]

All great points. A few test objects would also be good for seeing exactly what we can and cannot do at this points. I'm not sure whether sha256 will be the bottleneck though. My money would be on the base 58 conversion.

sha256 (SHA2-256) costs:
60 gas + 12 gas for each word (rounded up) for input data to a SHA2-256 operation.

So if my math is correct we would have 60 + (256000/32)*12 in gas costs for the hashing. Appreciate all PRs. I will provide more analysis of the gas costs and bottlenecks at a later time

[bonedaddy]

Did this end up getting anywhere?

I just started working on a project where being able to verify objects with links would be very useful. I'd be happy to help in any way possible.

[MrChico]

This project has been dormant for a while, so no, there are still no support for linked IPFS objects. Would be happy to include support for links in PRs!

[bonedaddy]

No worries! Can't promise anything but if I figure out a solution I'll submit a PR.