nsfw: fnIngress upgrade

[rosmcmahon]

classifiers must process with some changes,

• without inbox...
• input comes straight to shepherd-input-q with added TxRecord object metadata
• classifiers get "ObjectCreate"-like event
• output now goes to SQS.
• input comes from SQS.
  • format to be determined, but ideally put into shepherd-plugin-interfaces
    • this is actually a bit awkward as the format includes import { S3EventRecord } from 'aws-lambda'
    • just going to have to copy it around or put in libs/ ?
  • SQS queues to be determined from config
• ensure classifiers pass on metadata
• do not delete s3 objects

bugs:

• auto-scaling messed up - wrong q for metric. fixed, now min=>0 also
• testing: (do async and check event logs)
  • 1 msg, out/in
  • 1000, out/in
  • 3000+
  • indexer
• permissions may also be messed up for chained sqses?

[rosmcmahon]

this issue has turned into something a bit more extensive.

• generating linking SQSes for multiple classifiers
• refactor Stacks to move some infra into InfraStack
• completely remove legacy ServicesStack and all it's services 💀 ⚰️
• move the private repo shepherd-next into shepherd main repo as addons/services
• ...more to follow

[rosmcmahon]

move other infra like listsBucket into InfraStack

[rosmcmahon]

• library function in Config to figure out i/o Qs for classifiers
• final Q for http-api (might be blank!)
• need intermediary output to keep an eye on suspicious classified wallets

[rosmcmahon]

made nsfw renamable. just rename the folder it will pick up the rest. ✅

[rosmcmahon]

http-api receiving end

• checks an SQS for input
• sqs message should have TxRecord in metadata
• only delete file data from InputBucket here
• byte-ranges done here <= not if data was received from node, but don’t get hung up on this.

[rosmcmahon]

cleanup

• remove db tables
  • inflights
  • inbox
• remove fnIndex
• remove legacy Services
  • save shepherd-plugin-interfaces first!
  • also any useful/relevant db scripts