a port of Mime.processDecoded (to MsgBlocks)

[tomholub]

For decoding mime content, in TS we use this method:

  public static decode = async (mimeMsg: Uint8Array): Promise<MimeContent> => {
    let mimeContent: MimeContent = { attachments: [], headers: {}, subject: undefined, text: undefined, html: undefined, signature: undefined, from: undefined, to: [], cc: [], bcc: [] };
    const parser = new MimeParser();
    const leafNodes: { [key: string]: MimeParserNode } = {};
    parser.onbody = (node: MimeParserNode) => {
      const path = String(node.path.join('.'));
      if (typeof leafNodes[path] === 'undefined') {
        leafNodes[path] = node;
      }
    };
    return await new Promise((resolve, reject) => {
      try {
        parser.onend = () => {
          try {
            for (const name of Object.keys(parser.node.headers)) {
              mimeContent.headers[name] = parser.node.headers[name][0].value;
            }
            mimeContent.rawSignedContent = Mime.retrieveRawSignedContent([parser.node]);
            for (const node of Object.values(leafNodes)) {
              if (Mime.getNodeType(node) === 'application/pgp-signature') {
                mimeContent.signature = node.rawContent;
              } else if (Mime.getNodeType(node) === 'text/html' && !Mime.getNodeFilename(node)) {
                // html content may be broken up into smaller pieces by attachments in between
                // AppleMail does this with inline attachments
                mimeContent.html = (mimeContent.html || '') + Mime.getNodeContentAsUtfStr(node);
              } else if (Mime.getNodeType(node) === 'text/plain' && (!Mime.getNodeFilename(node) || Mime.isNodeInline(node))) {
                mimeContent.text = (mimeContent.text ? `${mimeContent.text}\n\n` : '') + Mime.getNodeContentAsUtfStr(node);
              } else if (Mime.getNodeType(node) === 'text/rfc822-headers') {
                if (node._parentNode && node._parentNode.headers.subject) {
                  mimeContent.subject = node._parentNode.headers.subject[0].value;
                }
              } else {
                mimeContent.attachments.push(Mime.getNodeAsAttachment(node));
              }
            }
            const headers = Mime.headerGetAddress(mimeContent, ['from', 'to', 'cc', 'bcc']);
            mimeContent.subject = String(mimeContent.subject || mimeContent.headers.subject || '');
            mimeContent = Object.assign(mimeContent, headers);
            resolve(mimeContent);
          } catch (e) {
            reject(e);
          }
        };
        parser.write(mimeMsg);
        parser.end();
      } catch (e) { // todo - on Android we may want to fail when this happens, evaluate effect on browser extension
        Catch.reportErr(e);
        resolve(mimeContent);
      }
    });
  }

But @DenBond7 probably already has some method for decoding mime. Therefore I don't want to reimplement the above, but instead use Den's methods (and its outputs) for the above purpose (if possible)

Then you take outputs from the MIME decode, and you feed them into a new method that needs implementing, similar to the one below:

  public static processDecoded = (decoded: MimeContent): MimeProccesedMsg => {
    const blocks: MsgBlock[] = [];
    if (decoded.text) {
      const blocksFromTextPart = MsgBlockParser.detectBlocks(Str.normalize(decoded.text)).blocks;
      // if there are some encryption-related blocks found in the text section, which we can use, and not look at the html section
      if (blocksFromTextPart.find(b => b.type === 'encryptedMsg' || b.type === 'signedMsg' || b.type === 'publicKey' || b.type === 'privateKey')) {
        blocks.push(...blocksFromTextPart); // because the html most likely containt the same thing, just harder to parse pgp sections cause it's html
      } else if (decoded.html) { // if no pgp blocks found in text part and there is html part, prefer html
        blocks.push(MsgBlock.fromContent('plainHtml', decoded.html));
      } else { // else if no html and just a plain text message, use that
        blocks.push(...blocksFromTextPart);
      }
    } else if (decoded.html) {
      blocks.push(MsgBlock.fromContent('plainHtml', decoded.html));
    }
    for (const file of decoded.attachments) {
      const treatAs = file.treatAs();
      if (treatAs === 'encryptedMsg') {
        const armored = PgpArmor.clip(file.getData().toUtfStr());
        if (armored) {
          blocks.push(MsgBlock.fromContent('encryptedMsg', armored));
        }
      } else if (treatAs === 'signature') {
        decoded.signature = decoded.signature || file.getData().toUtfStr();
      } else if (treatAs === 'publicKey') {
        blocks.push(...MsgBlockParser.detectBlocks(file.getData().toUtfStr()).blocks);
      } else if (treatAs === 'privateKey') {
        blocks.push(...MsgBlockParser.detectBlocks(file.getData().toUtfStr()).blocks);
      } else if (treatAs === 'encryptedFile') {
        blocks.push(MsgBlock.fromAttachment('encryptedAttachment', '', { name: file.name, type: file.type, length: file.getData().length, data: file.getData() }));
      } else if (treatAs === 'plainFile') {
        blocks.push(MsgBlock.fromAttachment('plainAttachment', '', {
          name: file.name, type: file.type, length: file.getData().length, data: file.getData(), inline: file.inline, cid: file.cid
        }));
      }
    }
    if (decoded.signature) {
      for (const block of blocks) {
        if (block.type === 'plainText') {
          block.type = 'signedText';
          block.signature = decoded.signature;
        } else if (block.type === 'plainHtml') {
          block.type = 'signedHtml';
          block.signature = decoded.signature;
        }
      }
      if (!blocks.find(block => ['plainText', 'plainHtml', 'signedMsg', 'signedHtml', 'signedText'].includes(block.type))) { // signed an empty message
        blocks.push(new MsgBlock("signedMsg", "", true, decoded.signature));
      }
    }
    return { headers: decoded.headers, blocks, from: decoded.from, to: decoded.to, rawSignedContent: decoded.rawSignedContent };
  }

[tomholub]

This is an interesting method that needs re-implementing, which will help the app decide how to treat each attachment. Here this is some sort of attachment object that has name, size, mime type and similar meta info on it.

  public treatAs = (): Attachment$treatAs => {
    if (['PGPexch.htm.pgp', 'PGPMIME version identification', 'Version.txt', 'PGPMIME Versions Identification'].includes(this.name)) {
      return 'hidden';  // PGPexch.htm.pgp is html alternative of textual body content produced by PGP Desktop and GPG4o
    } else if (this.name === 'signature.asc' || this.type === 'application/pgp-signature') {
      return 'signature';
    } else if (!this.name && !this.type.startsWith('image/')) { // this.name may be '' or undefined - catch either
      return this.length < 100 ? 'hidden' : 'encryptedMsg';
    } else if (this.name === 'msg.asc' && this.length < 100 && this.type === 'application/pgp-encrypted') {
      return 'hidden'; // mail.ch does this - although it looks like encrypted msg, it will just contain PGP version eg "Version: 1"
    } else if (Attachment.encryptedMsgNames.includes(this.name)) {
      return 'encryptedMsg';
    } else if (this.name.match(/(\.pgp$)|(\.gpg$)|(\.[a-zA-Z0-9]{3,4}\.asc$)/g)) { // ends with one of .gpg, .pgp, .???.asc, .????.asc
      return 'encryptedFile';
    } else if (this.name.match(/(cryptup|flowcrypt)-backup-[a-z0-9]+\.(key|asc)$/g)) {
      return 'privateKey';
    } else if (this.type === 'application/pgp-keys') {
      return 'publicKey';
    } else if (this.name.match(/^(0|0x)?[A-F0-9]{8}([A-F0-9]{8})?.*\.asc$/g)) { // name starts with a key id
      return 'publicKey';
    } else if (this.name.toLowerCase().includes('public') && this.name.match(/[A-F0-9]{8}.*\.asc$/g)) { // name contains the word "public", any key id and ends with .asc
      return 'publicKey';
    } else if (this.name.match(/\.asc$/) && this.hasData() && Buf.with(this.getData().subarray(0, 100)).toUtfStr().includes('-----BEGIN PGP PUBLIC KEY BLOCK-----')) {
      return 'publicKey';
    } else if (this.name.match(/\.asc$/) && this.length < 100000 && !this.inline) {
      return 'encryptedMsg';
    } else {
      return 'plainFile';
    }
  }

[DenBond7]

@IvanPizhenko Have you worked with JavaMail before?

[DenBond7]

@tomholub I've looked at TS code. It seems that your logic is clear and can be transferred to Kotlin. There is a lot of work with JavaMail. How I see that

1. Parse inputStream to MimeMessage
2. Analyze the parsed message. Part by part. @IvanPizhenko Please look for example at the following
   

   flowcrypt-android/FlowCrypt/src/main/java/com/flowcrypt/email/jetpack/viewmodel/MsgDetailsViewModel.kt

   Line 389 in 1cb3583

   private suspend fun findEncryptedPart(part: Part): Uri? = withContext(Dispatchers.Default) {

   
   here you will find how I iterate over the parts. Please feel free to look at other examples over the app.
3. During analyzing we can decrypt found parts and generate MsgBlock
4. As I understand in the end we should receive something similar to

data class ParseDecryptedMsgResult constructor(
    @Expose val text: String?,
    @Expose val replyType: String,
    @Expose val subject: String?,
    @SerializedName("error")
    @Expose override val apiError: ApiError?, // here we should have decryptionError
    var msgBlocks: MutableList<MsgBlock>?) {}

[IvanPizhenko]

@IvanPizhenko Have you worked with JavaMail before?

a bit, I've implemented some sending emails in the FES.

[IvanPizhenko]

@tomholub So which one I should continue with now - this one or #1057 ?

[tomholub]

This one please. It's a prerequisite for the other one.

[IvanPizhenko]

ok

[IvanPizhenko]

@tomholub where can I get some Typescript unit tests for this?