Threat model for passphrase-protected keys?

[musjj]

The README.md explains:

Passphrase-protected identity files are not necessary for most use cases, where access to the encrypted identity file implies access to the whole system. However, they can be useful if the identity file is stored remotely.

Is there a broader elaboration of the security implications between using passphrase-protected keys vs. non-passphrase-protected keys?

For example, GPG and SSH guides usually recommends setting a passphrase for your keys.

I've always thought that it's a bit unnecessary if you're using full disk encryption anyways. On top of that, most people use some sort of agent to cache the keys, which essentially makes them available to any malicious process that wants to read it. Doesn't that almost completely eliminate the benefits of passphrase protection?

Would love to hear your thoughts on this.