dvc run -m/-M: inconsistent behavior with dirs

[nik123]

dvc run with -M flag doesn't accept directories as metrics whereas dvc run with -m flag does. This behavior should be unified.

DVC version:

• 0.94 (snap package)

Additional Information :

• OS: Ubuntu 18.04
• Discord context: https://discord.com/channels/485586884165107732/565699007037571084/717079938506686543
• Possibly related to Adding multiple metrics files at once #2973

Steps to reproduce

1. Initialize git and dvc

2. Create evaluate.py:

import argparse
import os
import json

parser = argparse.ArgumentParser()
parser.add_argument("--metrics_dir", required=True)
args = parser.parse_args()
metrics_dir = args.metrics_dir

if not os.path.exists(metrics_dir):
    os.makedirs(metrics_dir)

with open(os.path.join(metrics_dir, "metrics1.json"), "w") as f:
    json.dump({"acc": 1.0}, f)

with open(os.path.join(metrics_dir, "metric2.json"), "w") as f:
    json.dump({"acc": 0.5}, f)

3. Create new stage with directory as cached metrics output:

$ dvc run -f metrics_cached.dvc -d evaluate.py -m metrics_cached  python3 evaluate.py --metrics_dir metrics_cached
Running command:
        python3 evaluate.py --metrics_dir metrics_cached

To track the changes with git, run:

        git add metrics_cached.dvc .gitignore

4. Create new stage with directory as not cached metrics output:

$ dvc run -f metrics_not_cached.dvc -d evaluate.py -M metrics_not_cached  python3 evaluate.py --metrics_dir metrics_not_cached
Running command:
        python3 evaluate.py --metrics_dir metrics_not_cached
ERROR: failed to run command - directory 'metrics_not_cached' cannot be used as metrics.

Both commands should either accept directories as metrics either discard them

[nik123]

Also found out that dvc run --help explicitly declares support for dirs for both of these options:

  -m <path>, --metrics <path>
                        Declare output metric file or directory.
  -M <path>, --metrics-no-cache <path>
                        Declare output metric file or directory (do not put
                        into DVC cache).

[nik123]

From Discord discussion I assume that -m/-M flags should not accept directories for now.

[efiop]

@nik123 That's correct! Thank you so much for looking into it! 🙏