Add multivalue genre tag support

[antlarr]

These changes allow beets to store multiple genre tags for multivalues, with the intent of fixing #505 . Note that this is not ready to be merged yet, but it's a beginning of discussion on how to properly fix the issue.

Note that these commits store lists in album.genre and item.genre properties, while storing them as text strings in the database using a separator. Also, I checked that if a user has something like this as path definition:

paths:
    default: $albumartist/$genre/$album%aunique/$genre/$track - $title

then $genre is formatted as a string correctly . In my test case, creating a directory called "Rock;Blues" .

This can break some complex configurations. For example, I had this as album_fields definition to add a album_genre_for_path tag that would contain the most used genre in an album:

album_fields:
    album_genre_for_path: |
        if 'album_genre' in globals():
            return album_genre.title()
        if hasattr(items[0], 'album_genre'):
            return items[0].album_genre.title()
        genres = {}
        cnt, candidate = 0, ''
        for item in items:
            genres[item.genre] = genres.get(item.genre, 0) + 1
            if genres[item.genre] > cnt:
                cnt, candidate = genres[item.genre], item.genre.title()
        return candidate

When evaluating this tag, beet broke sine item.genre is no longer hashable. I think this is a step better than silently breaking the directory structure. I fixed it easily by replacing the code above with:

    album_genre_for_path: |
        if 'album_genre' in globals():
            return album_genre.title()
        if hasattr(items[0], 'album_genre'):
            return items[0].album_genre.title()
        genres = {}
        cnt, candidate = 0, ''
        for item in items:
            if isinstance(item.genre, list):
                values = item.genre
            else:
                values = [item.genre]
            for genre in values:
                genres[genre] = genres.get(genre, 0) + 1
                if genres[genre] > cnt:
                    cnt, candidate = genres[genre], genre.title()
        return candidate

Then I also tested what happens when reading a file written by another application with multiple genre tags:

Python 3.6.1 (default, Mar 23 2017, 13:04:44) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import beets.mediafile
>>> mf=beets.mediafile.MediaFile('01 - Big Brother & the Holding Company -  Bye, Bye Baby.flac')
>>> mf.genre
['Rock', 'Album Rock']

Which is what I would have expected.

[mention-bot]

@antlarr, thanks for your PR! By analyzing the history of the files in this pull request, we identified @sampsyo, @Kraymer and @brunal to be potential reviewers.

[antlarr]

I tested this with flac and mp3 files (that is, with APE tags, where multiple genre tags are written and ID3 tags, where \0 bytes are used to separate genres in a single ID3 tag). Everything works as expected due to ListMediaField (btw, congratulations to whoever implemented it and the *ListStorageStyle classes, they are really nicely done)

The remaining questions are:

• Should a global parameter be added to the configuration? If yes, should it be named multivalue_separator or ... ? Also should it default to ';' ? '; ' ? ', ' ? This should be also set as the default value for tmpl_first, so it behaves correctly when handling the genre tag.
• What to use for queries in the StringList type? I think using SubstringQuery is probably "good enough", but that can be discussed as it could return some unexpected matches . I'm thinking on someone querying for "ck;Bl" for example, but on the other side, maybe someone would like to query for "Rock;Blues" to get items with both values (although that wouldn't match genres stored as "Blues;Rock" which should be considered ).
• Should a config option be added (in the mood of id3v23), like ... use_single_tags_with_separator (defaulting to False) just in case any user prefers to use the current behavior in case (s)he is using the music files with another application that doesn't support multivalue genre tags yet?

[sampsyo]

Very cool! This is diff is surprisingly small—I expected this to be a far more disruptive change.

A couple of ideas come to mind for improving backwards compatibility, neither of which we necessarily have to use:

• We could keep a field called genre that just gets the joined string, and add a new genres field for the list. That's what MediaFile previously did.
• We could somehow allow a string to be assigned into the genre field, which would then be treated as a single-element list.

The only thing that makes me somewhat uncomfortable here is the age-old slippery slope question. How many other fields would it make sense to transform into lists? Should artist be a list? albumartist? If we make too many changes like this, will we disrupt simpler code that just cares about a single value?

Anyway, genre is clearly the best use case for this, so maybe it's not worth worrying too much about other tags.

[antlarr]

Btw, I guess you'll doubt that getall('TIPL') can be replaced with get('TIPL') (mainly because I doubted about it too :) ). So here is the test I did with mutagen to check that it doesn't allow two TIPL frames:

>>> import mutagen
>>> m=mutagen.File('full.mp3')
>>> m.tags.get('TIPL')
>>> frame=mutagen.id3.Frames['TIPL'](encoding=mutagen.id3.Encoding.UTF8, people=[['aa','bb'],['cc','dd']])
>>> m.tags.add(frame)
>>> m.save()

Here, I checked with okteta (an hex editor) that the file included the added TIPL frame.

>>> frame2=mutagen.id3.Frames['TIPL'](encoding=mutagen.id3.Encoding.UTF8, people=[['ee','ff'],['gg','hh']])
>>> frame
TIPL(encoding=<Encoding.UTF8: 3>, people=[['aa', 'bb'], ['cc', 'dd']])
>>> frame2
TIPL(encoding=<Encoding.UTF8: 3>, people=[['ee', 'ff'], ['gg', 'hh']])
>>> m.tags.add(frame2)
>>> m.tags.getall('TIPL')
[TIPL(encoding=<Encoding.UTF8: 3>, people=[['ee', 'ff'], ['gg', 'hh']])]
>>> m.save()

and here, I checked again with an hex editor that the frame wasn't just added, but it replaced the previous TIPL frame.

[sampsyo]

Things are coming along nicely! Here are a few more comments on the latest.

[sampsyo]

This, for what it's worth, is one of the things that worries me about making arbitrary fields into lists: now client code will frequently need to have boilerplate for the form if isinstance(value, list):. This sort of thing puts additional weight on the side of having an explicit way to access lists, or some other foolproof conversion to produce a scalar value.

[antlarr]

I understand what you mean. Note that list fields (genre, arranger ... ) now are always lists, the problem comes with code like that which tries to work with all fields the same way. Would you prefer that I create a class that inherit from lists and reimplements __str__ so it returns '; '.join(self) ? That way code like that could be changed to:

if value:
    formatted[key] = str(value)

But still, that's not perfect neither.

[sampsyo]

That's odd—I'm a little surprised that ListStorageStyle is allowed to return None from get. That seems like it makes things more complicated for clients in general. Maybe that's worth fixing?

[antlarr]

In fact, I'd say it's ok. get is implemented as:

          try:
              return self.get_list(mutagen_file)[0]
          except IndexError:
              return None

So None is returned when get_list returns an empty lists, that is, when the tag doesn't exist in the file. This allows to identify when a tag doesn't exist and allows to remove a tag from a file. Do you think that it would be better to return an empty string and always interpret an empty value as a non-existant tag? (so setting ... say artist to '' would delete the tag entry)

[antlarr]

I just noticed I had this written in a browser tab but forgot to click comment. I'll post it for the records, even if it's probably old now.

Very cool! This is diff is surprisingly small—I expected this to be a far more disruptive change.

Yes, I also was surprised, but most of the work was already implemented in ListMediaField, the *ListStorageStyle classes and mutagen.

A couple of ideas come to mind for improving backwards compatibility, neither of which we necessarily have to use:

We could keep a field called genre that just gets the joined string, and add a new genres field for the list. That's what MediaFile previously did.

Yes, I saw that, but the genres tag is (afaik) not used by any other application, but genre is.

We could somehow allow a string to be assigned into the genre field, which would then be treated as a single-element list.

Have a look at the test_write_genre_list_from_string test I added in test_mediafile.py to test fad5809 :). I think that's even better since it allows assigning a string but automatically converts it to a list.

The only thing that makes me somewhat uncomfortable here is the age-old slippery slope question. How many other fields would it make sense to transform into lists? Should artist be a list? albumartist? If we make too many changes like this, will we disrupt simpler code that just cares about a single value?

Hmm, I don't think that would be too much work. I can give it a try if you want. (This is now done)

[antlarr]

@sampsyo Do you think it's worth it to add a get_primary_key_field to Model so instead of hardcoding id I call self.model_cls.get_primary_key_field() which would iterate over its fields and search which has a PRIMARY_ID type? I think that would be more elegant, but I'm not sure if it's worth the effort.

[antlarr]

@sampsyo should I still allow Query objects to maybe return 2-tuples just in case any external plugin still uses the old API? It's easy to check for unpack errors but it's a bit ugly, so I preferred to ask your opinion.

[arcresu]

@antlarr thanks for your work on this PR. It's been a while since it's had any attention, but it seems that multi-valued tags are still a feature that people would like to see!

Just a heads up that we're planning on splitting MediaFile out into a standalone Python module & git repository during the current beets release cycle, and this is clearly going to affect this PR. Your branch is anyway conflicting with master already, but after the split it'll mean that the changes will also need to be split into two separate sets. Are you still interested in working on this? If so it might be sensible to start with the MediaFile changes at beetbox/mediafile.

[jtpavlock]

Closing as no response from the original author. There's still a lot of good info to be gained from this, but any further implementation ideas should be directed to #505. Also, I think it'd be ideal to first implement the basis for multi-value tags, and then add genres as an additional PR after.