DV cannot parse DOIs that don't have a separator in authority/prefix

[tdilauro]

I encountered a serious problem related to retaining our DOIs in Step 4 (batch migrate) of the migration instructions.

The problem appears to originate here. The code is overly restrictive, returning false (error) for perfectly legit DOIs (e.g., doi:10.7281/T1J10120).

This is an urgent bug for us, as I have limited time to get this migration completed and this may be a showstopper.

[djbrooke]

Hey @tdilauro - @scolapasta is running point on migration issues and I'll make sure he checks this out today.

I'll also tag @landreev as it looks like he added code to handle what you mention in a commit that was in 4.5: 7ba9194

Anyway, we'll be in touch as soon as possible.

[djbrooke]

Almost forgot! We should also check with two of the recent upgraders from the Dataverse community - @donsizemore and @4tikhonov - to see if they have any helpful info here.

[pdurbin]

Hi @tdilauro I see from your post at https://groups.google.com/d/msg/dataverse-migration-wg/bFYugiKVvd8/IpsLxRB6AwAJ that you were having trouble migrating datasets with DOIs such as "doi:10.7281/T1J10120" to Dataverse 4.3 but can you please try this on Dataverse 4.6? I'm getting different behavior than you reported there. I'm not getting "IllegalArgumentException: Failed to parse identifier: doi:10.7281/T1J10120" but I also don't want to get your hopes up that DOIs like this are completely supported. I suspect there still may be an issue with DOIs like this. We'll see. 😄

On a related note, I played around with the code a bit and pushed some scratch work to pdurbin@ca6ecb4 that I'd be happy to review with @scolapasta @landreev @sekmiller and others.

[scolapasta]

Hi Tim, I'll be curious to here what happens when you try what Phil suggested, but please let me know how else I can help with anything migration related. (I have a general e-mail I sent out to those groups doing migration, will send to you shortly) Gustavo

…

On Fri, Jan 20, 2017 at 12:09 PM, Philip Durbin ***@***.***> wrote: Hi @tdilauro <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tdilauro&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=HKMPMX7WCKDZ2-0GLyckOPK7gK3k315enz0VXsiO-AM&e=> I see from your post at https://groups.google.com/d/ msg/dataverse-migration-wg/bFYugiKVvd8/IpsLxRB6AwAJ <https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msg_dataverse-2Dmigration-2Dwg_bFYugiKVvd8_IpsLxRB6AwAJ&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=YyeE03Hsn969T_5hEp9Is4elObvN0BJC7GE9eMbUZ7A&e=> that you were having trouble migrating datasets with DOIs such as "doi:10.7281/T1J10120" to Dataverse 4.3 but can you please try this on Dataverse 4.6? I'm getting different behavior than you reported there. I'm not getting "IllegalArgumentException: Failed to parse identifier: doi:10.7281/T1J10120" but I also don't want to get your hopes up that DOIs like this are completely supported. I suspect there still may be an issue with DOIs like this. We'll see. 😄 On a related note, I played around with the code a bit and pushed some scratch work to ***@***.*** <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pdurbin_dataverse_commit_ca6ecb49bb4fb1695e4a87ba54d768e22b2728c1&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=Q4YDnoCoxi7AEQG5w75KC4t1K7TeUsmKjduvroAUI_Y&e=> that I'd be happy to review with @scolapasta <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_scolapasta&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=iPt8g6T78RlGqq7xLYws-eT9fc6uSPIl7ITlXHmahpE&e=> @landreev <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_landreev&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=00UyD91Jl2KiRcN38M_tuh5DEclDeMkNqQtOGFRg8H0&e=> @sekmiller <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sekmiller&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=rqjTEQ6B1eIR_Wah1ckveT4_VWbBOuGG1R5pL09znP4&e=> and others. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_issues_3583-23issuecomment-2D274125417&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=6yjchhqyJ0-9gQih9MhXSaGpPoExSQVC7fddklL0PjY&e=>, or mute the thread <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AEEhBFHhq0SNLF8eYjmSrwwOfELF2E99ks5rUOo4gaJpZM4LpWyJ&d=CwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xgAtlRJhZJjMyKX1J660N0IQRNfpw7XyvKRWo-rQjyo&m=uPO38ER7wWUbGudRmD-sQMN0cuknpns74nc8LnR8TRw&s=hc54IAAx88CmPmQFqaTX17IbeFtErmi1hCF7nRUUacQ&e=> .

[tdilauro]

Thanks @pdurbin, @djbrooke, and @scolapasta.

@scolapasta: A key question for me right now is for which DV versions the migration tools and process are supported. I have performed the migration several times against 4.3 and below, but not of the most current releases that supports migration directly from 3.6.3.

[djbrooke]

Hey @tdilauro - I know that you and @scolapasta have been communicating about this, so feel free to leave any helpful notes or scripts here. If this is something that other installations will run into, we should share the knowledge if helpful. Thanks!

[pdurbin]

@tdilauro and I discussed this issue briefly at http://irclog.iq.harvard.edu/dataverse/2017-02-01#i_47985

[tdilauro]

I was able to work around this issue in 4.3.1 to complete our migration by stripping the DOIs from the 3.6.3-exported DDI and feeding this to batch migration. Since our Handle prefix was not real and not registered, I needed only replace the Handles with their corresponding DOIs in the dataset table once the batch migration was complete. Using the original exported DDI, I was able to create a mapping from the Handles to their corresponding DOIs for each of the datasets. Using the same information, I created a script to rename (mv) the associated directories.

My migration path was 3.6.2 -> 4.3.1, followed by a series of upgrades from 4.3.1 -> ... -> 4.6. I did not have time to test a migration directly from 3.6.2 to 4.6. So, I'm not sure if this issue is resolved or not.

[pdurbin]

@tdilauro I think there's still an issue here. Please take a look at https://github.com/IQSS/dataverse/blob/v4.6/src/test/java/edu/harvard/iq/dataverse/GlobalIdTest.java#L127 for example, which looks like this:

    @Test
    public void testBadIdentifierTwoParts(){
        System.out.println("testBadIdentifierTwoParts");

        exception.expect(IllegalArgumentException.class);
        exception.expectMessage("Failed to parse identifier: doi:2part/blah");
        new GlobalId("doi:2part/blah");
}

@tdilauro you identifiers have two parts (i.e. doi:10.7281/T1J10120) and it sounds like the Dataverse code should be able to parse them.

[pdurbin]

Yesterday @scolapasta mentioned "The shoulder separator should not be part of authority but rather the identifier per examples on EZID site" from #898 which seems related. That issue also references http://ezid.cdlib.org/learn/id_concepts

[tdilauro]

Thanks for making that connection to #898, @pdurbin. I'll add some comments there.