ShibAttributeCharacterSetConversionEnabled has no influence on affiliation

[jahbra]

What steps does it take to reproduce the issue?

• When does this issue occur?
  If you get the affiliation via shibboleth and use ShibAttributeCharacterSetConversionEnabled set to true

• Which page(s) does it occurs on?

• What happens?
  affiliation has garbled chars from bad utf-8 conversation

• To whom does it occur (all users, curators, superusers)?
  all users

• What did you expect to happen?
  the following code has to be enabled for affiliation too
  Shib.java:
  if(systemConfig.isShibAttributeCharacterSetConversionEnabled()) {
  attributeValue= new String( attributeValue.getBytes("ISO-8859-1"), "UTF-8");
  }

Which version of Dataverse are you using?
5.9

Any related open or closed issues to this bug report?
no

[pdurbin]

@jahbra thanks for the bug report. This was added by @pallinger in pull request #6956 but I don't know if they're available to help troubleshoot.

The setting is on/true by default. You can prevent the character conversion by setting it to false. Does that help?

[jahbra]

Hi, Thanks for the answer.
Setting it to false did not work If I set it to false everything is not shown correct.
If I set character conversion to true name and everything is shown correct but affiliation is not. So my guess is, that affiliation is treated different and does not benefit from the (attributeValue.getBytes("ISO-8859-1"), "UTF-8") treatment.

[pallinger]

Hello, I will try to test this on our dev server, although I may need some setup to have appropriate test data on the IdP. If you have specific test data that you can share with me, we may be able to whip up a correct conversion function. It may also be that your IdP has different fields in different formats. :\

[jahbra]

Yes, testdata and idp is not easy, if you lack the permissions like I do. Solved it by configuring the shibboleth attribute-map.xml sp in this way:

<Attribute name="urn:oid:2.5.4.10" id="sn"/>
and
curl -X PUT -d "sn" http://localhost:8080/api/admin/settings/:ShibAffiliationAttribute

normally Organisation ("Universität") now surname and affiliation.
The same field for both surname and affiliation
As surname diacritics is displayed correct, as affiliation I get: Universität

[pdurbin]

@pallinger have you had a chance to test this? Thanks!

[PaulBoon]

@pdurbin I just stumbled into a similar problem, having a user with a diacritic in the firstname.
When inspecting the code Shib.java it seems that the affiliations are retrieved via the function getValueFromAssertion and others are using getRequiredValueFromAssertion. And only getRequiredValueFromAssertion is doing the encoding conversion. So that would possibly explain the difference @jahbra discovered for the affiliation versus surname.
The firstname, however, is converted, but not in the DANS forked code, so I still am confused ;-)

For instance when we look at the Universität, if we convert it like this:
echo Universität | iconv -f ISO-8859-1 -t UTF-8
we get Universität, so it looks like it is coverted while it should not?

Anyway, what was the reason to not do the conversion in the getValueFromAssertion?

[pdurbin]

Hmm, it looks like I approved this PR by @pallinger and didn't notice that we should have updated getValueFromAssertion as well:

• Shib attribute character set conversion enabled #6956

@PaulBoon @pallinger would either of you like to make a pull request to fix this? 😅

[PaulBoon]

@pdurbin Just made a DRAFT PR: #9461
Have not tested it with a Shibboleth login (and a diacritic for the affiliation), hoping this is easier for @pallinger to set up.

[pdurbin]

Nice. @pallinger do you have time to test this?

[PaulBoon]

@pdurbin The simplest thing to do was ignoring the UnsupportedEncodingException, but maybe we should let the login fail?
I am asking, because it might be against the way these kind of exceptions are handled in Dataverse.

[pallinger]

@pdurbin I could compile and run the version on our test server, but I could not really test it as I do not have a test IDP that I could use to get iso1-encoded affiliation values. Otherwise, shibboleth login continues to work.
That said, I have examined the pull request, and seems to be okay for me. Very unlikely to cause problems.