Shibboleth: restore Institution Log In functionality to dataverse.harvard.edu

[pdurbin]

Unfortunately, we saw enough instability (#2013) following the launch of Dataverse 4.0 at https://dataverse.harvard.edu that 14 days later (in the interest of simplifying our setup) we stopped fronting Apache with Glassfish over port 443.

Without Apache (mod_shib) in place, we were forced to disable Shibboleth login. (Because the Shibboleth feature is new, only ~20 accounts were affected, which were converted to builtin/local accounts.)

This ticket is about restoring Shibboleth login which we suspect might require a code change. Various options are being explored in a Google Doc including:

• re-introducing Apache (Address known issues with running Glassfish behind an Apache proxy #2180) assuming concerns are addressed and the configuration is tested
• ...

[pdurbin]

In 5c710d1 we are taking the approach of having Apache serve HTTPS over port 8181 so we can continue to use mod_shib. That port will need to be open in the firewall, of course. Example Apache configuration files can be found here:

https://github.com/IQSS/dataverse/tree/5c710d14c96a20eb027a3948657ef6534c0ae807/scripts/deploy/shibtest.dataverse.org/etc

In Vagrant and other environments, we have used a file at https://github.com/IQSS/dataverse/blob/master/conf/httpd/conf.d/dataverse.conf that includes Shibboleth config and ProxyPass settings but I consider this to be deprecated at this point. I've concentrated all the config at the bottom of /etc/httpd/conf.d/ssl.conf (like others seem to do): https://github.com/IQSS/dataverse/blob/5c710d14c96a20eb027a3948657ef6534c0ae807/scripts/deploy/shibtest.dataverse.org/etc/httpd/conf.d/ssl.conf

This will hopefully give a sense of how ssl.conf has changed:

[root@dvn-vm3 ~]# diff /etc/httpd/conf.d/ssl.conf.orig /etc/httpd/conf.d/ssl.conf
18c18,19
< Listen 443
---
> #Listen 443
> Listen 8181
74c75,76
< <VirtualHost _default_:443>
---
> #<VirtualHost _default_:443>
> <VirtualHost _default_:8181>
78c80,81
< ServerName shibtest.dataverse.org:443
---
> #ServerName shibtest.dataverse.org:443
> ServerName shibtest.dataverse.org:8181
220a224,248
> # We set this header to avoid this:
> # "Failed to download metadata from :8181/Shibboleth.sso/DiscoFeed"
> # Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://shibtest.dataverse.org:8181/Shibboleth.sso/DiscoFeed. This can be fixed by moving the resource to the same domain or enabling CORS.
> Header set Access-Control-Allow-Origin "*"
> 
> # Rewrite all non-Shib traffic to the standard HTTPS port (443)
> # Shib traffic includes:
> # - https://shibtest.dataverse.org:8181/Shibboleth.sso/DiscoFeed
> # - https://shibtest.dataverse.org:8181/Shibboleth.sso/Metadata
> # - https://shibtest.dataverse.org:8181/shib.xhtml (proxied to Glassfish)
> # This will allow users to click "Home" (for example) from
> # shib.xhtml and *not* stay on port 8181 like this:
> # https://shibtest.dataverse.org:8181
> RewriteEngine On
> RewriteCond %{REQUEST_URI} !^/(Shibboleth.sso|shib.xhtml)
> RewriteRule /?(.*) https://%{SERVER_NAME}/$1 [R,L]
> 
> ProxyPass /shib.xhtml ajp://localhost:8009/shib.xhtml
> 
> <Location /shib.xhtml>
>   AuthType shibboleth
>   ShibRequestSetting requireSession 1
>   require valid-user
> </Location>
> 
[root@dvn-vm3 ~]# 

For cosmetic reasons a servlet filter has been introduced so that images show up properly. In testing before the servlet filter was added, images were broken like this:

Please note that because the port has changed, you'll need to re-upload metadata to http://testshib.org/register.html to test with TestShib. Before you do, it is recommended that you adjust the entityId in /etc/shibboleth/shibboleth2.xml to end with "/sp" per #2104.

On any server where we reconfigure Apache to use 8181 Harvard PIN login is expected not to work until we have sent HUIT our updated metadata. I'll handle this in a batch but testing can begin with TestShib per above.

Passing to QA.

[pdurbin]

After chatting with @scolapasta @landreev and @kcondon today I think we've reached a consensus that we're not at all sure that the solution implemented (jumping between ports 443 and 8181) will work in production due to the fact that we load balance between two web servers.

Here's a screenshot that shows the jumping between ports (which also lead to the introduction of a servlet filter):

I'm grabbing this out of QA and will add to the Google doc some of the ideas we've been kicking around (ideas from @whorka too, thanks).