changed robot.txt so s3 has less requests#37

Louis-wr merged 1 commit intodataverse.noDataverseNO/dataverse-docker:dataverse.nofrom

dataverse.no-robotsDataverseNO/dataverse-docker:dataverse.no-robotsCopy head branch name to clipboard