Skip to content

only run regex replaces on DBs with UTF8 encoding #8278#8279

Merged
kcondon merged 1 commit intodevelopfrom
8278-invalid-utf8-only
Dec 7, 2021
Merged

only run regex replaces on DBs with UTF8 encoding #8278#8279
kcondon merged 1 commit intodevelopfrom
8278-invalid-utf8-only

Conversation

@pdurbin
Copy link
Member

@pdurbin pdurbin commented Dec 3, 2021

What this PR does / why we need it:

Deployment fails if the encoding is SQL_ASCII. Here we only run the regex replaces if the database is UTF8.

Which issue(s) this PR closes:

Closes #8278

Special notes for your reviewer:

Devs will need to delete the old "V5.8.0.2__8018-invalid-characters.sql" SQL migration from their flyway table because the checksum is changing.

Suggestions on how to test this:

Make sure the replaces still happen

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No.

Is there a release notes update needed for this change?:

No.

Additional documentation:

No

@scolapasta
Copy link
Contributor

@pdurbin For the PR, you should be able to just use "replace" function instead of reg_exp replace and not need all the extra logic checking encoding.

@pdurbin
Copy link
Member Author

pdurbin commented Dec 6, 2021

@scolapasta I tried just replace (please see below) and I got the same error:

dvndb=# UPDATE datasetfieldvalue SET value = replace(value, U&'\FFFE', '', 'g');
ERROR:  conversion between UTF8 and SQL_ASCII is not supported
LINE 1: ...E datasetfieldvalue SET value = replace(value, U&'\FFFE', ''...
                                                             ^
dvndb=# \l
                             List of databases
   Name    |  Owner   | Encoding  | Collate | Ctype |   Access privileges   
-----------+----------+-----------+---------+-------+-----------------------
 dvndb     | dvnapp   | SQL_ASCII | C       | C     | 

@pdurbin pdurbin assigned scolapasta and unassigned pdurbin Dec 6, 2021
@djbrooke djbrooke added this to the 5.9 milestone Dec 6, 2021
@kcondon kcondon self-assigned this Dec 7, 2021
@kcondon kcondon merged commit 5897231 into develop Dec 7, 2021
@kcondon kcondon deleted the 8278-invalid-utf8-only branch December 7, 2021 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ERROR: conversion between UTF8 and SQL_ASCII is not supported

4 participants