Output logical messages sent via pg_logical_emit_message by dpirotte · Pull Request #20 · eulerto/wal2json

dpirotte · 2017-06-12T01:32:58Z

In 9.6, the pg_logical_emit_message function sends arbitrary data to WAL for logical decoding. One can leverage this infrastructure to transactionally send application/domain events along with "change data capture" events. The events are presented like so:

{"change":[{"kind":"message","transactional":"1","prefix":"foo","content":"bar"}]}

This change supports transactional messages and ignores non-transactional messages. (I left "transactional" in the JSON anticipating support for non-transactional messages in the future.)

This is useful for me, and I hope it is useful for others as well. Feedback welcome on any/all aspects.

`change` now includes messages sent via pg_logical_emit_message when used transactionally. (i.e. first arg is true)

This helps easily build against several versions of Postgres.

dpirotte · 2017-07-19T04:41:35Z

@eulerto @dvarrazzo Any thoughts on this? I fixed a build issue against < 9.6 and added support for non-transactional logical messages.

dvarrazzo

@dpirotte it's a cool feature, I didn't know of pg_logical_emit_message.

I am not an active participant to this project: my needs were surpassing the scope of this project so I've put together replisome which has not been released yet and is being used experimentally by my company (ideally it should be released when the experimental projects it is part of will become production).

The codebase has largely diverged so it won't be possible to apply your MR as it is, however the feature is cool indeed and I will probably backport it (or happily accept your contribution :).

dvarrazzo · 2017-07-19T08:29:21Z

+		  delete3 delete4 savepoint specialvalue toast bytea
+
+PG_VERSION ?= 9.6
+PG_CONFIG = /usr/lib/postgresql/${PG_VERSION}/bin/pg_config


This is pretty hardcoded. You should allow overriding PG_CONFIG from outside and use pg_config --version to get the version number.

dvarrazzo · 2017-07-19T08:32:09Z

+PG_VERSION ?= 9.6
+PG_CONFIG = /usr/lib/postgresql/${PG_VERSION}/bin/pg_config
+
+ifeq ($(PG_VERSION),9.6)


This is not future proof, it shoud be >= 9.6, which is a bit painful in Makefile but doable: see https://github.com/dvarrazzo/pgparts/blob/master/Makefile (see PG_CONFIG, PGVERSION, INTVERSION).

dvarrazzo · 2017-07-19T08:36:27Z

+
+-- Two messages
+BEGIN;
+SELECT 'emit' FROM pg_logical_emit_message(true, 'foo', 'bar');


ISTM there is no test for bytea test. I'd check if all the bytes from 0x00 to 0xFF are sent ok.

dvarrazzo · 2017-07-19T08:38:03Z

+	if(data->pretty_print)
+	{
+		appendStringInfoString(ctx->out, "\t\t\t\"kind\": \"message\",\n");
+		appendStringInfo(ctx->out, "\t\t\t\"transactional\": \"%d\",\n", transactional);


transactional is a bool, JSON has bool... I don't think this value should be the string "0" or "1".

dpirotte · 2017-07-19T14:22:02Z

@dvarrazzo Thanks for the feedback! I'll incorporate as time permits and take a look at modifying for replisome.

* Allow passing in PG_CONFIG instead and infer PG_VERSION from there * Run `pg_logical_emit_message` tests if >= 9.6 Note: The version check works still needs refactoring. It works against `9.6`, `10.0`, but not `10beta2`. `dpkg --compare-versions` works beautifully on Debian flavors, but it would be uncouth to require dpkg.

* Write `true`/`false` instead of "1"/"0" when emitting messages.

This adds a test around bytea output. I don't think is quite correct, and I believe we actually want to force escape or hex output.

Non-transactional messages don't show up in WAL right away. They show up after short wait (< 0.5s) or with the next commit. This test simply demonstrates the former case.

I make an assumption that any letter in `pg_config --version` will be the start of alpha, beta, or rc. Simply axe all of that and replace with `.0`, i.e `10beta2` => `10.0`.

This feature allows software to insert data into WAL stream that can be read by wal2json. Those messages could be useful to control replication, for example. Messages can be sent as transactional or not. Non-transactional messages mean that it is sent even if the transaction is rollbacked. There was a PR #20 for this same feature but I didn't use it. Indeed, this code was dusty in my computer for a few months. NOTE: 'message' test will fail on <= 9.5 because this feature was coded in 9.6 (I don't want to complicate Makefile).

eulerto · 2017-08-22T23:14:34Z

@dpirotte I committed (645ab69) a similar feature although I have not used your PR. Let me know if it works for you.

dpirotte added 4 commits June 11, 2017 11:43

Output pg_logical_emit_message as a change

8af2eb6

`change` now includes messages sent via pg_logical_emit_message when used transactionally. (i.e. first arg is true)

Write correctly-sized logical message content

baeb2ca

Fix formatting

cc674bb

escape the logical message prefix

5d1d067

dpirotte force-pushed the pg_logical_emit_message branch from 594d674 to f222f15 Compare July 19, 2017 03:11

missed a guard around not having pg_logical_emit_message (i.e. < 9.6)

9cfeaf5

dpirotte force-pushed the pg_logical_emit_message branch from f222f15 to 9cfeaf5 Compare July 19, 2017 03:12

dpirotte added 2 commits July 18, 2017 22:39

Makefile now uses PG_VERSION environment variable to look for pg_config

404e252

This helps easily build against several versions of Postgres.

Support non-transactional logical messages

6cb5f91

dvarrazzo reviewed Jul 19, 2017

View reviewed changes

dpirotte added 8 commits July 19, 2017 21:11

Incorporate PR feedback

9148dce

* Write `true`/`false` instead of "1"/"0" when emitting messages.

Incorporate PR feedback

552a663

This adds a test around bytea output. I don't think is quite correct, and I believe we actually want to force escape or hex output.

Replaces spaces that leaked in with tabs (oops)

3fa6c40

Additional test around non-transactional messages

3ddc618

Non-transactional messages don't show up in WAL right away. They show up after short wait (< 0.5s) or with the next commit. This test simply demonstrates the former case.

Add a Travis CI config

2332dae

Improve version comparison to work with beta releases

89a444a

I make an assumption that any letter in `pg_config --version` will be the start of alpha, beta, or rc. Simply axe all of that and replace with `.0`, i.e `10beta2` => `10.0`.

CI build against Postgres 10

8f0e27c

eulerto closed this Nov 9, 2017

dvarrazzo mentioned this pull request Dec 9, 2017

Implement pg_logical_emit_message GambitResearch/replisome#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Output logical messages sent via pg_logical_emit_message#20

Output logical messages sent via pg_logical_emit_message#20
dpirotte wants to merge 15 commits intoeulerto:masterfrom
dpirotte:pg_logical_emit_message

dpirotte commented Jun 12, 2017

Uh oh!

dpirotte commented Jul 19, 2017

Uh oh!

dvarrazzo left a comment

Uh oh!

dvarrazzo Jul 19, 2017

Uh oh!

dvarrazzo Jul 19, 2017

Uh oh!

dvarrazzo Jul 19, 2017 •

edited

Loading

Uh oh!

dvarrazzo Jul 19, 2017

Uh oh!

dpirotte commented Jul 19, 2017

Uh oh!

eulerto commented Aug 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dpirotte commented Jun 12, 2017

Uh oh!

dpirotte commented Jul 19, 2017

Uh oh!

dvarrazzo left a comment

Choose a reason for hiding this comment

Uh oh!

dvarrazzo Jul 19, 2017

Choose a reason for hiding this comment

Uh oh!

dvarrazzo Jul 19, 2017

Choose a reason for hiding this comment

Uh oh!

dvarrazzo Jul 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dvarrazzo Jul 19, 2017

Choose a reason for hiding this comment

Uh oh!

dpirotte commented Jul 19, 2017

Uh oh!

eulerto commented Aug 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dvarrazzo Jul 19, 2017 •

edited

Loading