plugin: Add notifications from lightningd to plugins (Plugin Saga, 3rd reprise) by cdecker · Pull Request #2188 · ElementsProject/lightning

cdecker · 2018-12-18T11:32:52Z

We're nearing the end of the endless plugin saga, with this new feature. It allows plugins to subscribe to a number of notifications/events that will then be streamed from lightningd to the plugin when something happens.

Notice that, while this provides the infrastructure, I've only implemented two very simple notification topics (connect and disconnect). We can (and will) add more as we come up with new use-cases.

renepickhardt

As far as I see you forgot to bump the version number of the pylightning file to 0.0.7 that would be required (followed by a push to pylightning after merging) other than that from what I see this looks good to me and I am excited to test this out with other notifications than connect or disconnect

renepickhardt · 2018-12-18T13:04:14Z

contrib/plugins/helloworld.py

    plugin.log("Plugin helloworld.py initialized")


+@plugin.subscribe("connect")


this is an awesome API!

Thanks, Flask was a huge inspiration.

renepickhardt · 2018-12-18T13:14:01Z

lightningd/plugin.c

+	return false;
+}
+
+void plugins_notify(struct plugins *plugins,


LGTM ( I was searching if subscriptions to the same events would work if several plugins subscribed to the same event ) and from how I understand the code it seems to me that it works. I wasn't able to execute this code though

That is indeed supported, since unlike jsonrpc calls and hooks there is no problem with notification semantics.

renepickhardt · 2018-12-18T13:20:12Z

lightningd/notification.c

+	return false;
+}
+
+void notify_connect(struct lightningd *ld, struct pubkey *nodeid,


so just for the gist I could create a method void notify_incoming_htlc(struct lightningd *ld, struct pubkey *nodeid, struct channel *chan, struct htlc *h) (not sure if struct channel and struct htlc exist) and create a json string and send it via plugin_notify to the plugin. I would obviously have to call that function at the place where incoming htlc's are being handled. But other than that (and of course adding the definition of the method to notification.h) I would not have to extend other parts of c-lightning or am I missing something?

Yes, the idea is that you'd create a notify_xyz function that gathers all the information and formats it into a JSON message and then passes that to plugins_notify which will take care of dispatching notifications to all subscribers.

cdecker · 2018-12-18T15:54:58Z

I will push the new version of the pylightning library once we have the @plugin.hook functionality in there. I don't like churning through versions too quickly, especially when this is functionality that is as of now unreleased.

Should get a 0.0.7 out by the end of the week though.

ZmnSCPxj · 2018-12-19T00:29:31Z

Possible other notification:

sendpay-failure - autopilots might be interested in this and consider creating direct channels that skip failing channels or nodes.
sendpay-success - similarly autopilots might be interested, in that success implies we can lower priority to make channels to nodes at and near destination of payment.

rustyrussell

This is masterful code.

I was going to suggest we drop the static subscription-in-manifest idea for a dynamic RPC, but that actually makes it harder for plugins (at the moment they can treat STDIN as the source of all commands and notifications, and only read their rpc connection when they've sent some command).

But you should probably block subscriptions for a plugin until after it has returned from init (won't happen with these notifications, but may in future; my pay plugin actually makes rpc call before replying to init).

rustyrussell · 2018-12-20T03:28:28Z

lightningd/plugin.c

+			return false;
+		}
+		topic = tal_strndup(
+		    plugin, plugin->buffer + s->start, s->end - s->start);


Note: after rebase onto master, you'll be able to use json_strdup(plugin, plugin->buffer, s);

rustyrussell · 2018-12-20T03:28:41Z

lightningd/plugin.c

+		if (!notifications_have_topic(topic)) {
+			plugin_kill(
+			    plugin,
+			    "topic %s is not a know notification topic", topic);


s/know/known/

Slight simplification, I would skip the 's->type != JSMN_STRING' check above, and rely on it here (in which case, please put ' around the %s).

rustyrussell · 2018-12-20T03:31:36Z

lightningd/plugin.c

+static void plugin_send(struct plugin *plugin, struct json_stream *stream TAKES)
 {
-	*tal_arr_expand(&plugin->js_arr) = req->stream;
+	*tal_arr_expand(&plugin->js_arr) = tal_steal(plugin->js_arr, stream);


Make this two lines: we got bitten by this evaluation order indeterminism before.

rustyrussell · 2018-12-20T03:33:15Z

lightningd/plugin.c

 */
-static void plugin_request_queue(struct plugin *plugin,
-				 struct plugin_request *req)
+static void plugin_send(struct plugin *plugin, struct json_stream *stream TAKES)


TAKES isn't quite right here: that means it takes ownership iff caller calls take(). Perhaps call it something more obviously "takes-ownership"-is, like "plugin_attach_stream()"?

rustyrussell · 2018-12-20T03:36:17Z

lightningd/plugin.c

+		if (plugin_subscriptions_contains(p, n->method))
+			plugin_send(p, json_stream_dup(p, n->stream));
+	}
+	tal_free(n);


if (taken(n)) tal_free(n);

We could of course optimize this; when we optimize to a per-topic queue of plugins, we'd not have to copy the last one if it's taken().

niftynei · 2018-12-21T07:14:50Z

lightningd/plugin.c

-	    || !plugin_rpcmethods_add(plugin, buffer, resulttok))
+	    || !plugin_rpcmethods_add(plugin, buffer, resulttok)
+	    || !plugin_subscriptions_add(plugin, buffer, resulttok))
 		plugin_kill(plugin, "Failed to register options or methods");


There's one other thing here now: subscriptions

niftynei · 2018-12-21T07:28:13Z

lightningd/plugin.c

 }
+
+/**
+ * Determine whethe a plugin is subscribed to a given topic/method.


:s/whethe/if/

niftynei · 2018-12-21T07:35:34Z

lightningd/notification.c

 #include <ccan/array_size/array_size.h>

 const char *notification_topics[] = {
+	"connect",


i think it'd be a bit easier to tell what these refer to if we include that it's a node action, something like 'connect-node' ?

or 'node_connected'?

I bikeshedded a couple of variants for this, with broader topics, and naming variants, but in the end I just picked the simplest and shortest variant I could think of, i.e., just the actions that are performed. Happy to discuss the naming here further, but then I'd split the addition of this topic out and just get the infrastructure merged, so I can continue working on top of this 😉

sounds good. 👍

fwiw, i like how the 'actor_action' pattern (ie 'node_connected') gives a bit more insight at a glance as to what this refers to, but being as i'm more familiar with the codebase now, connect is pretty easy to map back to RPC commands, which is also nice :)

niftynei · 2018-12-21T07:38:12Z

lightningd/notification.c

+{
+	struct jsonrpc_notification *n =
+	    jsonrpc_notification_start(NULL, notification_topics[0]);
+	json_add_pubkey(n->stream, "id", nodeid);


i'd propose changing this to peer_id instead of just id

No, "id" is standard for the RPC layer, unless it's confusing. In this case, it's not.

Ack. I totally misread some python code earlier; I redact this comment.

niftynei · 2018-12-21T07:39:44Z

lightningd/notification.c

+void notify_disconnect(struct lightningd *ld, struct pubkey *nodeid)
+{
+	struct jsonrpc_notification *n =
+	    jsonrpc_notification_start(NULL, notification_topics[1]);


is there a way to not use hardcoded array indices for these? seems like this will be easy to break without meaning to if the notification_topics ever get reordered.

There is a facility to generate names from enums, maybe we want to use that instead? I had an enum that represented the indices into this array, but it got really repetitive, so I dropped it for now.

We use AUTODATA to generate an array of conmands, cannot we use same to generate array of notifications?

We could do that, and I think I'm doing something similar in the hooks PR. We can revisit this later I think. The hooks PR also has some nice things like non-dynamic dispatch of serializers and callback, which might be interesting here as an optimization (skipping serialization if nobody has subscribed).

cdecker · 2018-12-22T15:20:08Z

Rebased and addressed the feedback in fixups. @rustyrussell and @niftynei thanks for the reviews, if you're happy with the fixups I'll squash and merge :-)

P.S.: I know it looks like a lot of fixups, but they are all single-lines and address a single comment each to make reviewing easier 😉

rustyrussell · 2018-12-25T02:27:56Z

I do want "peer_id" changed back to "id" though, since that's the nomenclature used everywhere else. Rest is fine. Using an enum for notifications is a win, but that's a simple neatening which can be done later.

rustyrussell · 2018-12-25T02:28:56Z

Ack d9839bc

conscott

Right now the docstring for plugin commands includes newline junk in the description text - like

hello [params]
    This is the documentation string for the hello-function.\n\nIt gets reported as the description when registering the function\nas a method with `lightningd`.

You could just add a line here to convert docstring newlines to spaces (if desired)

doc = re.sub('\n+', ' ', doc)

conscott · 2018-12-27T07:02:40Z

Tested ACK 91c9ce7

Just left a minor nit above.

cdecker · 2018-12-29T11:30:07Z

Rebased, dropped the id -> peer_id fixups, added a docstring squash fixup as suggested by @rustyrussell 😉

rustyrussell

Ack 7de1399

conscott · 2018-12-30T06:26:47Z

Re-ACK 7de1399

Signed-off-by: Christian Decker <decker.christian@gmail.com>

Will be used in the next commit to fan out notifications to multiple subscribing plugins. We can't just use `tal_dup` from outside since the definition is hidden outside the compilation unit. Signed-off-by: Christian Decker <decker.christian@gmail.com>

This used to be request-specific, but we now want to send notifications and requests. As a drive-by we also clarify the ownership of the json_stream instance that is being sent. Signed-off-by: Christian Decker <decker.christian@gmail.com>

Signed-off-by: Christian Decker <decker.christian@gmail.com>

Just like we added the RPC methods, the notification handlers can also be registered using a function decorator, and we auto-subscribe when asked for a manifest. Signed-off-by: Christian Decker <decker.christian@gmail.com>

Signed-off-by: Christian Decker <decker.christian@gmail.com>

cdecker · 2018-12-30T13:34:47Z

Rebased and squashed on top of master. Reapplying @rustyrussell's ACK (@bitcoin-bot) is still having issues recognizing rebases and squashes)

ACK 5a8c2c6

cdecker added json-rpc plugin labels Dec 18, 2018

cdecker added this to the v0.6.3 milestone Dec 18, 2018

cdecker self-assigned this Dec 18, 2018

cdecker requested a review from rustyrussell December 18, 2018 11:32

renepickhardt suggested changes Dec 18, 2018

View reviewed changes

rustyrussell approved these changes Dec 20, 2018

View reviewed changes

niftynei approved these changes Dec 21, 2018

View reviewed changes

cdecker force-pushed the plugin-7 branch from 6299d4d to e8a9aa5 Compare December 22, 2018 15:18

cdecker force-pushed the plugin-7 branch 3 times, most recently from db3b21d to 91c9ce7 Compare December 22, 2018 15:49

conscott reviewed Dec 27, 2018

View reviewed changes

conscott mentioned this pull request Dec 27, 2018

[plugin] 0.0.1 of donation plugin and invoice query service #2152

Closed

cdecker force-pushed the plugin-7 branch from 91c9ce7 to 40dcf26 Compare December 29, 2018 11:27

cdecker force-pushed the plugin-7 branch from e6162c2 to 7de1399 Compare December 29, 2018 12:23

rustyrussell approved these changes Dec 30, 2018

View reviewed changes

cdecker added 5 commits December 30, 2018 13:31

jsonrpc: Create a struct for notifications that we send

5ee7dde

Signed-off-by: Christian Decker <decker.christian@gmail.com>

plugin: Add subscriptions when processing the plugin manifest

5df91c3

Signed-off-by: Christian Decker <decker.christian@gmail.com>

plugin: Dispatch notifications to subscribed plugins

cd8439b

Signed-off-by: Christian Decker <decker.christian@gmail.com>

cdecker added 5 commits December 30, 2018 13:31

plugin: Add connect and disconnect notifications

979315a

Signed-off-by: Christian Decker <decker.christian@gmail.com>

plugin: Document JSON-RPC passthrough and notifications

fc2f1bc

Signed-off-by: Christian Decker <decker.christian@gmail.com>

pylightning: Add notification subscription handlers

eee1e75

Just like we added the RPC methods, the notification handlers can also be registered using a function decorator, and we auto-subscribe when asked for a manifest. Signed-off-by: Christian Decker <decker.christian@gmail.com>

pytest: Add a test for the event subscription and notification

f655725

Signed-off-by: Christian Decker <decker.christian@gmail.com>

fixup! pylightning: Added a tiny library for python plugins

5a8c2c6

cdecker force-pushed the plugin-7 branch from 7de1399 to 5a8c2c6 Compare December 30, 2018 12:31

cdecker merged commit 4f16044 into master Dec 30, 2018

trueptolemy mentioned this pull request Aug 11, 2019

plugin: sendpay_success and sendpay_failure notifications #2945

Merged

rustyrussell deleted the plugin-7 branch August 15, 2022 00:41

		plugin.log("Plugin helloworld.py initialized")


		@plugin.subscribe("connect")

Conversation

cdecker commented Dec 18, 2018

Uh oh!

renepickhardt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

renepickhardt Dec 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

renepickhardt Dec 18, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdecker commented Dec 18, 2018

Uh oh!

ZmnSCPxj commented Dec 19, 2018

Uh oh!

rustyrussell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niftynei Dec 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdecker commented Dec 22, 2018

Uh oh!

rustyrussell commented Dec 25, 2018

Uh oh!

rustyrussell commented Dec 25, 2018

Uh oh!

conscott left a comment

Choose a reason for hiding this comment

Uh oh!

conscott commented Dec 27, 2018

Uh oh!

renepickhardt Dec 18, 2018 •

edited

Loading

renepickhardt Dec 18, 2018 •

edited

Loading

niftynei Dec 27, 2018 •

edited

Loading