zrepl replicates every snapshot including those created outside zrepl

[calistoc]

Hello,
just to ask some help please as I was searching the documentation but I’m confused and not sure whether I’m missing something.

For what I see zrepl replicates all the snapshots in a filesystem including those snaps created outside zrepl, for example I have some snaps created to perform a daily backup that I would like not to be considered by zrepl but they get replicated once I wakeup the jobs.
This happens for push jobs with periodic snapshotting and also with jobs with manual snapshotting where I need to wakeup the jobs via crontab and where I cannot specify a prefix in the snapshotting configuration:

snapshotting:
type: manual
prefix: zrepl_ (it gives this error: field prefix not found in type config.SnapshottingManual)

It seems to me that the snapshotting prefix for push jobs with periodic snapshotting is only used for pruning but it is not used to select the snapshots to be replicated, is this correct?
Is there a way to have zrepl replicating only its own snapshots or better only those specified by a prefix?

Thank you

[problame]

Hi, thanks for describing your use case.

It seems to me that the snapshotting prefix for push jobs with periodic snapshotting is only used for pruning but it is not used to select the snapshots to be replicated, is this correct?

Actually, it's even less than that. The prefix you specify in the snapshotting section is really only used to define the snapshot name, nothing more, nothing less.

Is there a way to have zrepl replicating only its own snapshots or better only those specified by a prefix?

No, currently there isn't a way to tell zrepl to do that.

• receiver's latest snapshot is not present on sender  #248 requests prefix-based filtering of snapshots for replication. It seems that if we implemented prefix-based filtering, your problem would be solved. Wouldn't it?
• It seems like you assumed zrepl would somehow 'own' the snapshots it created and either only act on the snapshots it created or treat them specially during replication or pruning. That is currently not the case - replication and pruning do not know whether a snapshot was created by zrepl or something else. But it's a very widespread belief, and zrepl should probably develop into a direction where snapshot management and replication are more tightly integrated.

If you feel like #248 would be the idea solution, please subscribe there and post a proposal for the configuration syntax in your scenario.

[calistoc]

Thanks for your feedbacks

Yes I think the #248 request would solve my problem.
About the proposal for the configuration syntax, I would add the following optional parameter in the configuration job, either push, pull or source, something like this:

• type: push
  name: to_server2
  replication_snapshots_prefix: zrepl_
  …

This would replicate only snapshots with zrepl_ prefix, not specifying this parameter would leave things as they are now replicating all snapshots and preserving the current functionalities
I posted the same on #248

Thank you for your work!

[beren12]

Just a few things. If you send a range of snapshots for incremental sends, then every snap in between would be sent, no way to not do that unless you broke it up into smaller chunks. You could manually destroy or prune non-zrepl snaps on the remote side. I believe you can set custom properties on snapshots, there was a patch for zfs-auto-snap to do this to mark things like daily/weekly/monthly snapshots while all having the same name prefix. zrepl could set some properties and look for it for filtering which snaps to send/prune

[problame]

Just a few things. If you send a range of snapshots for incremental sends, then every snap in between would be sent, no way to not do that unless you broke it up into smaller chunks

@beren12 What are you referring to here? {desired, current} zrepl behavior or zfs send -I?

You could manually destroy or prune non-zrepl snaps on the remote side.

What do you mean by 'non-zrepl snaps'? Snapshtos not created by zrepl on the sender but replicated by zrepl to the receiver?

I believe you can set custom properties on snapshots, there was a patch for zfs-auto-snap to do this to mark things like daily/weekly/monthly snapshots while all having the same name prefix. zrepl could set some properties and look for it for filtering which snaps to send/prune

There are two problems with attaching policy to snapshots via snapshot user properties:

• The first user property on a snapshot allocates the ZAP object for it and thus has some storage overhead. Probably negligible.
• If property replication is enabled, the snapshot property that was zfs set on the sending system is also present on the receiving system.
  • If it's a non-zrepl policy that is a problem the user has to deal with.
  • If it's a zrepl policy I'd want zrepl to be smart about it.
    • What should zrepl do with the snapshot property that would make a snapshot eligible for sending even if it's not matched by the to-be-done replication_snapshots_prefix?
    • What should zrepl do if a snapshot is marked as a ghost for pruning (i.e. pruning shouldn't even consider it)?

[problame]

(I originally marked this issue as a duplicate of #248 but it's actually the first original request for a 'filtering' kind of feature).

[beren12]

Sorry what I was trying to say was if you send incremental sends, there's no way to avoid this unless you prune the snapshot before, which some people don't want, or after it's send prune from the remote side. With the properties… I didn't think too much of the downsides to deal with, just threw that idea out there. But yes, you would need to have some way to tag the properties per host, like with the hostname but even then that may not be unique.

[beren12]

I guess you could stop early if the latest snapshot(s) weren't from zrepl though

[problame]

Sorry what I was trying to say was if you send incremental sends, there's no way to avoid this unless you prune the snapshot before, which some people don't want, or after it's send prune from the remote side.

What do you mean by this in this sentence? Please inline-quote what are referring to.

I guess you could stop early if the latest snapshot(s) weren't from zrepl though

zrepl doesn't have a concept of 'ownership' or 'knowing which snapshots it created' at this time. (see above). I don't understand what you mean by 'stop early'.