Send & Recv Options

Send Options

Source and push jobs have an optional send configuration section.

jobs:
- type: push
  filesystems: ...
  send:
    # flags from the table below go here
  ...

The following table specifies the list of (boolean) options. Flags with an entry in the zfs send column map directly to the zfs send CLI flags. zrepl does not perform feature checks for these flags. If you enable a flag that is not supported by the installed version of ZFS, the zfs error will show up at runtime in the logs and zrepl status. See the upstream man page (man zfs-send) for their semantics.

`send.`	`zfs send`	Comment
`encrypted`		Specific to zrepl, see below.
`bandwidth_limit`		Specific to zrepl, see below.
`raw`	`-w`	Use `encrypted` to only allow encrypted sends. Mixed sends are not supported.
`send_properties`	`-p`	Be careful, read the note on property replication below.
`backup_properties`	`-b`	Be careful, read the note on property replication below.
`large_blocks`	`-L`	Potential data loss on OpenZFS < 2.0, see warning below.
`compressed`	`-c`
`embedded_data`	`-e`
`saved`	`-S`

`encrypted`

The encrypted option controls whether the matched filesystems are sent as OpenZFS native encryption raw sends. More specifically, if encrypted=true, zrepl

checks for any of the filesystems matched by filesystems whether the ZFS encryption property indicates that the filesystem is actually encrypted with ZFS native encryption and
invokes the zfs send subcommand with the -w option (raw sends) and
expects the receiving side to support OpenZFS native encryption (recv will fail otherwise)

Filesystems matched by filesystems that are not encrypted are not sent and will cause error log messages.

If encrypted=false, zrepl expects that filesystems matching filesystems are not encrypted or have loaded encryption keys.

Note

Use encrypted instead of raw to make your intent clear that zrepl must only replicate filesystems that are actually encrypted by OpenZFS native encryption. It is meant as a safeguard to prevent unintended sends of unencrypted filesystems in raw mode.

`properties`

Sends the dataset properties along with snapshots. Please be careful with this option and read the note on property replication below.

`backup_properties`

When properties are modified on a filesystem that was received from a send stream with send.properties=true, ZFS archives the original received value internally. This also applies to inheriting or overriding properties during zfs receive.

When sending those received filesystems another hop, the backup_properties flag instructs ZFS to send the original property values rather than the current locally set values.

This is useful for replicating properties across multiple levels of backup machines. Example: Suppose we want to flow snapshots from Machine A to B, then from B to C. A will enable the properties send option. B will want to override critical properties such as mountpoint or canmount. But the job that replicates from B to C should be sending the original property values received from A. Thus, B sets the backup_properties option.

Please be careful with this option and read the note on property replication below.

`large_blocks`

This flag should not be changed after initial replication. Prior to OpenZFS commit 7bcb7f08 it was possible to change this setting which resulted in data loss on the receiver. The commit in question is included in OpenZFS 2.0 and works around the problem by prohibiting receives of incremental streams with a flipped setting.

Warning

This bug has not been fixed in the OpenZFS 0.8 releases which means that changing this flag after initial replication might cause data loss on the receiver.

Recv Options

Sink and pull jobs have an optional recv configuration section:

jobs:
- type: pull
  recv:
    properties:
      inherit:
        - "mountpoint"
      override: {
        "org.openzfs.systemd:ignore": "on"
      }
    bandwidth_limit: ...
    placeholder:
      encryption: unspecified | off | inherit
  ...

Jump to properties , bandwidth_limit , and placeholder.

`properties`

override maps directly to the zfs recv -o flag. Property name-value pairs specified in this map will apply to all received filesystems, regardless of whether the send stream contains properties or not.

inherit maps directly to the zfs recv -x flag. Property names specified in this list will be inherited from the receiving side’s parent filesystem (e.g. root_fs).

With both options, the sending side’s property value is still stored on the receiver, but the local override or inherit is the one that takes effect. You can send the original properties from the first receiver to another receiver using send.backup_properties.

A Note on Property Replication

If a send stream contains properties, as per send.properties or send.backup_properties, the default ZFS behavior is to use those properties on the receiving side, verbatim.

In many use cases for zrepl, this can have devastating consequences. For example, when backing up a filesystem that has mountpoint=/ to a storage server, that storage server’s root filesystem will be shadowed by the received file system on some platforms. Also, many scripts and tools use ZFS user properties for configuration and do not check the property source (local vs. received). If they are installed on the receiving side as well as the sending side, property replication could have unintended effects.

zrepl currently does not provide any automatic safe-guards for property replication:

Make sure to read the entire man page on zfs recv (man zfs recv) before enabling this feature.
Use recv.properties.override whenever possible, e.g. for mountpoint=none or canmount=off.
Use recv.properties.inherit if that makes more sense to you.

Below is an non-exhaustive list of problematic properties. Please open a pull request if you find a property that is missing from this list. (Both with regards to core ZFS tools and other software in the broader ecosystem.)

Mount behaviour

mountpoint
canmount
overlay

Note: Before OpenZFS 2.0.5, inheriting or overriding the mountpoint property on ZVOLs fails in zfs recv. If you are on such an older version, consider creating separate zrepl jobs for your ZVOL and filesystem datasets.

Systemd

With systemd, you should also consider the properties processed by the zfs-mount-generator .

Most notably:

org.openzfs.systemd:ignore
org.openzfs.systemd:wanted-by
org.openzfs.systemd:required-by

Encryption

If the sender filesystems are encrypted but the sender does plain sends and property replication is enabled, the receiver must inherit the following properties:

keylocation
keyformat
encryption

Placeholders

placeholder:
  encryption: unspecified | off | inherit

During replication, zrepl creates placeholder datasets on the receiving side if the sending side’s filesystems filter creates gaps in the dataset hierarchy. This is generally fully transparent to the user. However, with OpenZFS Native Encryption, placeholders require zrepl user attention. Specifically, the problem is that, when zrepl attempts to create the placeholder dataset on the receiver, and that placeholder’s parent dataset is encrypted, ZFS wants to inherit encryption to the placeholder. This is relevant to two use cases that zrepl supports:

encrypted-send-to-untrusted-receiver In this use case, the sender sends an encrypted send stream and the receiver doesn’t have the key loaded.
send-plain-encrypt-on-receive The receive-side root_fs dataset is encrypted, and the senders are unencrypted. The key of root_fs is loaded, and the goal is that the plain sends (e.g., from production) are encrypted on-the-fly during receive, with root_fs’s key.

For encrypted-send-to-untrusted-receiver, the placeholder datasets need to be created with -o encryption=off. Without it, creation would fail with an error, indicating that the placeholder’s parent dataset’s key needs to be loaded. But we don’t trust the receiver, so we can’t expect that to ever happen.

However, for send-plain-encrypt-on-receive, we cannot set -o encryption=off. The reason is that if we did, any of the (non-placeholder) child datasets below the placeholder would inherit encryption=off, thereby silently breaking our encrypt-on-receive use case. So, to cover this use case, we need to create placeholders without specifying -o encryption. This will make zfs create inherit the encryption mode from the parent dataset, and thereby transitively from root_fs.

The zrepl config provides the recv.placeholder.encryption knob to control this behavior. In undefined mode (default), placeholder creation bails out and asks the user to configure a behavior. In off mode, the placeholder is created with encryption=off, i.e., encrypted-send-to-untrusted-rceiver use case. In inherit mode, the placeholder is created without specifying -o encryption at all, i.e., the send-plain-encrypt-on-receive use case.

Common Options

Bandwidth Limit (send & recv)

bandwidth_limit:
  max: 23.5 MiB # -1 is the default and disabled rate limiting
  bucket_capacity: # token bucket capacity in bytes; defaults to 128KiB

Both send and recv can be limited to a maximum bandwidth through bandwidth_limit. For most users, it should be sufficient to just set bandwidth_limit.max. The bandwidth_limit.bucket_capacity refers to the token bucket size.

The bandwidth limit only applies to the payload data, i.e., the ZFS send stream. It does not account for transport protocol overheads. The scope is the job level, i.e., all concurrent sends or incoming receives of a job share the bandwidth limit.

Send & Recv Options

Send Options

encrypted

properties

backup_properties

large_blocks

Recv Options

properties

A Note on Property Replication

Mount behaviour

Systemd

Encryption

Placeholders

Common Options

Bandwidth Limit (send & recv)

`encrypted`

`properties`

`backup_properties`

`large_blocks`

`properties`