Avoid resources for atomic compare-and-swap operations?

[fitzgen]

Right now, compare-and-swap operations are represented as resources:

wasi-keyvalue/wit/atomic.wit

Lines 24 to 32 in fb6e23d

	/// A handle to a CAS (compare-and-swap) operation.
	resource cas {
	/// Construct a new CAS operation. Implementors can map the underlying functionality
	/// (transactions, versions, etc) as desired.
	new: static func(bucket: borrow<bucket>, key: string) -> result<cas, error>;
	/// Get the current value of the key (if it exists). This allows for avoiding reads if all
	/// that is needed to ensure the atomicity of the operation
	current: func() -> result<option<list<u8>>, error>;
	}

Creating a resource is a relatively expensive operation in the component model as it involves resource table manipulation and potentially growing the resource table and its underlying allocation. Furthermore, engines/embedders may have implementation limits on the size of their resource tables to limit guest-controlled memory consumption, so creating a resource can fail non-deterministically from the Wasm guest's point of view (similar to the memory.grow instruction in core Wasm).

Furthermore, because swap takes ownership of the cas resource and, when the operation fails, the cas resource is returned again as part of the error, we are required to do even more resource table manipulation: removing the resource from the table on call and then adding it back in again on error return. These bits could potentially be avoided if the swap operation took a borrow<cas> instead of taking the cas by ownership.

But even better would be to avoid resources completely for compare-and-swap operations. A (perhaps naive) initial sketch could look something like this:

interface atomics {
  	use store.{bucket, error};

	/// The error returned by a CAS operation.
	variant cas-error {
		/// A store error occurred when performing the operation.
		store-error(error),

		/// The CAS operation failed because `swap`'s `current-value` argument
		/// did not match the entry's current value.
		///
		/// This returns the entry's current value.
		cas-failed(option<list<u8>>),
	}

	/// Perform the swap on a CAS operation.
	///
	/// If the entry's current value does not match `current-value` then
	/// an error is returned with the current value, so that the `swap`
	/// operation can be retried.
	swap: func(
		bucket: borrow<bucket>,
		key: string,
		new-value: list<u8>,
		current-value: option<list<u8>>,
	) -> result<_, cas-error>;
}

This does imply copying the new and current values on every call, which may end up being more expensive than the resource table manipulation for large values. (FWIW, the existing API also exhibits this to some degree, since swap takes value: list<u8>, rather than the cas resource owning/containing the value so that if you are trying to swap in a loop you don't have to keep copying it.)

For large keys and values, we would want to move the API in the other direction: towards using resources more. (Right now, the API is in a weird middle ground where it isn't a great match for either scenario.) The maximally-resourced version of the API might look something like this:

interface atomics {
  	use store.{bucket, error};

	/// The error returned by a CAS operation.
	variant cas-error {
		/// A store error occurred when performing the operation.
		store-error(error),
		/// The CAS operation failed because the value was too old.
		cas-failed,
	}

	resource cas {
		new: static func(
			bucket: borrow<bucket>,
			key: string,
			new-value: list<u8>,
			current-value: option<list<u8>>,
		) -> result<cas, error>;

		/// Getters and setters for this `cas` resource's internal
		/// fields so that one `cas` resource can be reused for
		/// multiple compare-and-swap operations.
		get-bucket: func() -> bucket;
		set-bucket: func(bucket: borrow<bucket>);
		get-key: func() -> string;
		set-key: func(key: string);
		get-new-value: func() -> list<u8>;
		set-new-value: func(new-valu: list<u8>);
		get-current-value: func() -> option<list<u8>>;
		set-current-value: func(current-value: option<list<u8>>);
		
		/// Perform the configured compare-and-swap operation.
                ///
		/// On success, clears the inner `key`, `new-value`, and 
		/// `current-value` so that this `cas` resource can be
		/// reused for other compare-and-swap operations.
		///
		/// On error, updates this `cas` resource's `current-value`
		/// with the entry's current value. This avoids copying the
		/// current value in and out of the guest, which can be
		/// expensive for large values.
		swap: func() -> result<_, cas-error>;
	}
}

It may actually make sense to have both variants of the API.

Thoughts?