Fuzzing components with wasm-smith needs to be updated

[alexcrichton]

I'm disabling the tests in #966 since the preexisting support for fuzzing components with wasm-smith is outdated enough that I ended up not trying to update it. This may be as easy as "implement recent component model changes" and may also range up to "oh dear much of this needs to be rewritten". In any case I'm writing this down to record on some ignored tests.

[0xalpharush]

How would you propose generating components that conform to a given WIT? I am thinking of using wasm-smith to generate modules with the available imports/necessary exports and then embedding the component info (wasm-tools component embed sample.wit test.wasm -o output.wasm).

[alexcrichton]

One option is to use wasm-tools component embed --dummy sample.wit and that gives you a core wasm module to "fill out". That's unlikely to be too too useful though because although you can generate the core wasm signatures it's pretty unlikely that you'll generate a module that happens to use the correct canonical ABI, so almost all modules are likely to fail/trap when actually receiving or giving an argument.

[0xalpharush]

I think I'm okay with invalid ABI for my use case. Does wasm-mutate not change function bodies? I haven't found a mutator that does

[alexcrichton]

wasm-mutate should change function bodies yes, it may not always do so but with the right seed it has various changes it can make

[0xalpharush]

wasm-tools component embed --dummy sample.wit produces functions with an unreachable instruction in the body. I don't think ConstExpressionMutator, PeepholeMutator, or CodemotionMutator will add instructions for this case, and those seem to be the only mutations that would result in different, non-empty instructions.

wasm-tools/crates/wasm-mutate/src/lib.rs

Lines 20 to 26 in 2c0c3fd

	add_function::AddFunctionMutator, add_type::AddTypeMutator, codemotion::CodemotionMutator,
	custom::AddCustomSectionMutator, custom::CustomSectionMutator,
	custom::ReorderCustomSectionMutator, function_body_unreachable::FunctionBodyUnreachable,
	modify_const_exprs::ConstExpressionMutator, modify_data::ModifyDataMutator,
	peephole::PeepholeMutator, remove_export::RemoveExportMutator, remove_item::RemoveItemMutator,
	remove_section::RemoveSection, rename_export::RenameExportMutator, snip_function::SnipMutator,
	Item,

Would adding a mutation that adds instructions like this be a good idea or there is a reason it's not done?

wasm-tools/crates/wasm-smith/src/core.rs

Lines 2267 to 2279 in 2c0c3fd

	fn arbitrary_code(&mut self, u: &mut Unstructured) -> Result<()> {
	self.compute_interesting_values();
	self.code.reserve(self.num_defined_funcs);
	let mut allocs = CodeBuilderAllocations::new(self, self.config.exports.is_some());
	for (idx, ty) in self.funcs[self.funcs.len() - self.num_defined_funcs..].iter() {
	let shared = self.is_shared_type(*idx);
	let body = self.arbitrary_func_body(u, ty, &mut allocs, shared)?;
	self.code.push(body);
	}
	allocs.finish(u, self)?;
	Ok(())
	}

[alexcrichton]

Ah yes sorry good point, while wasm-mutate does modify function bodies I don't think it has mutations for bodies-with-just-unreachable. Adding a mutation would be an interesting idea yeah! Currently wasm-mutate doesn't receive too much love or maintenance but it'd be good to expand its functionality