Add multi-microphone beamformer component #2925

singalsu · 2020-05-11T16:58:50Z

Please find the descriptions from commit messages. Known limitation:

Pipelines with different channel count in DAI and PCM trigger an error check. Therefore I've made in test topologies the input channels count and output channels count equal. The beamformer duplicates to output a sufficient number of channels. It's not ideal but needs to be done until the limitation can be prevented.

lgirdwood · 2020-05-13T12:41:44Z

@cujomalainey @sebcarlucci any review comments ?

singalsu · 2020-05-13T12:48:55Z

@perexg I just accidentally added you as reviewer but you are welcome if you have time.

lgirdwood

LGTM, need to sort out the ABI and comp type though.

lgirdwood · 2020-05-13T12:46:57Z

src/include/kernel/abi.h

Not sure, this could be 16 - just depends on when kernel PR is approved.

Not sure if ABI needs bump (MINOR is now 17, would be to 18). There's the change to the user header fir.h.

src/include/ipc/topology.h

singalsu · 2020-05-13T13:04:59Z

I should create soon documentation for this. It may be hard to figure this out without. The improvement in capture SNR is the larger the more microphones, 3 or 4 or even more preferable. With only 2 microphones and steer angle 0 (broadside) it reduces to trivial sum of microphones with only 3 dB SNR improve. Though it might be still audible, I'll do tests here and make some demo recordings.

The plots from the tool help to understand more what it is and what it isn't. Here's a 4 mic example steered to 30 deg azimuth, superdirective design criteria (minimize diffuse noise):

singalsu · 2020-05-13T13:18:46Z

Another example to show difference with beam patterns of 1D line and 2D (circular) arrays, this is with 8 microphones.

singalsu · 2020-05-13T13:26:28Z

Also as noted currently the beam is fixed and defined by blob coefficients. The blob does not currently contain multiple presets. A coefficients presets system would allow some IPC or internal direction of arrival tracker to change the beam direction. Since as it can be seen the beams with lower mic counts are not narrow, a few of them could cover 180 or 360 degree space. I wonder if I should add it. Though multiple pre-defined beams need plenty of .bss section size. With symmetrical 2D arrays it's possible to also simply rotate the mic inputs for changed beam direction.

cujomalainey · 2020-05-13T20:03:53Z

@lgirdwood @sebcarlucci finished his internship. @chiang831 and @dgreid for thoughts.

cujomalainey · 2020-05-13T20:59:15Z

@singalsu to confirm your comment, the filter does not spatial tracking, it takes in a target direction and sticks with that? Also, could the beam be disabled at runtime?

singalsu · 2020-05-14T09:17:18Z

@singalsu to confirm your comment, the filter does not spatial tracking, it takes in a target direction and sticks with that?

Yep, the single beam or multiple beams are fixed for the azimuth and elevation angle in design phase with the Matlab/Octave tool.

So, in current form it works only for use cases like notebook video call where the desired audio capture source is very likely in camera direction. Out of beam audio sources have especially their higher frequencies attenuated.

The multiple simultaneous beams are done by designing one beam at time for the mic array and then merging them into the same blob with different output channels mixing (I made such simple stereo image enhance beamformer example as 2mic in -> 2ch out).

I've done a Matlab concept for sound source tracking and it seems to work pretty well. In a future version the FW could be let to change the beam direction freely within designed presets. Though there's a risk that it locks to something undesired noisy in the room (fan?). Therefore fixed or user space controlled beam angle has benefits if the use case is known.

Also, could the beam be disabled at runtime?

That would be useful. Currently it is supported in runtime/idle only via user space (sof-ctl) sent new pass-through blob. The update is identical to IIR and FIR.

Maybe the beam on/off control could fit ALSA switch control type. @juimonen @kv2019i what do you think? I think currently the process type component implementation supports only binary control but adding more control types should be possible. Also I wonder if some other ALSA control type (enum?) could cover 360 degree azimuth angle with some granularity. What options would there be?

Beam disable/enable is straightforward for case where number of input channels and output channels is equal. E.g. for 2 microphones when beamformer is "on" the output would be typically double mono and with beamformer "off" the mic channels as such.

If the beamformer configuration is e.g. 4 microphones to 2 channels (double mono) there should be in blob a control for which of 4 pass into 2 outputs. Maybe there should be in the blob always a beams "off" preset included.

I think I should add to this design some beam presets control capability.

zrombel · 2020-08-01T18:30:21Z

Due to recent PR #3194 merge, please rebase for TGL Multicore tests to Pass on Internal CI.

lgirdwood

@singalsu just blocking on the UUID now. LGTM, will approve once UUID is merged and UUID changes is added here.

tools/topology/sof-apl-pcm512x.m4

singalsu · 2020-09-09T16:53:47Z

tools/topology/CMakeLists.txt

@juimonen Tested that this topology is more silent than a plain sof-hda-generic. I noticed in topology for UP2 a -6 dB level change vs. normal topology. I'll check the filters that there's no extra 6 dB attenuation in place.

Done, there was a FIR shift value polarity issue in hifi3 version.

juimonen · 2020-09-10T08:03:42Z

tools/topology/platform/intel/intel-generic-dmic.m4

should these changes to dmic-generic and sof-apl-512 be in separate, possible extra commit, before this commit?

Yes probably should. I made one commit that contains the topology preparations. There should be a clear no-difference to any built topologies but with all needed preparations step.

This is now fixed. I moved the non-TDFB related improvemets to the previous commit.

singalsu · 2020-09-10T11:57:37Z

tools/topology/m4/tdfb_coef_flat.m4

There's quite many of these and more to come later. I think I should rename these to m4/tdfb/coef_flat.m4 etc. to avoid clutter.

This move is now done.

singalsu · 2020-09-10T16:31:06Z

@lgirdwood I'm still searching where/why the gain drop happens, so this is not yet OK to merge.

singalsu · 2020-09-10T17:21:47Z

tools/topology/sof/pipe-tdfb-capture-16khz.m4

This should be tdfb/coef_line2_pass.m4. The file has been renamed.

This patch enables FIR filter core usage independently from FIR equalizer component. The inline of FIR core is removed to reduce the code size when FIR is used from several components. Each component also typically used inline FIR version for each supported PCM format that increased further the size. Most of the changes are due to rename and directory move of some FIR data structures and macros. The code functionality is not changed. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu · 2020-09-11T16:22:12Z

tools/topology/CMakeLists.txt

The tdfb-volume pipeline does not filter out the DC thump in the beginning. I'll change this to tdfb-eq-iir-volume.

This patch adds the multi-microphone beamforming component. It enhances microphone capture via spatial noise suppression. The component is a quite generic FIR time domain filter bank and the fixed filter band needs to be programmed with super directive or other beamformer criteria filter coefficients. The coefficients are fixed but they can be re-programmed during run-time. The component reuses the FIR filter core but has different inputs selection and outputs mixing features than FIR EQ so it is made a separate new processing component. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds the tool for creating beamformer configurations. The microphone array geometry and beam angle (azimuth, elevation) need to be specified. See the example scripts and sample array helper functions. The FIR blob quantize function needed a minor change to prevent strip of trailing zero coefficients. The beamformer filter bank needs to use equal length filters. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds support to define from CMakeLists.txt or a higher level platform topology file the definition of DMICPROC and DMIC16KPROC to select desired capture processing algorithms pipeline from pipe-x-capture.m4 and pipe-x-capture-16khz.m4 macros instead of hard coded processing eq-iir-volume. It is preparation to add support for beamformer processing for microphones. The impacted platforms are sof-hda-generic, sof-cml-rt5682, and sof-apl-pcm512x. This patch does not change built topologies. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

These are data files created by the example scripts in tools/tune/tdfb. The generation is time consuming and requires Octave or Matlab. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds the playback (for test only) and capture pipelines with Time Domain Fixed Beamformer (TDFB) component. Topologies variants to test capture with beamformer are built for sof-hda-generic and sof-apl-pcm512x platforms. The beam direction is +/- 10 degrees as compromise between notebook camera and stereo capture. The dual beams preserve the stereo characteristic. The beamformers are added to both 48 kHz and 16 kHz DMIC capture pipelines. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

The patch adds the playback and capture test pipelines. The configuration is set to `tdfb_coef_line2_50mm_pm90deg_16khz.m4'. A mistake in PIPELINE_FILTERx macro defining is fixed for IIR and FIR. The pipeline macros expect it to contain an include file or no macro defined at all. Defining it for empty string caused fail in topologies build when PIPELINE_FILTER1 is used for TDFB. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

The script adds tdfb_test.m to check the TDFB beam pattern versus theoretical. It also measures the noise suppression capability of the test beamformer in simulated diffuse and random noise field. As simple quick test this patch adds TDFB to cell array of accepted components for process test. Note: There tests can't be used until load of UUID based non-legacy components is added to testbench. The scripts were used with earlier legacy mode version of the component. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds the needed information to testbench to load the TDFB component. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

The if test needs to be done for comp_type. The index does not refer to comp types but items in lib_table that is not correct. As result testbench loads crossover for all UUID based components. The load of beamformer works correctly with this change. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu · 2020-09-11T18:16:47Z

@lgirdwood This version is at my best knowledge OK to merge based on my own build tests and tests with UP2. I'll continue testing with a suitable hda-generic device that I just got on Monday. @juimonen has tested an earlier version of this PR with his.

lgirdwood · 2020-09-13T14:52:18Z

@singalsu I'm going to merge this now so validation has enough time to check before the window closes for v1.6 and since it's a large PR. It will also be marked "initial" in the release notes.

singalsu · 2020-09-14T07:56:39Z

Thanks @lgirdwood ! Initial makes sense. This version does not have the control for processing on/off except the sof-ctl based re-configuring. Also need to work with pipelines framework to allow different channels count in source vs. sink.

singalsu requested review from jajanusz, lgirdwood and tlauda May 11, 2020 16:58

singalsu requested review from a team, dbaluta, lbetlej, mmaka1, plbossart and ranj063 as code owners May 11, 2020 16:58

cujomalainey mentioned this pull request May 11, 2020

audio: Add Crossover Filter component #2802

Closed

singalsu force-pushed the add_beamformer branch 4 times, most recently from eeb46f0 to 935a490 Compare May 12, 2020 14:36

singalsu requested review from cujomalainey, perexg and sebcarlucci May 13, 2020 12:46

lgirdwood reviewed May 13, 2020

View reviewed changes

cujomalainey removed the request for review from sebcarlucci May 13, 2020 20:53

kv2019i mentioned this pull request May 20, 2020

[WIP] ext_manifest: Add UUID dictionary #2914

Closed

lgirdwood reviewed Aug 4, 2020

View reviewed changes

singalsu force-pushed the add_beamformer branch from 620f847 to 8ba8f7f Compare September 9, 2020 15:52

singalsu commented Sep 9, 2020

View reviewed changes

tools/topology/sof-apl-pcm512x.m4 Outdated Show resolved Hide resolved

singalsu commented Sep 9, 2020

View reviewed changes

juimonen reviewed Sep 10, 2020

View reviewed changes

singalsu commented Sep 10, 2020

View reviewed changes

singalsu force-pushed the add_beamformer branch from 8ba8f7f to 5507aaa Compare September 10, 2020 16:24

singalsu requested review from juimonen and removed request for tlauda September 10, 2020 16:28

singalsu commented Sep 10, 2020

View reviewed changes

singalsu commented Sep 11, 2020

View reviewed changes

singalsu added 9 commits September 11, 2020 19:42

Topology: Add TDFB setup m4 data blobs for mic arrays

c2e3292

These are data files created by the example scripts in tools/tune/tdfb. The generation is time consuming and requires Octave or Matlab. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Tools: Testbench: Add TDFB component UUID information

5f93f77

This patch adds the needed information to testbench to load the TDFB component. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu force-pushed the add_beamformer branch from 5507aaa to 47ea071 Compare September 11, 2020 17:58

singalsu requested review from cujomalainey and lgirdwood September 11, 2020 18:05

lgirdwood approved these changes Sep 13, 2020

View reviewed changes

lgirdwood merged commit 46e49cc into thesofproject:master Sep 13, 2020

singalsu deleted the add_beamformer branch January 25, 2021 11:15

Add multi-microphone beamformer component #2925

Add multi-microphone beamformer component #2925

Uh oh!

Conversation

singalsu commented May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lgirdwood commented May 13, 2020

Uh oh!

singalsu commented May 13, 2020

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

singalsu commented May 13, 2020

Uh oh!

singalsu commented May 13, 2020

Uh oh!

singalsu commented May 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cujomalainey commented May 13, 2020

Uh oh!

cujomalainey commented May 13, 2020

Uh oh!

singalsu commented May 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zrombel commented Aug 1, 2020

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singalsu commented Sep 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singalsu commented Sep 11, 2020

Uh oh!

lgirdwood commented Sep 13, 2020

Uh oh!

singalsu commented Sep 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

singalsu commented May 11, 2020 •

edited

Loading

singalsu commented May 13, 2020 •

edited

Loading

singalsu commented May 14, 2020 •

edited

Loading

singalsu commented Sep 14, 2020 •

edited

Loading