Audio: TDFB: Track sound direction of arrival #4826

singalsu · 2021-09-29T14:13:06Z

This patch adds sound direction tracking. TDFB component updates
the sound direction control to user space. User space may control
the beam direction with help of this.

Signed-off-by: Seppo Ingalsuo seppo.ingalsuo@linux.intel.com

singalsu · 2021-09-29T14:17:40Z

First draft, needs a lot of cleanup and testing and optimization. It tracked in simulated 2mic acoustics in testbench quite well a 360 degree rotating sound source within -90 .. +90 deg that the 2D array can handle.

tools/tune/tdfb/bf_design.m

tools/test/audio/tdfb_test.m

tools/test/audio/process_test.m

tools/test/audio/comp_run.sh

src/audio/tdfb/tdfb_generic.c

src/audio/CMakeLists.txt

src/audio/tdfb/tdfb.c

src/audio/tdfb/tdfb_math.c

tools/tune/eq/eq_uint32t_write.m

src/audio/tdfb/tdfb_direction.c

src/audio/tdfb/tdfb.c

src/audio/tdfb/tdfb_direction.c

singalsu · 2021-10-04T16:45:20Z

I just pushed a version that "worked" correctly in a notebook device. I will next tune it to be more responsive, now only fairly loud sounds trigger angle update. Also I should change the enum scale for line array topologies for better precision. Alsatplg allows 16 different values and now 12 is used to cover 0-330 degrees. I could use 15 steps for -90..0..+90 with ~13 degrees precision. Or 13 steps with exactly 15 degrees step.

lgirdwood · 2021-10-15T14:54:48Z

I just pushed a version that "worked" correctly in a notebook device. I will next tune it to be more responsive, now only fairly loud sounds trigger angle update. Also I should change the enum scale for line array topologies for better precision. Alsatplg allows 16 different values and now 12 is used to cover 0-330 degrees. I could use 15 steps for -90..0..+90 with ~13 degrees precision. Or 13 steps with exactly 15 degrees step.

Best to use integer step size for alsamixer rendering.

singalsu · 2021-10-18T08:09:57Z

Best to use integer step size for alsamixer rendering.

Yep, my working version is using exact 15 degree step for line arrays now. I will share soon a testbench PR to enable testing of different input and output channels count in pipeline and improved TDFB test scripts. Then share the update to this PR.

singalsu · 2021-11-17T16:34:07Z

This PR needs #4941. I will meanwhile share related parts to this work (TDFB comp bug fixes, TDFB tool improvement, testbench improvement, TDFB test improvement). Also avoids huge single PR since I had to change plenty of things to be able to test this. This version with internal math functions remains [WIP].

lgirdwood · 2021-11-18T16:30:16Z

@singalsu ok pls work with @ShriramShastry for #4941. Changes are localized so can be included for rc2

lgirdwood · 2021-11-23T13:30:20Z

@singalsu ready for review now ?

tools/topology/topology1/m4/tdfb/coef_circular8_100mm_az0el0deg_16khz.m4

singalsu · 2021-11-26T17:02:32Z

@singalsu ready for review now ?

Sorry, not yet. Need to solve issue in #4941 to compute square root, the power function there is not suitable for that. There's a another PR coming for square root. Backup is to convert power function part of DRC to math library.

singalsu · 2021-11-26T18:06:39Z

@lgirdwood I'm now testing this with PR #5034 .

singalsu · 2021-11-30T16:46:36Z

New WIP draft with Sriram's sqrt() lib patch temporarily included.

lyakh · 2021-12-22T08:23:20Z

src/audio/tdfb/tdfb.c

is this the same as ((a + 180) % 360) - 180? Maybe the below can be similarly simplified too, but be careful with division remainder of negative numbers

Thanks, that worked, and slightly different version for negative angles.

lyakh · 2021-12-22T08:51:31Z

src/audio/tdfb/tdfb_direction.c

nitpicking: how about

cd->direction.rp = cd->direction.d; cd->direction.wp = cd->direction.d + ch_count * (cd->direction.max_lag + 1);

Good catch, this was a quick bug fix test that I forgot to clean up.

lyakh · 2021-12-22T08:53:08Z

src/audio/tdfb/tdfb_direction.c

you could use goto for clean up

lyakh · 2021-12-22T08:55:59Z

src/audio/tdfb/tdfb_direction.c

cd->direction.trigger <<= 1;

src/audio/tdfb/tdfb_direction.c

lyakh · 2021-12-22T09:11:17Z

src/audio/tdfb/tdfb_direction.c

something above makes me suspicious. If the first test ds2 < ds1 was true, 360 was subtracted from az. And now you add 360 back to az for another test? An else missing somewhere?

There could be an else. The second squared error eval is unnecessary if the first 360 deg addition was done. The second test is false if first was true. One multiplication saved in that case.

lyakh · 2021-12-22T09:26:25Z

src/include/sof/audio/tdfb/tdfb_comp.h

unsigned int?

OK, changed!

src/audio/tdfb/tdfb_direction.c

src/include/sof/math/numbers.h

lgirdwood · 2021-12-23T20:34:24Z

@cujomalainey good for you ?

singalsu · 2022-01-18T15:15:37Z

I just updated the emphasis IIR response from 2nd order 2 kHz high-pass to 1 - 4 kHz 4th order band-pass. It prevents highest frequencies to alias spatially and add error to angle. The lower high-pass picks up a bit more voice signal while still attenuates lowest frequencies those are usually noise. No other changes.

src/audio/tdfb/tdfb_direction.c

lgirdwood · 2022-01-26T11:04:10Z

@singalsu any updates ?

src/audio/tdfb/tdfb_direction.c

The DF2T IIR core has been moved earlier to library but the helper functions for configuring it were left to EQ component. This patch eases use of IIR core from other components. The file src/audio/eq_iir/iir.c is moved to src/math. There are no changes except of remove iir.h since the contents specific for DF2T type were merged to iir_df2t.h Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds function cross_product_s16(). Cross product is commonly used function in computational geometry. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

This patch adds sound direction estimation. TDFB component updates the sound direction control to user space. User space may control the beam direction towards the source with help of the reported angle. Sound direction estimation is based on cross correlation maximums pattern and match of theoretical propagation delays to measured pattern. Cross correlation search is triggered by short term level when it exceeds the estimated background noise level. An emphasis filter is used to filter out typical low frequency noise. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Comment text is added to tdfb.c to help find the settings generator shell script. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu · 2022-02-02T15:51:06Z

I just added a commit that adds to tdfb.c a comment about settings generator script to help find it. No other changes done.

lgirdwood · 2022-02-03T16:13:29Z

@cujomalainey good for you ?

singalsu commented Sep 29, 2021

View reviewed changes

tools/tune/tdfb/bf_design.m Outdated Show resolved Hide resolved

singalsu commented Sep 29, 2021

View reviewed changes

tools/test/audio/tdfb_test.m Outdated Show resolved Hide resolved

singalsu commented Sep 29, 2021

View reviewed changes

tools/test/audio/process_test.m Outdated Show resolved Hide resolved

singalsu commented Sep 29, 2021

View reviewed changes

tools/test/audio/comp_run.sh Outdated Show resolved Hide resolved

singalsu commented Sep 29, 2021

View reviewed changes

src/audio/tdfb/tdfb_generic.c Outdated Show resolved Hide resolved

lgirdwood requested review from cujomalainey, johnylin76 and lyakh September 29, 2021 15:57

singalsu commented Sep 29, 2021

View reviewed changes

src/audio/CMakeLists.txt Outdated Show resolved Hide resolved

singalsu commented Sep 29, 2021

View reviewed changes

src/audio/tdfb/tdfb.c Outdated Show resolved Hide resolved

singalsu commented Sep 29, 2021

View reviewed changes

src/audio/tdfb/tdfb_math.c Outdated Show resolved Hide resolved

singalsu force-pushed the tdfb_direction branch from 87bbba2 to 5c0285a Compare October 1, 2021 16:45

singalsu commented Oct 1, 2021

View reviewed changes

tools/tune/eq/eq_uint32t_write.m Outdated Show resolved Hide resolved

lgirdwood reviewed Oct 4, 2021

View reviewed changes

singalsu force-pushed the tdfb_direction branch from 5c0285a to 5d03612 Compare October 4, 2021 16:37

singalsu requested a review from ShriramShastry October 11, 2021 07:06

singalsu force-pushed the tdfb_direction branch from 5d03612 to 4ca2747 Compare November 18, 2021 18:44

singalsu commented Nov 25, 2021

View reviewed changes

tools/topology/topology1/m4/tdfb/coef_circular8_100mm_az0el0deg_16khz.m4 Outdated Show resolved Hide resolved

singalsu force-pushed the tdfb_direction branch from 4ca2747 to 0f71036 Compare November 30, 2021 16:45

singalsu force-pushed the tdfb_direction branch from 0f71036 to f8dba81 Compare November 30, 2021 17:26

lyakh reviewed Dec 22, 2021

View reviewed changes

singalsu commented Dec 22, 2021

View reviewed changes

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

singalsu commented Dec 22, 2021

View reviewed changes

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

singalsu force-pushed the tdfb_direction branch from cbf96c8 to 98a604f Compare December 22, 2021 13:34

singalsu requested review from cujomalainey and lyakh December 22, 2021 14:22

singalsu commented Dec 22, 2021

View reviewed changes

src/include/sof/math/numbers.h Outdated Show resolved Hide resolved

singalsu force-pushed the tdfb_direction branch 3 times, most recently from 310d25b to aa32c9f Compare December 22, 2021 18:49

lyakh approved these changes Dec 23, 2021

View reviewed changes

singalsu force-pushed the tdfb_direction branch from aa32c9f to 8c2432f Compare January 18, 2022 15:11

cujomalainey reviewed Jan 18, 2022

View reviewed changes

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

lgirdwood added this to the v2.1 milestone Jan 25, 2022

singalsu force-pushed the tdfb_direction branch from 8c2432f to 4851a52 Compare January 27, 2022 17:40

singalsu requested a review from cujomalainey January 27, 2022 17:41

lgirdwood approved these changes Jan 31, 2022

View reviewed changes

src/audio/tdfb/tdfb_direction.c Outdated Show resolved Hide resolved

singalsu added 4 commits February 2, 2022 17:38

Math: Add cross product function to sof/math/numbers.h

1dcbdb3

This patch adds function cross_product_s16(). Cross product is commonly used function in computational geometry. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Audio: TDFB: Add note how to updata the settings data files

a6f06d0

Comment text is added to tdfb.c to help find the settings generator shell script. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

singalsu force-pushed the tdfb_direction branch from 4851a52 to a6f06d0 Compare February 2, 2022 15:46

cujomalainey approved these changes Feb 3, 2022

View reviewed changes

lgirdwood merged commit f753241 into thesofproject:main Feb 4, 2022

singalsu deleted the tdfb_direction branch September 16, 2022 07:54

Audio: TDFB: Track sound direction of arrival #4826

Audio: TDFB: Track sound direction of arrival #4826

Uh oh!

Conversation

singalsu commented Sep 29, 2021

Uh oh!

singalsu commented Sep 29, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singalsu commented Oct 4, 2021

Uh oh!

lgirdwood commented Oct 15, 2021

Uh oh!

singalsu commented Oct 18, 2021

Uh oh!

singalsu commented Nov 17, 2021

Uh oh!

lgirdwood commented Nov 18, 2021

Uh oh!

lgirdwood commented Nov 23, 2021

Uh oh!

Uh oh!

singalsu commented Nov 26, 2021

Uh oh!

singalsu commented Nov 26, 2021

Uh oh!

singalsu commented Nov 30, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lgirdwood commented Dec 23, 2021

Uh oh!

singalsu commented Jan 18, 2022

Uh oh!

Uh oh!

Uh oh!