XCORE Design with additonal microphones of different purposes

New to XMOS and XCore? Get started here.
Post Reply
florian.ott
Member
Posts: 8
Joined: Mon Jun 26, 2017 1:08 pm

XCORE Design with additonal microphones of different purposes

Post by florian.ott »

Hello,

for our project (robotics) we want to extend the xCORE SMART MICROPHONE board with a lot of additonal microphones.
The goals are the following:
- have microphones for audio streaming for speech recognition
- have separate microphones for direction of arrival information, which will be used by the robot for navigation

we wanted to achieve this by:
- using the existing microphones on the board for direction of arrival calculation (existing demo functionality)
- transmit the DOA information over the USB interface
- add 14 additional MEM microphones (uniform circular array) to the board by using addiotional ports (expansion ports, LEDs)
- stream the audio gathered from the 14 microphones over USB to a host system running speech recognition software

We are not sure if this design is possible, mainly the following questions popped up:
- is the performance of the board (number of cores) sufficient?
- is is possible to separte the algorithms for the two different microphone purposes (DOA and streaming)?
- is it even possible to use the LED ports to gather audio information from additonal MEM microphones connected to them?

We would really appreciate your feedback.
Thanks and regards
Florian


User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

HI Florian & welcome to the forum.

So you are planning to have 6 mics for DoA and and additional 14 for streaming audio to the host. Total 20 ??

Firstly, you are not the first to ask about how many you can fit. Have a look at:

http://www.xcore.com/viewtopic.php?f=20&t=5773
and
http://www.xcore.com/viewtopic.php?f=3&t=5623
and
http://www.xcore.com/viewtopic.php?f=47&t=4879
But in short:
- is the performance of the board (number of cores) sufficient?
- Should be. You will need 3 PDM tasks and 5 decimators. That's a whole tile. SO you have a tile left for DoA and USB. You didn't mention the sample rate.. but 20 channels at 48kHz shouldn't be an issue for USB audio class 2
- is is possible to separte the algorithms for the two different microphone purposes (DOA and streaming)?
Not quite sure what you mean. You will basically have a frame of 20 channels per sample period. What you then do with those samples is up to you. No reason why you can't send the same data to two places
- is it even possible to use the LED ports to gather audio information from additonal MEM microphones connected to them?
The input to lib_mic_array is an 8b port. LEDs 0..7 are on XS1_PORT_8C so you could repurpose those as pdm mic input.
LEDs 10..12 are on XS1_PORT_8D which gets you to 19 but bit 3 of that port comes out on the header as X0D39. So the answer looks like (luckily) yes..

Note that overall this is quite a lot for an XU216 and so it's not a trivial project, but I am confident that it is feasible. It may be a bit of a squeeze to do without some tile placement of tasks flexibility (which you don't get with an off the shelf board) but you should be able to prove the concept on the dev board perhaps using just 16 mics initially.

Oh, this may also be helpful for adding a control path across USB..
http://www.xcore.com/viewtopic.php?t=5025
florian.ott
Member
Posts: 8
Joined: Mon Jun 26, 2017 1:08 pm

Post by florian.ott »

Thank you very much for the impressively fast response.

The sample rate we wanted to achieve is 44 kHz. Or should we use the decimation factor 2 to get 48 kHz? We got a requirement of 44 by the speech recognition software.

For the usage of LED ports, how about instead of LED 10...12 we use the expansion ports/header pins (X0D00, X0D11, X0D22, X0D23, X0D24, X0D25)?
I think then all microphones are connected to tile[0], where we could put the 3 PDM and 5 decimator tasks?

The locations of the microphones, like their purpose, we plan to separate. Although all microphones are connected to the same smart microphone board.
So there will be the smart microphone board with its existing microhones somewhere on top for DOA calculation.
If we know the direction of the voice, we can move our robot, so that the additonal microphones point to the speaker.
Those additonal microphones for streaming are located to the front in a uniform circular array similar to the microphones directly on the board.

DOA microphones:
Do we need the center microphone for DOA calculation?

Streaming microphones:
The audio from the DOA microphones will not be useful, because of their different location. Therefor we do not want to stream this audio.
Is there an influence of the DOA microphones on the streaming microphones? Especially considering any noise canceling algorithms?
As you said, there will then be a frame of 20 channels. So we could use the first 6 channels for DOA calculation and stream the other channels?

Additional:
I read, that the decimators need to be on the same tile as the application to share the memory. Does this mean we have another problem here? Or do we just need a separate way of transportation for the data between tiles?
So PDM and decimators on tile[0], transport somehow to tile[1], calculation and USB streaming on tile [1].
We do not need ethernet or I2S or I2C connections.
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

The sample rate we wanted to achieve is 44 kHz. Or should we use the decimation factor 2 to get 48 kHz? We got a requirement of 44 by the speech recognition software.
44.1kHz should be OK. You just need to set everything for 48kHz and then tune the MCLK from 24.576MHz to 22.5792MHz and all the filter bands will shift down nicely.
For the usage of LED ports, how about instead of LED 10...12 we use the expansion ports/header pins (X0D00, X0D11, X0D22, X0D23, X0D24, X0D25)?
I'm afraid that won't work. You need to map the pdm in[ut pins to a logical 8b port - the pdm front end is optimised to use this port type. It would require a re-write to use 1b ports instead.
I think then all microphones are connected to tile[0], where we could put the 3 PDM and 5 decimator tasks?
This is an issue. As you discovered below, the app and decimator share memory and so must be on the same tile. We only have 8 cores per tile, so it means that one of the PDM front ends MUST be on tile [1]...OR... one of decimators must be on tile[1] and you will need two cores receiving the data from the decimators. That's what I was hinting at in the last post but wasn't explicit about. There are a few options here but it's why you may struggle to fully utilise both tiles resources without some flexibility on port placement.

DOA microphones:
Do we need the center microphone for DOA calculation?
No. Mic 0 (center) is not passed to DoA:

Code: Select all

                /* Simply copy input to output buffer unmodified */
                for(int i = 0; i < 8; i++) 
                    output[i] = audio->data[i][0];
                 
                if(*doDoa && (*vadState == VAD_IDLE))
                {   
                    int doaDir = lib_voice_doa_naive_incorporate(doaState, output[1], output[2], output[3], output[4], output[5], output[6]);

Streaming microphones:
The audio from the DOA microphones will not be useful, because of their different location. Therefor we do not want to stream this audio.
Is there an influence of the DOA microphones on the streaming microphones?
None at all. They are all independent channels.
Especially considering any noise canceling algorithms?
Noise cancelling is only included on the XVSM3000 series devices. Currently this is done on a single channel which has been derived post beamformer and post AEC which uses 4 mics.
As you said, there will then be a frame of 20 channels. So we could use the first 6 channels for DOA calculation and stream the other channels?
Yes. You can stream any block of the raw decimated mics back to the host you want.
Additional:
I read, that the decimators need to be on the same tile as the application to share the memory. Does this mean we have another problem here? Or do we just need a separate way of transportation for the data between tiles?
Only channels or interfaces can get data across the tiles.

Connections between the PDM front end and decimators can be across tiles. It is just that the pdm_buffer() task must be on the same tile as all of the decimators.
So PDM and decimators on tile[0], transport somehow to tile[1], calculation and USB streaming on tile [1].
We do not need ethernet or I2S or I2C connections.
ok - but you will still need the audio task as this is the audio hub. You could merge pdm_buffer() and audio_io() into a single task saving one core if needed. Those tasks just pass data about and handle control requests from the USB host to change sample frequency etc.
florian.ott
Member
Posts: 8
Joined: Mon Jun 26, 2017 1:08 pm

Post by florian.ott »

Thanks again for your response.
So this basically means we have to be creative with the placement of the tasks on the tiles.
As I understood there are two options you mentioned.
1)
it means that one of the PDM front ends MUST be on tile [1]
2)
one of decimators must be on tile[1] and you will need two cores receiving the data from the decimators
Number 1) means, that the need a mic_array_pdm_rx() task on tile[1]. Therefor the new microphone ports (former LEDs) need to be mapped to tile[1]. Can this be done by simply moving the definition from tile[0] to tile[1] and renaming it correspondingly, e.g. to PORT_MIC_DATA1? We can then use it as input for the mic_array_pdm_rx() task.

Code: Select all

          
          <Tile Number="0" Reference="tile[0]"> 
            <!-- LED ports -->
            <Port Location="XS1_PORT_8C" Name="PORT_LED0_TO_7"/>
        
          </Tile>
          
          <Tile Number="1" Reference="tile[1]">
            <Port Location="XS1_PORT_8C" Name="PORT_MIC_DATA1"/>
          </Tile>
If we do this, all decimators (mic_array_decimate_to_pcm_4ch() tasks) need to be on tile[0] as well as all application tasks (streaming, DOA calculation)? I am not sure, if this helps giving us enough cores. Are there any additional application tasks?

Number 2) I do not fully understand. Maybe you mean to separate the DOA microphones and the streaming microphones. So we would have the decimators and application part for DOA calcuation and USB interface on tile[0] and streaming microphones and USB streaming on tile[1]?
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

Had a bit more of a think about it and you can do it..
IMG_20170706_154349.jpg
(66.78 KiB) Not downloaded yet
IMG_20170706_154349.jpg
(66.78 KiB) Not downloaded yet
You could potentially combine PDM BUF2 and AUD too.
florian.ott
Member
Posts: 8
Joined: Mon Jun 26, 2017 1:08 pm

Post by florian.ott »

We did some more thinking about the pins and ports to use. The plan was to have 14 additional microphones.
8 microphones on LED pins X0D26 – X0D33 (LED_0 to LED_7) -> no problems here for the existing algrorithms, mapped to port XS1_PORT_8C.
For the other microphones, we are still searching for a solution. Is it possible to map pins to ports ourself? Or do we have to use predefined ports.
We thought about the following pins to use as well:
4 microphonesX0D02 (BUTTON_A), X0D03 (BUTTON_B), X0D08 (BUTTON_C), X0D09 (BUTTON_D)
2 microphones on LED pins X0D36 – X0D37 (LED_10 to LED_11)
Can we map them together on the same 8bit port, e.g. XS1_PORT_8D or XS1_PORT_4A? And how? And can we then use the inputs of those microphones for one PDM task? Otherwise would we have to use only the 4 microhones here and have 12 as a total?
What happens to the unused ressources if we only have 6 of 8 channels in use? And how are the algorithms affected?
For using only 4 we think we could save cores.
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

We did some more thinking about the pins and ports to use. The plan was to have 14 additional microphones.
8 microphones on LED pins X0D26 – X0D33 (LED_0 to LED_7) -> no problems here for the existing algrorithms, mapped to port XS1_PORT_8C.
For the other microphones, we are still searching for a solution. Is it possible to map pins to ports ourself? Or do we have to use predefined ports.
Not 100% sure I understand the question. The processor core talks to logical ports P8A, P8B etc. The logical ports connect to physical pins on the device. Where the same pin connects to multiple logical ports (eg. X0D14), the active port with the smallest width wins.

For the mic array front end, it is hard coded to talk to a single 8b port. That will also define which physical pins it will connect to on the chip. You can pass in any 8b port on that tile. So you could use P8A or P8D etc..

This document can be helpful when planning port usage https://www.xmos.com/published/xcore-20 ... es-portmap
We thought about the following pins to use as well:
4 microphonesX0D02 (BUTTON_A), X0D03 (BUTTON_B), X0D08 (BUTTON_C), X0D09 (BUTTON_D)
2 microphones on LED pins X0D36 – X0D37 (LED_10 to LED_11)
Can we map them together on the same 8bit port, e.g. XS1_PORT_8D or XS1_PORT_4A? And how?
No - you can't use some bits of one 8b port and some from another. The API expects you to hand a single 8b port..
And can we then use the inputs of those microphones for one PDM task? Otherwise would we have to use only the 4 microhones here and have 12 as a total?
I am not sure I quite understand but the PDM task can only process the inputs from a single 8b port.
What happens to the unused ressources if we only have 6 of 8 channels in use? And how are the algorithms affected?
For using only 4 we think we could save cores.
If you only use 6 of 8 channels, then you still need 1x PDM front end. It's just 2 of the channels will be don't care. To decimate 6 channels you will need 2 x tasks because each task handles a bundle of 4 channels. You'll get 8 outputs from the 2 decimators but you only care about 6 of them.

One possibility is to consider a currently non-merged branch of lib_mic_array that uses one or two 4b ports instead of a single 8b port. That may increase your flexibility, but even though it has been tested, it's not officially supported yet. The API is:

void mic_array_pdm_rx_4_bit(
in buffered port:32 p_pdm_mics0,
in buffered port:32 ?p_pdm_mics1,
streaming chanend c_4x_pdm_mic_0,
streaming chanend ?c_4x_pdm_mic_1);

You can see the PR here https://github.com/xmos/lib_mic_array/pull/48
florian.ott
Member
Posts: 8
Joined: Mon Jun 26, 2017 1:08 pm

Post by florian.ott »

We would like to prototype our design on the existing microphone board by removing the buttons and LEDs and soldering additional microphones on the lines.
"the active port with the smallest width wins"
If we want to use the pins of the button (X0D02/3/8/9), we would just need to make sure, that the pins are only used by the 8-bit port P8A. Then it would be possible using this 8-bit port?
It seems like the additional pins on this port are not connected? Is there any place where we could connect a microphone? Otherwise they are just ignored?
FOr a short test we tried to switch the definition of the ports in the file mic_array_ref.xn in app_usb_aud_mic_array to use the button port for mic data (and mic port buttons, but they are not used). The test should output the signal of one connected microphone.
I was told that it seems like there is a pull-up resistor on the button pins because we got no signal? Can we disable this somehow?

What would be the problem if we used additional pins of a logical 8-bit port on tile 1? In your painting one decimator was on tile 1.

Do the not used channels of a logical 8-bit port (don't care) influence the algorithms on the DSP like beamforming etc.? Or are they automatically ignored if there is no signal?

Do we need to adapt anything else for our own additinal uniform circular array of the additional microphones, as there might be other delays than on the existing board? e.g. filtering, ...
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

We would like to prototype our design on the existing microphone board by removing the buttons and LEDs and soldering additional microphones on the lines.
OK - that does place constraints on your setup. The devil will be in the detail as to how this all hangs together.
If we want to use the pins of the button (X0D02/3/8/9), we would just need to make sure, that the pins are only used by the 8-bit port P8A. Then it would be possible using this 8-bit port?
Yes exactly.
It seems like the additional pins on this port are not connected? Is there any place where we could connect a microphone? Otherwise they are just ignored?
The pins are not connected but if you are feeling brave with some mod wire you could tack on to the QFP pins directly.
FOr a short test we tried to switch the definition of the ports in the file mic_array_ref.xn in app_usb_aud_mic_array to use the button port for mic data (and mic port buttons, but they are not used). The test should output the signal of one connected microphone.
I was told that it seems like there is a pull-up resistor on the button pins because we got no signal? Can we disable this somehow?
There are pull up resistors on the board. See - https://www.xmos.com/download/private/x ... 1v1%29.pdf
What would be the problem if we used additional pins of a logical 8-bit port on tile 1? In your painting one decimator was on tile 1.
You could do that. Maybe hold the ETH phy in reset and get at the pins of P8A..
The main thing is that the receiving task has to be on the same tile as the decimator because they share memory.
Do the not used channels of a logical 8-bit port (don't care) influence the algorithms on the DSP like beamforming etc.? Or are they automatically ignored if there is no signal?
Do we need to adapt anything else for our own additinal uniform circular array of the additional microphones, as there might be other delays than on the existing board? e.g. filtering, ...
I am not sure exactly what you mean - the digital/firmware side of things will be perfectly synchronised.
Post Reply