多说话人分离技术及应用进展分析报告-培训课件

多说话人分离技术及应用进展2024.3纲

要1.研究背景2.工业版本—模块化系统3.

改进方案4.落地应用1.研究背景多说话人分离（说话人日志）：给定一个包含多人交替说话的语音，系统需要判断每个时间段是谁在说话。音频分割信息多说话人分离系统1.研究背景应用场景：会议纪要，多说话人转录，智能客服，录音质检等

...终端设备：录音笔智能手机个人电脑支持厂商：科大讯飞（智能办公本）、（AI纪要）、声云（语音转写）

...1.研究背景DIHARD(I,II,III)CHiME-6VoxSRC(20,21,22,23)AliMeetingRichTranscription（RT）MIXER62013CALLHOMEAMI竞赛/数据集M2MeT,

AISHELL-4

M2MeT2.0,

CHiME-7200020022006

2009201820192020202120222023模块化架构架构端到端架构研究趋势：简单场景→复杂场景挑战：噪声干扰，人数未知，语音重叠等应用：离线=>在线，单麦克风=>多麦克风，适配新场景1.研究背景—模块化系统聚类方法：AHC[1]、SC

[2,3]

、VB

/VBx

[4,5]

、UIS-RNN

[6]

、DNC

[7][1]

K.

C.

Gowda

and

G.

Krishna,

“Agglomerative

Clustering

Using

the

Concept

of

Mutual

Nearest

Neighbourhood,”

Pattern

Recognition,

vol.

10,

pp.

105–112,

1978.[2]

U.

von

Luxburg,

“A

tutorial

on

spectral

clustering,”

Statistics

and

Computing,

vol.

17,

pp.

395–416,

2007.[3]

T.

Park,

Kyu

J.

Han,

Manoj

Kumar,

and

Shrikanth

S.

Narayanan,

“Auto-tuning

Spectral

Clustering

for

Speaker

Diarization

Using

Normalized

Maximum

Eigengap,”

IEEE

SignalProcessing

Letters,

vol.

27,

pp.

381–385,

2020.[4]

M.

Diez,

L.

Burget,

S.

Wang,

J.

Rohdin,

H.

Cernocky,

“Bayesian

HMM

based

x-vector

Clustering

for

Speaker

Diarization,”

Interspeech,

2019,

pp.346-350.[5]

M.

Diez,

L.

Burget,

F.

Landini,

J.

Cernocky,

"Analysis

of

Speaker

Diarization

based

on

Bayesian

HMM

with

Eigenvoice

Priors,"

IEEE/ACM

Transactions

on

Audio

Speech

andLanguage

Processing,

vol.

28,

p

355-368,

2020.[6]

A.

Zhang,

Q.

Wang,

Z.

Zhu,

J.

Paisley,

and

C.

Wang,

“Fully

Supervised

Speaker

Diarization,”

ICASSP,

2019.[7]

Q.J.

Li,

F.

L.

Kreyssig,

C.

Zhang,

P.C.

Woodland,

“Discriminative

Neural

Clustering

for

Speaker

Diarisation,”

IEEE

Spoken

Language

Technology

Workshop

(SLT

2021),

Jan

2021,Shenzhen,

China.1.研究背景—端到端系统基于Bi-LSTM的端到端模型EEND[1]SA-EEND[2]基于Transformer

encoder的端到端模型端到端系统EDA-EEND[3]可以预测人数的EEND模型…TS-VAD[4]目标说话人音频端点检测模型[1]

Y.

Fujita,

N.

Kanda,

S.

Horiguchi,

K.

Nagamatsu,

and

S.

Watanabe,

“End-to-end

Neural

Speaker

Diarization

with

Permutation-free

Objectives,”

in

Interspeech,

2019,

pp.

4300–4304.[2]

Y.

Fujita,

N.

Kanda,

S.

Horiguchi,

Y.

Xue,

K.

Nagamatsu

and

S.

Watanabe,

“End-to-End

Neural

Speaker

Diarization

with

Self-Attention,”

2019

IEEE

Automatic

Speech

Recognitionand

Understanding

Workshop

(ASRU),

SG,

Singapore,

2019,

pp.

296-303.[3]

S.

Horiguchi,

Y.

Fujita,

S.

Watanabe,

Y.

Xue,

and

K.

Nagamatsu,

“End-to-end

speaker

diarization

for

an

unknown

number

of

speakers

with

encoder-decoder

based

attractors,”

inInterspeech,

2020,

pp.

269–273.[4]

I.

Medennikov,

M.

Korenevsky,

et

al.,

“Target-speaker

Voice

Activity

Detection:

a

Novel

Approach

for

Multi-speaker

Diarization

in

a

Dinner

Party

Scenario,”

arXiv,

vol.abs/2005.07272,

2020.1.研究背景—聚类算法汇总聚类算法AHC训练方式无监督聚类无监督聚类无监督聚类无监督聚类有监督聚类有监督聚类有监督聚类有监督聚类输入特征x-vectori-vectorx-vectorx-vectord-vectord-vector声学特征i-vector重叠检测不支持不支持不支持不支持不支持支持预测人数阈值VB初始化调节初始化调节阈值/NME适合2人VBxSCUIS-RNNDNC输出节点输出节点输出节点EENDTS-VAD支持支持在线版本：研究主要集中在EEND[1,2]或UIS-RNN[3,4]框架麦阵版本：多通道输入TS-VAD[5]或前后端联合优化特定场景：不同场景采用不同策略[6][1]

Y.

Xue,

S.

Horiguchi,

Y.

Fujita,

S.

Watanabe,

P.

Garcia,

and

K.

Nagamatsu,

“Online

end-to-end

neural

diarization

with

speaker-tracing

buffer,”

in

IEEE

Spoken

LanguageTechnology

Workshop

(SLT),

2021,

pp.

841–848.[2]

E.

Han,

C.

Lee,

and

A.

Stolcke,

“Bw-eda-eend:

Streaming

end-toend

neural

speaker

diarization

for

a

variable

number

of

speakers,”

in

ICASSP,

2021.[3]

E.

Fini

and

A.

Brutti,

“Supervised

online

diarization

with

sample

mean

loss

for

multi-domain

data,”

in

ICASSP,

2020,

pp.

7134–7138.[4]

X.

Wan,

K.

Liu,

H.

Zhou,

"Online

speaker

diarization

equipped

with

discriminative

modeling

and

guided

inference,”

in

Interspeech,

2021.[5]

I.

Medennikov,

M.

Korenevsky,

et

al.,

“Target-speaker

Voice

Activity

Detection:

a

Novel

Approach

for

Multi-speaker

Diarization

in

a

Dinner

Party

Scenario,”

arXiv,

vol.abs/2005.07272,

2020.[6]

Y.-X.

Wang,

J.

Du,

M.-K.

He,

S.-T.

Niu,

L.

Sun,

C.-H.

Lee,

"Scenario-dependent

speaker

diarization

for

DIHARD-III

challgenge,"

in

Interspeech,

2021.2.工业版本—模块化系统2.1

分割音频功能：转换为聚类问题2.工业版本—模块化系统2.2

提取说话人表征vectorvectorvector功能：提取段级别说话人表征2.工业版本—模块化系统2.3

聚类—凝聚层次聚类(AHC)AHC功能：对相同说话人片段聚类K.

C.

Gowda

and

G.

Krishna,

“Agglomerative

Clustering

Using

the

Concept

of

Mutual

Nearest

Neighbourhood,”

Pattern

Recognition,

vol.

10,

pp.

105–112,

1978.2.工业版本—模块化系统第一代产品（与ASV-Subtools*结合）语音端点检测*/Snowdar/asv-subtools说话人日志（SD）语音识别识别后处理（VAD）(ASR)说话人1说话人2说话人3说话人4原始音频算法流程：VAD->平均分割->Subtools提取x-vector->PCA降维->Cosine打分->AHC聚类2.工业版本—模块化系统存在问题—语音重叠说话人混叠：

目标区域是否发生了说话重叠？

谁和谁发生了重叠？图片和音频：https://herve.niderb.fr/fastpages/2022/10/23/One-speaker-segmentation-model-to-rule-them-all3.改进方案—神经网络分割解决办法：进行分段，每段用神经网络判断说话人，最多3人。图片：https://herve.niderb.fr/fastpages/2022/10/23/One-speaker-segmentation-model-to-rule-them-all3.改进方案—神经网络分割解决办法：进行分段，每段用神经网络判断说话人，最多3人。每5秒一段，窗移2.5秒图片：https://herve.niderb.fr/fastpages/2022/10/23/One-speaker-segmentation-model-to-rule-them-all3.改进方案—神经网络分割解决办法：进行分段，每段用神经网络判断说话人，最多3人。提取x-vector时，去除重叠语音，并合并同一人语音H.Bredin,

“Pyannote.audio

2.1speaker

diarization

pipeline:

principle,

benchmark,

and

recip