Publications

Abstract

Historically, criminal investigations hinging on recorded audio data required manual application of forensic techniques to extract relevant information. These methods usually focus mainly on voices and speaker identification, but rarely focus on the wealth of forensic information available in the background noises present in the recording. Our paper introduces methods of automatically extracting, separating, and classifying background noises, allowing for the difficult, time-consuming process of audio analysis to be handled by software. Once the audio has been classified and examined by our proposed tools, the results can be used by investigators and forensic experts to aid in traditional investigative methods. Using environment information as an example, we propose a fully automated environment inference process based on background noise. Detailed experimental results show that our framework is effective and fast. Our proposed framework intends to provide a neat, automated, and accurate analysis of the information present in background audio, and to provide a new source of forensic information for investigators to leverage. In contrast to existing similar work, our scheme not only realistically considers mixed human voice speech, but also considers the case of multiple background noise mixes. To the best of our knowledge, this is the first forensic work that considers background noise in a complex environment.

Abstract

With the rapid development of machine learning technologies, voiceprint has become widely used as a personal identifier in daily life. Because of that, it is essential to determine to what extent a voiceprint derived from machine learning can be inverted to obtain the original speaker characteristic. However, the reconstruction of voiceprint templates is still a challenging issue. It has also not been proven whether the widespread use of voiceprint poses a privacy leakage risk. In this paper, we implement the first comprehensive, holistic, and systematic reconstruction study targeting voiceprint templates. We present Voxstructor, a voiceprint-based voice constructor that can be used for bulk template reconstruction attacks. An attacker can reconstruct a new voice based only on the victim's voiceprint data instead of the voice itself. Specifically, we formalize the voice reconstruction work as an objective optimization problem and merge voice cloning with voiceprint template conversion work. We have conducted extensive experiments on multiple mapping models, loss functions, voiceprint template extraction models, scoring methods, and two types of speaker verification attacks. Thorough experiments show that our attacks are effective, achieving a fairly high success rate which is similar to the results generated by voice cloning methods. The time overhead of Voxstructor is far less than other attacks. Our study not only demonstrates the need for protection of voiceprint templates in speaker recognition systems, but also shows that Voxstructor can be used as a privacy measure tool for voiceprint privacy-preserving schemes.