An Embarrassingly Simple Approach for LLM with Strong ASR …
2024年2月13日 · To be more specific, we benchmark and explore various combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup and little task-specific design, where only the linear projector is trained.
Leveraging Large Language Models for Exploiting ASR Uncertainty
In this work, we focus on the latter setup where the accuracy of LLM on SLU tasks is constrained by the accuracy of a frozen ASR system on the given speech input.
In this paper, we focus on solving one of the most important tasks in the field of speech processing, i.e., automatic speech recognition (ASR), with speech foundation encoders and large language models (LLM).
X-LLM
We propose X-LLM, a Multimodal LLM which injects multiple modalities (such as images, speech, and videos) into the LLM through X2L interfaces, giving the LLM the ability to process multimodal data. This method has good scalability and can be extended to more modalities.
An Embarrassingly Simple Approach for LLM with Strong ASR …
We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task.
GitHub - Tencent/VITA
2024年8月12日 · VITA is capable of processing inputs in the form of pure text/audio, as well as video/image combined with text/audio. Besides, two key techniques are adopted to advance the multimodal interactive experience: State token.
[2405.02132] Unveiling the Potential of LLM-Based ASR on …
2024年5月3日 · Large Language Models (LLMs) have demonstrated unparalleled effectiveness in various NLP tasks, and integrating LLMs with automatic speech recognition (ASR) is becoming a mainstream paradigm....
Leveraging Large Language Models for Exploiting ASR Uncertainty
2023年9月9日 · Instead of chasing a high accuracy by designing complex or specialized architectures regardless of deployment costs, we seek to answer how far we can go without substantially changing the underlying ASR and LLM, which …
To validate this hypothesis and endow LLM with multimodal capabilities, we propose X-LLM. It converts multimodal information, such as images, speech, and videos, into foreign languages using X2L interfaces, and then feeds converted multimodal information into a …
Paper page - X-LLM: Bootstrapping Advanced Large Language …
2023年5月6日 · And we also conduct quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote the era of LLM-based speech recognition. Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
- 某些结果已被删除