PALM: Few-Shot Prompt Learning for Audio Language Models

ZeroShot Inference in ALMs - Primer

ZeroShot inference in audio-language models (ALMs) refers to making predictions on new, unseen data without specific training. Let's denote an ALM with \(f_{\theta} = \{f_{_{A}},f_{_{T}}\}\), whereas \(f_{_{A}}\) and \(f_{_{T}}\) are audio and text encoders, respectively. For classification in zero-shot scenario, the audio \(\boldsymbol{\mathrm{x}}\) is first passed to the audio encoder \(f_{_{A}}\), resulting in a \(d-\) dimensional feature vector \(f_{_{A}}(\boldsymbol{\mathrm{x}}) \in \mathbb{R}^{d}\). Similarly, on the text encoder side, each class label \(y_i \in \{\mathit{y}_{1}, \mathit{y}_{2}, \dots, \mathit{y}_{C} \}\) is wrapped within the class-specific text template, such as: $$t_i = \mathrm{''An~audio~recording~of~\{CLASS~y_i\}.''}$$ Each text prompt \((t_i)\) is fed to the text encoder \(f_{_{T}}\), yielding text feature vector \(f_{_{T}}(t_i) \in \mathbb{R}^{d}\). The relationship between the audio's feature vector and the text prompt feature vector is quantified using cosine similarity, \(\mathtt{sim}(f_{A}(\boldsymbol{\mathrm{x}}),f{_{T}}(t_i))\), to evaluate the audio's alignment with \(i_{\text{th}}\) class. Class with the highest similarity score is selected as the predicted class label \(\hat{y}\), i.e. $$\hat{y} = \underset{ i\in \{1,2,\dots,C\} }{\mathbf{argmax}} ~~~ \mathtt{sim}\big(f_{_{A}}(\boldsymbol{\mathrm{x}})~,~f_{_{T}}(t_i)\big)$$

ZERO SHOT The audio feature vector is compared with the feature vector of text prompt (of each class) using cosine similarity. The class with the highest similarity score is then assigned to the input audio.

Prompt Learning

Zero-shot inference in vision-language-models (VLMs) and audio-language-models (ALMs) relies on manually crafted text prompts, which significantly impact performance. Prompt Learning, as explored by Gu et al. 2023, automates this by learning text prompts from training data, eliminating manual effort. The first notable method, COOP, learns the context of text prompts in the token-embedding space using few-shot training setup. This compute-efficient approach improves VLMs' performance in downstream tasks, requiring only a small data subset to learn prompts.

COOP learns the context of text prompts in the token-embedding space by minimizing the cross-entropy loss using few-shot training setup, improving performance in downstream tasks.

Prompt learning in ALMs is a relatively new and under-explored area of research. In this work, we first show the efficacy of prompt learning methods (originally introduced for VLMs) in ALMs and then propose a novel method, PALM, that optimizes the feature space of the text encoder branch in ALMs. Our method is either on par with or outperforms baseline approaches while being computationally less demanding.

PALM

PALM (Prompt Learning for Audio Language Models) method does not require hand-crafted prompts; instead, it simply uses class names as the input to the text encoder i.e. \(t_i =\mathrm{''\{CLASS~y_i\}''}\). Moreover, unlike COOP, which learns the context of input text prompts in the token embedding space, PALM learns the context in the feature space of prompts. Specifically, after obtaining the feature vector of the \(i_{\text{th}}\) class text prompt via the text encoder, i.e., \(f_{_{T}}(t_i) \in \mathbb{R}^{d}\), it adds a learnable vector \(z_i \in \mathbb{R}^{d}\) to it to get the updated text feature vector as follows: $$f_{_{T}}^{\prime}(t_i) = (1-\lambda_i)\cdot f_{_{T}}(t_i)~+~\lambda_i \cdot z_i$$ where \(\lambda_i \in [0,1]\) is a learnable parameter that determines the contributions of both vectors. Assuming \(\boldsymbol{\mathrm{t}}=\{t_1,t_2,\dots,t_C\}\) denotes text prompts of all classes, the raw/un-normalized prediction scores (logits), denoted as \(f_{_{\theta}}(\boldsymbol{\mathrm{x}},\boldsymbol{\mathrm{t}}) \in \mathbb{R}^{C}\), for an audio waveform \((\boldsymbol{\mathrm{x}})\) are obtained as follows: $$f_{_{\theta}}(\boldsymbol{\mathrm{x}},\boldsymbol{\mathrm{t}}) = \bigg\{~\mathtt{sim}\bigg(f_{_{A}}(\boldsymbol{\mathrm{x}})~,~f_{_{T}}^{\prime}(t_i)\bigg)~\bigg\}_{i=1}^{C},$$ where \(\texttt{sim}(\cdot)\) is cosine-similarity function and \(C\) is the number of classes. \(f_{_{A}}(\boldsymbol{\mathrm{x}})\) is the feature vector from the audio encoder, and \(f_{_{T}}^{\prime}(t_i)\) is the updated text feature vector of \(i_{\text{th}}\) class. Following objective function is optimized to learn feature-space context embeddings \(\boldsymbol{\mathrm{z}}=\{z_1,z_2,\dots,z_C\}\) and their corresponding contributions \(\lambda=\{\lambda_1,\lambda_2,\dots,\lambda_C\}\): $$\underset{ \boldsymbol{\mathrm{z}}~,~\lambda }{\mathbf{minimize}}~~ \sum_{(\boldsymbol{\mathrm{x}},y)\in\mathcal{D}} \mathcal{L}\big(f_{_{\theta}}(\boldsymbol{\mathrm{x}},\boldsymbol{\mathrm{t}}),y\big),$$ where \(\mathcal{D}=\{\boldsymbol{\mathrm{x}}_i,y_i\}_{i=1}^{N}\) is training dataset consisting of \(N\) audio-class pairs and \(\mathcal{L}(\cdot)\) denotes cross-entropy loss. After learning the parameters, following equation is used for audio classification during inference stage: $$\hat{y} = \underset{ i\in \{1,2,\dots,C\} }{\mathbf{argmax}} ~~~ \mathtt{sim}\big(f_{_{A}}(\boldsymbol{\mathrm{x}})~,~f_{_{T}}^{\prime}(t_i)\big)$$ An overview of our proposed approach (PALM) can be found in the following figure.

PALM optimizes the feature space of text prompts in audio-language models by adding the learnable vectors to text features and minimizing the cross-entropy loss using few-shot training setup, enhancing performance without the need for hand-crafted prompts while being computationally efficient.