AudioSep-Hive: turning language into a query for open-world sound separation
CLAP conditioning, time-frequency masking and mixture consistency in one query-driven model trained across 283 sound concepts.
SOUND SEPARATION / TECHNICAL NOTE
A fixed-stem separator decides the answer space before inference. AudioSep-Hive reads a description first and uses it to decide which component of the mixture is the target.
The conditioned objective
For x(t)=s(t)+n(t), the text encoder yields q=E_text(c). The separator estimates M=F(X,q) and reconstructs Ŝ=M⊙X. A compact view of the objective is:
L = ||ŝ-s||₁ + λ[1-cos(E_audio(ŝ),q)]
The semantic term makes related descriptions occupy a continuous query space. It does not remove ambiguity: broad prompts, weak targets and overlapping semantic descriptions remain difficult cases.
Scope of the output
The released inference path is primarily 32 kHz mono. It is useful for language-queried extraction and open-domain research, but it is not a replacement for 44.1 kHz stereo production stems. Changing the description changes the condition vector and may move the target boundary.
From AudioSep to Hive
AudioSep introduced open-domain separation through natural-language queries in 2023. Hive extends that line through semantically consistent data construction and data-efficient training. The paper, Hive repository and model card respectively document the method, dataset implementation and released weights.
The query became a variable beyond fixed stems
Traditional music separators define vocals, drums and bass before training. AudioSep connected CLAP text-audio representations to a separator, allowing targets such as a siren, tenor saxophone or female speech. Hive continues that line with larger weakly labeled audio-text coverage for open categories.
Open vocabulary does not mean 283 independent checkpoints. Training classes provide acoustic and language coverage; the text vector is what changes at inference. Moving from 'dog' to 'small dog barking behind music' shifts the condition, but language cannot create a separable source boundary that is absent from the mixture.
How a language query becomes a time-frequency mask
ŝ_q=iSTFT(Mθ(X,e_q)⊙X)Xmixture STFTe_qtext-query embeddingMθquery-conditioned mask
Query q is embedded in a shared audio-language space as e_q, which conditions a mask estimator. The 283 labels describe training coverage; they are not 283 fixed output stems in one task.
One sound has many phrasings, and negation or relations can change the result. Hold the audio fixed, paraphrase the query, then inspect target retention, leakage and mixture consistency.
Paper, code and weights can be followed through Hive source / AudioSep-Hive weights / AudioSep paper.
Open queries versus fixed stems
This compares input contracts and validation, not scores from incompatible tasks.
| Route | Query | Primary acceptance check |
|---|---|---|
| AudioSep-Hive | natural-language target | does the output match the requested sound |
| fixed music stems | predefined classes | cross-bleed and stereo preservation |
| sound event detection | class plus time span | localizes events but does not return a clean waveform |
Writing a query is acoustic annotation
A broad noun makes the model compromise among candidates; a very long description may lose rare modifiers. Start with the audible source, then add location, register or action instead of an abstract scene narrative.
The open-category path is primarily mono 32 kHz research inference, useful for retrieval, cleanup and discovery. Fixed 44.1 kHz stereo music stems remain a different production task.
CLAP space is not a physical separator
Highly correlated sources, weak targets and strong reverb do not have a unique mask merely because their semantics differ. Neighboring scene audio can leak into the result.
The language query is itself an experimental variable. Comparisons should keep the mixture and model fixed while auditing target retention, non-target leakage and mixture consistency after recombination.
Papers and public files for Hive source
Model names, numbers and limitations trace to the papers, repositories or model cards below.