NeuralGEN

NeuralGEN consists of three major modules:

1. EMu, a joint neural-music embedding model;
2. Mapping-1, a music semantic modeling model
3. Mapping-2, mapping joint neural-music tokens and semantic tokens to audio codec.

Dependencies

Rely on functions from lucidrains' implementation of MusicLM and AudioLM

pip install musiclm-pytorch
pip install audiolm-pytorch

Pretrained Components

1. Hubert (w2v-bert): HuBERT Base (~95M params) Download and put it in ./hubert
2. Unofficial implementation of Soundstream:

import torch

model = torch.hub.load("kaiidams/soundstream-pytorch", "soundstream_16khz")

Demo

• Train Mulan and EMu: ./demo/Train_Mulan_EMu.ipynb
• Train semantic modeling: ./demo/Train Mapping.ipynb
• Inference: ./demo/NeuralGEN_Inference.ipynb

Output

An awkward output ./output/output.wav. (due to using soundstream pretrained on speech dataset.)