Model training on multiple GPUs failed

[Larrycpan]

Since I have a large dataset (millions of cells) and want to include as many as possible genes (e.g. n_var_genes= 15000 in c2f.utils.get_training_data), I want to perform mod.train using multiple GPUs. Given the pytorch_lightening version (1.5.10post0) in cell2fate environment only supporting DDPPlugin, the training process is adapted to

mod.train(accelerator = "gpu", devices=-1, strategy=DDPPlugin(find_unused_parameters=True), lr=1e-04, batch_size=1024)

However, I encountered the error

RuntimeError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.

Changing find_unused_parameters=True to find_unused_parameters=False did not resolve the problem. But when I train the model using one GPU, like

mod.train(accelerator = "gpu", devices=1, lr=1e-04, batch_size=1024)

everything goes smoothly. Does cell2fate support training on multiple GPUs? If yes, how can I adapt the tutorial code? Thanks for your help.

[AlexanderAivazidis]

Hi! We have never tried it on multiple GPUs, so I can't help much with that. But if you want to use multiple GPUs you could also just split your dataset in advance according to some biologically meaningful criteria, such as different regions of a tissue and then supply one partial dataset to each GPU.

[Larrycpan]

Hi! We have never tried it on multiple GPUs, so I can't help much with that. But if you want to use multiple GPUs you could also just split your dataset in advance according to some biologically meaningful criteria, such as different regions of a tissue and then supply one partial dataset to each GPU.

Can I split the data by cell types, and fed data of each cell type into one GPU?

[AlexanderAivazidis]

That is not idea because you want each cell2fate run to have some data with a trajectory in it. But maybe you already know based on cluster annotations that some clusters belong to lineage A and others to lineage B? Or some cells come from a completely different anatomical location of the tissue? Then you could split based on those criteria.