Multiprocessing task failed: Multiprocessing task did not complete successfully

[nh13]

Hello,

a user is getting an error using a script I wrote (see https://github.com/nh13/basespace-invaders), with the following stack trace using python 2.7.6 (is it absolutely crucial to use 2.6?). Any ideas on how to debug this error within the basespace python API?

python samples2files.py --client-key xxxxxx --client-secret xxxx --access-token xxxxx --sample-id 12345678 --create-basespace-directory-structure  
Will download 4 files.
Downloading (1/4): P12-HFhm-F_S12_L001_R1_001.fastq.gz
File Path: P12-HFhm-F_S12_L001_R1_001.fastq.gz
WARNING:BaseSpacePy.model.MultipartFileTransfer:Task failed after too many retries
Traceback (most recent call last):
  File "samples2files.py", line 165, in <module>
    outputDirectory=options.outputDirectory, createBsDir=options.createBsDir)
  File "samples2files.py", line 107, in download
    sampleFile.downloadFile(myAPI, outputDirectory, createBsDir=createBsDir)
  File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/File.py", line 82, in downloadFile
  File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/api/BaseSpaceAPI.py", line 1240, in fileDownload
  File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/api/BaseSpaceAPI.py", line 1323, in multipartFileDownload
  File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/MultipartFileTransfer.py", line 399, in download
  File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/MultipartFileTransfer.py", line 452, in _start_workers
  File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/MultipartFileTransfer.py", line 281, in start_workers
BaseSpacePy.api.BaseSpaceException.MultiProcessingTaskFailedException: 'Multiprocessing task failed: Multiprocessing task did not complete successfully’

[psaffrey-illumina]

It shouldn’t be crucial to use Python2.6.

One thing your user can try is to increase the logging level, which should provide better information on why the task is failing. You can do this by adjusting your call to logging.basicConfig() on line 30. Make it look like this:

logging.basicConfig(level="DEBUG")

Then you’ll get output like this:

Will download 2 files.
Downloading (1/2): B2-LIB14_S2_L004_R1_001.fastq.gz
File Path: B2-LIB14_S2_L004_R1_001.fastq.gz
INFO:BaseSpacePy.model.MultipartFileTransfer:Total File Size 1.31 GB
INFO:BaseSpacePy.model.MultipartFileTransfer:Using File Part Size 25 MB
INFO:BaseSpacePy.model.MultipartFileTransfer:Processes 10
INFO:BaseSpacePy.model.MultipartFileTransfer:File Chunk Count 54
INFO:BaseSpacePy.model.MultipartFileTransfer:Start Chunk 1
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Worker Consumer-2 processing task: File piece 1 of 54, piece size 25.00 MB of total 1.31 GB
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Worker Consumer-3 processing task: File piece 3 of 54, piece size 25.00 MB of total 1.31 GB
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Worker Consumer-5 processing task: File piece 4 of 54, piece size 25.00 MB of total 1.31 GB
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Worker Consumer-4 processing task: File piece 5 of 54, piece size 25.00 MB of total 1.31 GB
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Worker Consumer-8 processing task: File piece 8 of 54, piece size 25.00 MB of total 1.31 GB
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Worker Consumer-1 processing task: File piece 2 of 54, piece size 25.00 MB of total 1.31 GB
DEBUG:BaseSpacePy.model.MultipartFileTransfer:Workers started
…

Hopefully you’ll also get a better error as to why the individual job failed.

If you want to make this a bit cleaner, you could add a verbose mode to your script which will switch the logging level.

Peter

From: Nils Homer [mailto:notifications@github.com]
Sent: 06 May 2015 16:50
To: basespace/basespace-python-sdk
Subject: [basespace-python-sdk] Multiprocessing task failed: Multiprocessing task did not complete successfully (#13)

Hello,

a user is getting an error using a script I wrote (see https://github.com/nh13/basespace-invaders), with the following stack trace using python 2.7.6 (is it absolutely crucial to use 2.6?). Any ideas on how to debug this error within the basespace python API?

python samples2files.py --client-key xxxxxx --client-secret xxxx --access-token xxxxx --sample-id 12345678 --create-basespace-directory-structure

Will download 4 files.

Downloading (1/4): P12-HFhm-F_S12_L001_R1_001.fastq.gz

File Path: P12-HFhm-F_S12_L001_R1_001.fastq.gz

WARNING:BaseSpacePy.model.MultipartFileTransfer:Task failed after too many retries

Traceback (most recent call last):

File "samples2files.py", line 165, in

outputDirectory=options.outputDirectory, createBsDir=options.createBsDir)

File "samples2files.py", line 107, in download

sampleFile.downloadFile(myAPI, outputDirectory, createBsDir=createBsDir)

File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/File.py", line 82, in downloadFile

File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/api/BaseSpaceAPI.py", line 1240, in fileDownload

File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/api/BaseSpaceAPI.py", line 1323, in multipartFileDownload

File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/MultipartFileTransfer.py", line 399, in download

File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/MultipartFileTransfer.py", line 452, in _start_workers

File "/usr/local/lib/python2.7/dist-packages/BaseSpacePy/model/MultipartFileTransfer.py", line 281, in start_workers

BaseSpacePy.api.BaseSpaceException.MultiProcessingTaskFailedException: 'Multiprocessing task failed: Multiprocessing task did not complete successfully’

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/13.

[tgjohnst]

Hi @nh13 - I just finished developing a similar script for downloading basespace data (sadly didn't realize yours existed til after the fact) and am also encountering the same issue with the parallel download. I am using a version of the SDK ported to python 3, but the error is exactly the same. It looks like the workers get spun up, start trying to download, and I get a file double the filesize of what it should be, then nothing happens and it errors out after a while.
Did you ever figure out a solution or the source of the error?

[nh13]

@tgjohnst I just wrapped in retries and it usually succeeds after that. nh13/basespace-invaders#7

[tgjohnst]

Thanks for the quick response! Hmm, I'll give that a try, but this has failed a few runs in a row, so not sure if retries will help.

I activated debug-level logging and my debug output is looking a little weird. Specifically, there is a 166MB file, but the downloader is aiming for 8x 25MB chunks when I think there should only be 7 for a file that size, and the 8th chunk is the one failing.

[28/07/2021 00:00:25] Download NEBNextindex06_S1_L001_R2_001.fastq.gz to test_output/testprojname
[28/07/2021 00:00:25] Total File Size 166.34 MB
[28/07/2021 00:00:25] Using File Part Size 25 MB
[28/07/2021 00:00:25] Processes 10
[28/07/2021 00:00:25] File Chunk Count 8
[28/07/2021 00:00:25] Start Chunk 1
[28/07/2021 00:00:25] Worker Consumer-1 processing task: File piece 1 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-5 processing task: File piece 2 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-3 processing task: File piece 3 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Workers started
[28/07/2021 00:00:26] Worker Consumer-6 processing task: File piece 4 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-2 processing task: File piece 5 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-7 processing task: File piece 6 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-4 processing task: File piece 7 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-8 processing task: File piece 8 of 8, piece size 25.00 MB of total 166.34 MB
[28/07/2021 00:00:26] Worker Consumer-9 exiting, found final task
[28/07/2021 00:00:26] Worker Consumer-10 exiting, found final task
[28/07/2021 00:00:29] Worker Consumer-4 exiting, found final task
[28/07/2021 00:00:30] Worker Consumer-2 exiting, found final task
[28/07/2021 00:00:30] Worker Consumer-7 exiting, found final task
[28/07/2021 00:00:30] Worker Consumer-5 exiting, found final task
[28/07/2021 00:00:31] Worker Consumer-3 exiting, found final task
[28/07/2021 00:00:33] Worker Consumer-6 exiting, found final task
[28/07/2021 00:00:33] Worker Consumer-1 exiting, found final task
[28/07/2021 00:00:51] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 1, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:01:15] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 2, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:01:50] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 3, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:02:16] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 4, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:02:59] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 5, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:03:18] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 6, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:03:32] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 7, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:03:59] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 8, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:04:24] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 9, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:04:46] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 10, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:05:17] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 11, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:05:45] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 12, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:06:02] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 13, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:06:23] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 14, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:06:42] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 15, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:07:08] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 16, with error msg: name 'DownloadFailedException' is not defined
[28/07/2021 00:07:23] Worker Consumer-8 retrying task File piece 8 of 8, piece size 25.00 MB of total 166.34 MB after failure, retry attempt 17, with error msg: name 'DownloadFailedException' is not defined

[nh13]

Sorry I can’t be more help, just relating what worked for me. Hopefully Illumina support can help.

[tgjohnst]

Thanks anyway!

I'm thinking for now, I'll manually avoid multipart and just use getFileUrl() to pull the entire file at once. Kind of a workaround but at least it doesn't fail for anything large enough to require multipart.

[amaynard]

Sorry, this is not supported. It does indeed look to me like the first stumbling block is an off-by-one error. Not sure why you'd need to add one on the ceiling:
https://github.com/basespace/basespace-python-sdk/blob/master/src/BaseSpacePy/model/MultipartFileTransfer.py#L413

We do have a fully supported CLI that does a great job of downloading you can easily call from your script if that helps.
https://developer.basespace.illumina.com/docs/content/documentation/cli/cli-overview