blob.upload_from_filename blocks for upwards of an hour.

I am using the following code to upload files to GCS.

blob = bucket.blob(gcs_filepath)
logging.info('Uploading file "%s" to "%s".', filepath, gcs_filepath)
blob.upload_from_filename(filepath)
logging.info('Uploaded file "%s" to "%s".', filepath, gcs_filepath)

This works most of the time. However, it has been stuck uploading for the last hour and has not raised an exception yet. It seems like this method should have a timeout or something for situations like this. It would also be helpful if it had the option to log the progress of the method for further diagnosis.

Duplicate of #3442, which has more information.

Hmm, not a duplicate.

@tseaver

@tseaver @tweeter0830 It may be somewhat of a duplicate if the real answer is that httplib2 + threading == a bad time.

I am using this in a thread.

@tweeter0830

@tweeter0830 Or switch to using a separate Client instance per thread (they should be relatively cheap to construct).

@tweeter0830

@tweeter0830 I'm closing this out as that is the culprit. As @tseaver said, you'll want Client instances to be thread local (for now). We will soon be dumping httplib2 altogether (for requests).

I moved the client into a single thread and I am still having this issue.

@tweeter0830

@tweeter0830 Can you provide some example code so we can discuss the implementation?

I'm seeing similar issues. I'm creating new Client Objects each time I want to upload a file.

client = storage.Client()
bucket = client.get_bucket(config.STORAGE_BUCKET)
blob = bucket.blob(out_file)
blob.upload_from_filename(in_file)
blob.acl.all().grant_read()
blob.acl.save()
blob.content_disposition = 'attachment; filename="' + name + '.mp4"'
blob.patch()
public_url = blob.public_url

This is running in a Celery Task.

@GEverding

Thanks for the snippet @GEverding. I'm not a regular user of Celery so I'm not sure what the implications are here. Are you saying you put this in a Python module at global scope and then point Celery at that module?

@dhermes

@dhermes Yes the function is in the global scope. We run each task on a separate process so there should only be one thread calling this piece of code.

Sorry, I was mistaken. It appears to have fixed the issue.

same problem. 100mb upload ~3-4 minutes. using webbrouser gui ~less than 20 sec

storage_client = storage.Client()
bucket = storage_client.get_bucket(GOOGLE_BUCKET)
blob = bucket.blob(file_name)
blob.upload_from_filename(os.path.join(local_path, file_name))

@kain64

@kain64 It appears you have a different issue than the original reporter. It's likely that the webbrowser GUI is using multiple threads to upload the file in chunks simultaneously (but I'm not sure).

If you'd really like fast uploads, I recommend looking into gsutil (we would like to speed up our uploads, but gsutil is fast today).

chore(deps): update dependency numpy to v1.18.3 [(#3441)](GoogleCloud…

Navigation Menu

Repository navigation

blob.upload_from_filename blocks for upwards of an hour. #3441

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

blob.upload_from_filename blocks for upwards of an hour. #3441

Description

Activity

tseaver commented on May 19, 2017

tseaver commented on May 19, 2017

dhermes commented on May 19, 2017

tweeter0830 commented on May 19, 2017

tseaver commented on May 19, 2017

dhermes commented on May 19, 2017

tweeter0830 commented on May 22, 2017

dhermes commented on May 22, 2017

GEverding commented on May 23, 2017

dhermes commented on May 23, 2017

GEverding commented on May 23, 2017

tweeter0830 commented on May 24, 2017

kain64 commented on Jun 14, 2017

dhermes commented on Jun 14, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions