Skip to content

Vision: Google Cloud Storage Client project=None bootstrapping problem #4239

Closed
Closed
Vision: Google Cloud Storage Client project=None bootstrapping problem#4239
Assignees
tseaver
Labels
api: storageIssues related to the Cloud Storage API.authpriority: p2Moderately-important priority. Fix may not be included in next release.

Description

@safi-manar
safi-manar
opened on Oct 23, 2017

Hello,

In using the Google Cloud API's, I noticed a small discrepancy in the documentation's expected behavior, and the actual behavior.
The documentation and Google cloud examples code typically assumes that credentials should be created using a JSON key store saved on persistent storage. However, this isn't always an option, as with deployments using ephemeral storage. Luckily, a Credentials object can be created directly from a dictionary, which houses the same mapping that the JSON keystore would hold. This is the procedure that I use in my project (the specifics of which, is shown below).

The Storage.Client documentation suggests that the project parameter is optional, and thus may be None. We found, however, that when it is not specified, an error is thrown, as a result of a sort of bootstrapping problem in which the Google Cloud APIs check to see if the project is None, and if so, attempt to create it (which is impossible, without first having the keys).
To elaborate:

  1. When we make a call to Storage.Client(), we step into storage/google/cloud/storage/client.py.
  2. This class calls super(Client, self).__init__(project=project, credentials=credentials, _http=_http), which takes us to core/google/cloud/client.py .
  3. This calls _ClientProjectMixin.__init__(self, project=project), where we see the following call:
        project = self._determine_default(project)
  1. Following this call, we see:
    if project is None:
        _, project = google.auth.default()

So, there is an attempt to infer the project parameter based on the environment credentials. However, as discussed above, we are using a dictionary-created Credentials object as our credentials, so no such credentials exist in the environment. This is a problem because the documentation claims that project is optional, when, in fact, it is not if the project is not stored in the environment.

Note, however, that this seems to be a problem specific with Storage.Client(), and not other API clients. Trying this procedure (see the code below) for google.cloud.vision.ImageAnnotatorClient seems to work perfectly fine with just a credentials object, without the project argument.

  1. OS: Ubuntu 16.04

  2. Python Version: Python 2.7.10

  3. Google APIs:
    google-auth==1.1.1
    google-cloud-core==0.27.1
    google-cloud-storage==1.4.0
    google-cloud-vision==0.27.0
    google-gax==0.15.15
    google-resumable-media==0.2.3
    googleapis-common-protos==1.5.3
    oauth2client==4.1.2

  4. Stacktrace:

Traceback (most recent call last):
  File "storage_client.py", line 23, in <module>
    client = _get_client()
  File "storage_client.py", line 18, in _get_client
    client = storage.Client(credentials=creds)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/storage/client.py", line 59, in __init__
    _http=_http)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 211, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 165, in __init__
    project = self._determine_default(project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 178, in _determine_default
    return _determine_default_project(project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.
  1. Steps to reproduce:

  • So, create an OAuth2 credentials object from a python dictionary (let's call it ketfile_dict using google.oauth2.service_account.Credentials.from_service_account_info(keyfile_dict).

  • Create an instance of a storage client using google.cloud.storage.Client(credentials=creds)

  • Although the documentation states that the "project" argument is optional, the API will in fact throw an error when no project argument is presented.

  1. Code example

Assume that in the same directory, you create a file, constants.py, with the following fields defined from the appropriate fields of the service account key JSON.

constants.GCP_SCOPE
constants.GCP_TYPE
constants.GCP_CLIENT_EMAIL
constants.GCP_PRIVATE_KEY
constants.GCP_PRIVATE_KEY_ID
constants.GCP_CLIENT_ID
constants.GCP_TOKEN_URI
constants.GCP_PROJECT_ID

from google.cloud import storage
from google.oauth2 import service_account
import constants


# Return an Oauth2 credentials instance
def get_credentials(with_keyfile=False):
    keyfile_dict = _get_keyfile_dict()
    scope = [constants.GCP_SCOPE]
    creds = service_account.Credentials.from_service_account_info(keyfile_dict).with_scopes(scope)

    if not with_keyfile:
        return creds
    else:
        # Return creds and the keyfile.
        return creds, keyfile_dict


# Construct a dictionary of the required key:value pairs.
def _get_keyfile_dict():
    keyfile_dict = {}
    keyfile_dict['type'] = constants.GCP_TYPE
    keyfile_dict['client_email'] = constants.GCP_CLIENT_EMAIL
    keyfile_dict['private_key'] = constants.GCP_PRIVATE_KEY
    keyfile_dict['private_key_id'] = constants.GCP_PRIVATE_KEY_ID
    keyfile_dict['client_id'] = constants.GCP_CLIENT_ID
    keyfile_dict['token_uri'] = constants.GCP_TOKEN_URI
    keyfile_dict['project_id'] = constants.GCP_PROJECT_ID
    return keyfile_dict


def _get_client():
    creds, keyfile_dict = get_credentials(with_keyfile=True)
    project = keyfile_dict['project_id']
    #client = storage.Client(project=project, credentials=creds) # This WILL work because a project is passed in.
    #client = vision.ImageAnnotatorClient(credentials=creds) #Note that the Vision client will work without a project argument
    client = storage.Client(credentials=creds) # This will throw an error because project=None.
    return client


# Instantiates a client
client = _get_client()

Let me know if you have any other questions. :)

Activity

  1. tseaver
    added
    api: storageIssues related to the Cloud Storage API.
    auth
    on Oct 23, 2017

  2. tseaver commented on Oct 23, 2017

    @tseaver
    tseaver
    on Oct 23, 2017
    Contributor

    @jonparrott Is this just a documentation issue, or should the client be willing to infer the project from the passed credentials?

  3. theacodes commented on Oct 23, 2017

    @theacodes
    theacodes
    on Oct 23, 2017
    Contributor

    This is interesting, because storage effectively doesn't need a project to operate. Buckets are global resources.

  4. tseaver
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Nov 2, 2017

  5. tseaver commented on Nov 2, 2017

    @tseaver
    tseaver
    on Nov 2, 2017
    Contributor

    Places which touch client.project:

    • list_buckets is the only client method which uses the project: I guess we could add an optional project argument, falling back to the client's project. Likely we should raise an exception if neither is set: project is a required parameter for 'buckets.list'.
    • Bucket.create is the only bucket method which uses it: it is likewise required, so we could allow an optional project argument and fall back / fail in the same way. We would also need to plumb it through to Client.create_bucket.
    • BucketNotification.__init__ takes and an optional topic_project parameter, and falls back to the client's project. We would need to have it raise if neither was set.

    To implement this, we could have the default for the project argument be a marker, and then treat None specially (pass through a dummy to the super() version and then assign None.

  6. theacodes commented on Nov 2, 2017

    @theacodes
    theacodes
    on Nov 2, 2017
    Contributor

    Sounds reasonable, @tseaver.

  7. chemelnucfin
    changed the title [-]Google Cloud Storage Client project=None bootstrapping problem[/-] [+]Vision: Google Cloud Storage Client project=None bootstrapping problem[/+] on Nov 3, 2017
  8. tseaver
    self-assigned this
    on Nov 10, 2017
  9. tseaver
    closed this as completedin #4381on Nov 13, 2017
  10. tseaver
    added a commit that references this issue on Nov 13, 2017

    Make 'project' optional / overridable for storage client (#4381)

    edade4c
  11. chemelnucfin
    added 2 commits that reference this issue on Nov 15, 2017

    Make 'project' optional / overridable for storage client (googleapis#…

    9605da8

    Make 'project' optional / overridable for storage client (googleapis#…

    b3c09b5
  12. parthea
    added 2 commits that reference this issue on Oct 21, 2023

    testing(translate): parameterize the timeout [(#4247)](GoogleCloudPla…

    7af4c99

    feat!: migrate API to microgenerator (#31)

    00a2690
  13. parthea
    added a commit that references this issue on Mar 9, 2026

    Make 'project' optional / overridable for storage client (#4381)

    21d3e56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.authpriority: p2Moderately-important priority. Fix may not be included in next release.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions