Skip to content

Bigtable: Snippet setup fails due to rate-based quota. #8454

Closed

Description

@tseaver

From this failiure today:

google/cloud/bigtable/cluster.py:340: in delete
    client.instance_admin_client.delete_cluster(self.name)
google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client.py:1179: in delete_cluster
    request, retry=retry, timeout=timeout, metadata=metadata
../api_core/google/api_core/gapic_v1/method.py:143: in __call__
    return wrapped_func(*args, **kwargs)
../api_core/google/api_core/retry.py:273: in retry_wrapped_func
    on_error=on_error,
../api_core/google/api_core/retry.py:182: in retry_target
    return target()
../api_core/google/api_core/timeout.py:214: in func_with_timeout
    return func(*args, **kwargs)
../api_core/google/api_core/grpc_helpers.py:59: in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
value = ResourceExhausted(u"Quota exceeded for quota group 'InstancesWriteGroup' and l... 'bigtableadmin.googleapis.com' for consumer 'project_number:1065521786570'.",)
from_value = <_Rendezvous of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUS...ogleapis.com' for consumer 'project_number:1065521786570'.","grpc_status":8}"
>
    def raise_from(value, from_value):
>       raise value
E       ResourceExhausted: 429 Quota exceeded for quota group 'InstancesWriteGroup' and limit 'CLIENT_PROJECT-1d' of service 'bigtableadmin.googleapis.com' for consumer 'project_number:1065521786570'.
.nox/snippets-2-7/lib/python2.7/site-packages/six.py:737: ResourceExhausted

Including many more failures. @sduskis, could we get this particular quota / rate limit eased for our CI project?

Activity

  1. added
    api: bigtableIssues related to the Bigtable API.
    type: processA process-related concern. May include testing, release, or the like.
    on Jun 20, 2019
  2. sduskis commented on Jun 25, 2019

    @sduskis
    Contributor

    InstancesWriteGroup relates to the quota for admin writes. The service can't do more than some number of updates per time period. InstancesWriteGroup/CLIENT_PROJECT-1d is 500 per day. Basically, the tests run too often, and Cloud Bigtable isn't happy about it.

  3. mf2199 commented on Aug 2, 2019

    @mf2199
    Contributor

    @tseaver @igorbernstein2 Any prospects of where do we intend to go with this? If the limits stay, maybe this can be reflected in Best Practices or something. Any thoughts?

  4. tseaver commented on Aug 2, 2019

    @tseaver
    ContributorAuthor

    @mf2199 Running the actual system tests less often seems to be the only cure. #8476 seems to have helped with this in the mono-repo: we see far fewer Bigtable runs which don't involve actual changes to Bigtable source in the PR. Splitting the repository should be the final remdeiation.

    I'm going to close this for now, given that #8476 seems to have helped (I don't think we've seen this failure since).

  5. mf2199 commented on Aug 2, 2019

    @mf2199
    Contributor

    @tseaver 👍 Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

api: bigtableIssues related to the Bigtable API.flakytestingtype: processA process-related concern. May include testing, release, or the like.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions