Skip to content

allow overriding HttpResponse.getContent to install download throttler #586

Open

Description

@srosenberg

We need to throttle downloads from cloud storage in order to minimize GC disturbances during (data) reload. I was unable to find a clean way to do this via existing API. Instead, the following workaround was implemented which suffers from extending HttpURLConnection. Is it possible to provide a proper hook for intercepting InputStream?

protected Storage installHttpThrottler(StorageOptions.Builder storageOptions) {
        return storageOptions
                .setTransportOptions(HttpTransportOptions.newBuilder().setHttpTransportFactory(new ThrottledHttpTransportFactory()).build())
                .build()
                .getService();
    }
class ThrottledHttpTransportFactory implements HttpTransportFactory {
        public HttpTransport create() { return new NetHttpTransport.Builder().setConnectionFactory(new ThrottledConnectionFactory()).build(); }
    }
class ThrottledConnectionFactory extends DefaultConnectionFactory {
        @Override
        public HttpURLConnection openConnection(URL url) throws IOException {
            return new ThrottledUrlConnection(super.openConnection(url), url);
        }
    }
class ThrottledUrlConnection extends HttpURLConnection {
        HttpURLConnection delegate;

        ThrottledUrlConnection(HttpURLConnection delegate, URL url) {
            super(url);
            this.delegate = delegate;
        }

        @Override
        public InputStream getInputStream() throws IOException {
            return new ThrottledInputStream(delegate.getInputStream(), rateLimiter, throughputMeter);
        }
// followed by a long list of overridden methods to redirect via 'delegate'
....
}

Activity

  1. srosenberg commented on Feb 10, 2019

    @srosenberg
    Author

    Per @ajaaym suggestion in googleapis/google-cloud-java#3534, we should be able to use the interceptor (https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpResponseInterceptor.java); problem is getContent needs to be overridden in order to install the throttler.

  2. added
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    and removed
    triage meI really want to be triaged.
    on Feb 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions