Skip to content

LoggingHandler is re-entrant when logging FINE #1386

Closed

Description

@gregw

When using the LoggingHandler with logging turned up to FINE, it becomes re-entrant causing either an error or a stack overflow.

This appears due to the fact that classes used to implement LoggingHandler are themselves instrumented with Java Util Logging! Examples I have found are io.netty (first stack trace below)
and HttpURLConnection (second stack trace using a cut/paste of LoggingHandler to com.google.cloud.runtimes.jetty9.LoggingHandler as I'm working on Issue #1329

Nov 10, 2016 5:57:44 PM com.google.cloud.runtimes.jetty9.StackDriverLogging init
INFO: test info
Nov 10, 2016 5:57:45 PM io.netty.util.internal.logging.InternalLoggerFactory newDefaultFactory
FINE: Using java.util.logging as the default logging framework
Exception in thread "main" java.lang.ExceptionInInitializerError
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:36)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:58)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
	at io.grpc.netty.Utils$DefaultEventLoopGroupResource.create(Utils.java:187)
	at io.grpc.netty.Utils$DefaultEventLoopGroupResource.create(Utils.java:171)
	at io.grpc.internal.SharedResourceHolder.getInternal(SharedResourceHolder.java:124)
	at io.grpc.internal.SharedResourceHolder.get(SharedResourceHolder.java:94)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:311)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:280)
	at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:230)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:239)
	at com.google.api.gax.grpc.ServiceApiSettings$Builder$3.getOrBuildChannel(ServiceApiSettings.java:349)
	at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.<init>(ConfigServiceV2Api.java:155)
	at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.create(ConfigServiceV2Api.java:145)
	at com.google.cloud.logging.spi.DefaultLoggingRpc.<init>(DefaultLoggingRpc.java:140)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:68)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:62)
	at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:491)
	at com.google.cloud.logging.LoggingImpl.<init>(LoggingImpl.java:96)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:43)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:38)
	at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:478)
	at com.google.cloud.logging.LoggingHandler.getLogging(LoggingHandler.java:276)
	at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
	at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
	at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
	at java.util.logging.Logger.log(Logger.java:738)
	at io.netty.util.internal.logging.JdkLogger.log(JdkLogger.java:606)
	at io.netty.util.internal.logging.JdkLogger.debug(JdkLogger.java:186)
	at io.netty.util.internal.logging.InternalLoggerFactory.newDefaultFactory(InternalLoggerFactory.java:62)
	at io.netty.util.internal.logging.InternalLoggerFactory.<clinit>(InternalLoggerFactory.java:37)
	at io.netty.channel.MultithreadEventLoopGroup.<clinit>(MultithreadEventLoopGroup.java:34)
	at io.grpc.netty.Utils$DefaultEventLoopGroupResource.create(Utils.java:187)
	at io.grpc.netty.Utils$DefaultEventLoopGroupResource.create(Utils.java:171)
	at io.grpc.internal.SharedResourceHolder.getInternal(SharedResourceHolder.java:124)
	at io.grpc.internal.SharedResourceHolder.get(SharedResourceHolder.java:94)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:311)
	at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:280)
	at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:230)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:239)
	at com.google.api.gax.grpc.ServiceApiSettings$Builder$3.getOrBuildChannel(ServiceApiSettings.java:349)
	at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.<init>(ConfigServiceV2Api.java:155)
	at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.create(ConfigServiceV2Api.java:145)
	at com.google.cloud.logging.spi.DefaultLoggingRpc.<init>(DefaultLoggingRpc.java:140)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:68)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:62)
	at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:491)
	at com.google.cloud.logging.LoggingImpl.<init>(LoggingImpl.java:96)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:43)
	at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:38)
	at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:478)
	at com.google.cloud.logging.LoggingHandler.getLogging(LoggingHandler.java:276)
	at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
	at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
	at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
	at java.util.logging.Logger.log(Logger.java:738)
	at java.util.logging.Logger.doLog(Logger.java:765)
	at java.util.logging.Logger.log(Logger.java:788)
	at java.util.logging.Logger.info(Logger.java:1489)
	at com.google.cloud.runtimes.jetty9.StackDriverLogging.init(StackDriverLogging.java:37)
	at com.google.cloud.runtimes.jetty9.StackDriverLogging.main(StackDriverLogging.java:47)
Caused by: java.lang.NullPointerException
	at io.netty.util.internal.logging.InternalLoggerFactory.getInstance(InternalLoggerFactory.java:97)
	at io.netty.util.internal.logging.InternalLoggerFactory.getInstance(InternalLoggerFactory.java:90)
	at io.netty.util.concurrent.DefaultPromise.<clinit>(DefaultPromise.java:35)
	... 65 more
Exception in thread "main" java.lang.StackOverflowError
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
	at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
	at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
	at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:434)
	at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
	at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
	at java.security.Signature$Delegate.chooseProvider(Signature.java:1094)
	at java.security.Signature$Delegate.engineInitSign(Signature.java:1176)
	at java.security.Signature.initSign(Signature.java:527)
	at com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:145)
	at com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:208)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
	at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
	at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
	at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
	at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
	at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
	at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
	at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
	at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
	at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
	at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
	at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
	at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
	at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
	at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
	at com.google.cloud.runtimes.jetty9.AsyncLoggingHandler.write(AsyncLoggingHandler.java:101)
	at com.google.cloud.runtimes.jetty9.LoggingHandler.flush(LoggingHandler.java:365)
	at com.google.cloud.runtimes.jetty9.LoggingHandler.publish(LoggingHandler.java:298)
	at java.util.logging.Logger.log(Logger.java:738)
	at java.util.logging.Logger.doLog(Logger.java:765)
	at java.util.logging.Logger.log(Logger.java:788)
	at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
	at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
	at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
	at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
	at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
	at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
	at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
	at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
	at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
	at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
	at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
	at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
	at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)

...

	at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
	at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
	at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
	at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
	at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
	at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
	at com.google.cloud.runtimes.jetty9.AsyncLoggingHandler.write(AsyncLoggingHandler.java:101)
	at com.google.cloud.runtimes.jetty9.LoggingHandler.flush(LoggingHandler.java:365)
	at com.google.cloud.runtimes.jetty9.LoggingHandler.publish(LoggingHandler.java:298)
	at java.util.logging.Logger.log(Logger.java:738)
	at java.util.logging.Logger.doLog(Logger.java:765)
	at java.util.logging.Logger.log(Logger.java:788)
	at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
	at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
	at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
	at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
	at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
	at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
	at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
	at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
	at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
	at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
	at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
	at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
	at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
	at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
	at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
	at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
	at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
	at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
	at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
	at com.google.cloud.runtimes.jetty9.AsyncLoggingHandler.write(AsyncLoggingHandler.java:101)
	at com.google.cloud.runtimes.jetty9.LoggingHandler.flush(LoggingHandler.java:365)
	at com.google.cloud.runtimes.jetty9.LoggingHandler.publish(LoggingHandler.java:298)
	at java.util.logging.Logger.log(Logger.java:738)
	at java.util.logging.Logger.doLog(Logger.java:765)
	at java.util.logging.Logger.log(Logger.java:788)
	at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
	at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
	at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
	at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
	at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
	at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)

Activity

  1. mziccard commented on Nov 10, 2016

    @mziccard
    Contributor

    How are you registering the logging handler?

    We are aware of this and documented this behavior in our javadoc. To overcome this issue you should register the handler either using logging.properties or LoggingHandler.addHandler(Logger, LoggingHandler).

    To add a LoggingHandler to an existing Logger and be sure to avoid infinite recursion when logging, use the LoggingHandler.addHandler(Logger, LoggingHandler) method. Alternatively you can add the handler via logging.properties. For example using the following line: com.example.mypackage.handlers=com.google.cloud.logging.LoggingHandler

  2. gregw commented on Nov 10, 2016

    @gregw
    Author

    @mziccard The first of those example did incorrectly use addHandler. When I modify to use the static addHandler, that fixes the NPE in initialization, but I still get a stack overflow when actually logging at FINE.

    I setup with a combination of logging.properties:

    .level = FINE
    
    handlers = java.util.logging.ConsoleHandler
    java.util.logging.ConsoleHandler.level = FINE
    

    and then code that creates a LoggingHandler with specific JSON auth file:

    public class StackDriverLogging {
    
      static void init() throws Exception {
    
        final LoggingOptions options;
        String credentials = System.getenv("GOOGLE_OAUTH2_JSON_CREDENTIALS");
    
        if (credentials == null) {
          options = LoggingOptions.getDefaultInstance();
        } else {
          AuthCredentials authCredentials =
              AuthCredentials.createForJson(new FileInputStream(credentials));
          options = LoggingOptions.newBuilder().setAuthCredentials(authCredentials).build();
        }
    
        LoggingHandler handler = new LoggingHandler("test.log", options);
        handler.setLevel(Level.FINE);
        LoggingHandler.addHandler(Logger.getLogger(""), handler);
    
        Logger log = Logger.getLogger(StackDriverLogging.class.getName());
        log.info("test info");
        log.fine("test fine");
    
        //handler.flush();
    
      }

    This code results in a stack overflow exception, with the following repeated part:

        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:121)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:121)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:121)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
    

    and the console log issue the repeated log:

    FINE: sun.net.www.MessageHeader@65f8f5ae8 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@384671168 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@790da4778 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@57bc27f58 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@4b8d604b8 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@68c9d1798 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@2dfaea868 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@33ecda928 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    FINE: sun.net.www.MessageHeader@6e2829c78 pairs: {POST /o/oauth2/token HTTP/1.1: null}{Accept-Encoding: gzip}{User-Agent: Google-HTTP-Java-Client/1.21.0 (gzip)}{Content-Type: application/x-www-form-urlencoded; charset=UTF-8}{Host: accounts.google.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 174}
    Nov 11, 2016 9:16:38 AM sun.net.www.protocol.http.HttpURLConnection writeRequests
    

    My second example above is initialized entirely in a logging.properties file, but it uses a cut/paste/extended version of LoggingHandler that does the JSON auth initialization in the constructor and has some extra code in entryFor to add a traceid. I don't think my changes are relevant to this problem, but I created the first example just to see if I could reproduce with the standard LoggingHandler

  3. mziccard commented on Nov 10, 2016

    @mziccard
    Contributor

    I am not able to reproduce. I tried the following code:

    public class AddLoggingHandler {
    
      public static void main(String... args) throws Exception {
    
        LoggingOptions options = LoggingOptions.getDefaultInstance();
    
        LoggingHandler handler = new LoggingHandler("test.log", options);
        handler.setLevel(Level.FINE);
        LoggingHandler.addHandler(Logger.getLogger(""), handler);
    
        Logger log = Logger.getLogger(AddLoggingHandler.class.getName());
        log.info("test info");
        log.fine("test fine");
      }
    }

    With the following logging.properties:

    .level = FINE
    
    handlers = java.util.logging.ConsoleHandler
    java.util.logging.ConsoleHandler.level = FINE
    

    I have the GOOGLE_APPLICATION_CREDENTIALS set so my credentials are being loaded from file as in your example. The code works fine, entries are written to Logging and I get the following output:

    Nov 10, 2016 11:46:52 PM com.google.cloud.examples.logging.snippets.AddLoggingHandler main
    INFO: test info
    Nov 10, 2016 11:46:54 PM com.google.cloud.examples.logging.snippets.AddLoggingHandler main
    FINE: test fine
    
  4. gregw commented on Nov 10, 2016

    @gregw
    Author

    @mziccard I switched to your exact version of the code and moved my credentials to GOOGLE_APPLICATION_CREDENTIALS. I still get the infinite recursion?!?!

    I also tried passing in null for the options, to allow the internals to get the default instance and I get the same result.

    I'm using the 0.5.1 release.

    could it be something about my credentials? I can send them to you privately (they are just for a google test project) if you think it worth trying them?

  5. gregw commented on Nov 10, 2016

    @gregw
    Author

    @mbrukman I've simplified my test even more. I now just setup logging in logging.properties:

    .level = FINE
    
    handlers = com.google.cloud.logging.LoggingHandler
    com.google.cloud.logging.LoggingHandler.level = FINE
    

    and my test code is just:

    package com.google.cloud.runtimes.jetty9;
    import java.util.logging.Logger;
    public class StackDriverLogging {
      static void init() throws Exception {
        Logger log = Logger.getLogger(StackDriverLogging.class.getName());
        log.info("test info");
        log.fine("test fine");
      }
      public static void main(String... args) throws Exception {
        init();
      }
    }

    which gets the following stack overflow (added in full just in case the top or the bottom has something meaningful to you I just edited to take out most of the repeats]):

    Exception in thread "main" java.lang.StackOverflowError
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
        at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
        at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
        at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:434)
        at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
        at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
        at java.security.Signature$Delegate.chooseProvider(Signature.java:1094)
        at java.security.Signature$Delegate.engineInitSign(Signature.java:1176)
        at java.security.Signature.initSign(Signature.java:527)
        at com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:145)
        at com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:208)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
    
    ...
    
    
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1292)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
    
  6. gregw commented on Nov 10, 2016

    @gregw
    Author

    I modified the code to printout the state of the loggers:

      static void init() throws Exception {
        Logger log = Logger.getLogger(StackDriverLogging.class.getName());
    
        for (Logger l : new Logger[]{ log,Logger.getLogger("sun.net.www.protocol.http.HttpURLConnection")}) {
          while (l!=null) {
            System.err.printf("Log '%s' upl=%b %s%n",l.getName(),l.getUseParentHandlers(),Arrays.asList(l.getHandlers()));
            l = l.getParent();
          }
        }
    
        log.info("test info");
        log.fine("test fine");
      }

    which produced:

    Log 'com.google.cloud.runtimes.jetty9.StackDriverLogging' upl=true []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@71c7db30]
    Log 'sun.net.www.protocol.http.HttpURLConnection' upl=true []
    Log 'sun.net.www.protocol.http' upl=false []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@71c7db30]
    Exception in thread "main" java.lang.StackOverflowError
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
        at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
        at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
        at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:434)
        at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
        at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
        at java.security.Signature$Delegate.chooseProvider(Signature.java:1094)
        at java.security.Signature$Delegate.engineInitSign(Signature.java:1176)
        at java.security.Signature.initSign(Signature.java:527)
        at com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:145)
        at com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:208)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:657)
    ...
    

    So the masking of the loggers looks to have worked as sun.net.www.protocol.http has use parent set to false??? I wonder if it is something to do with the LoggingProxyImpl???

  7. mziccard commented on Nov 11, 2016

    @mziccard
    Contributor

    @gregw great, this is what I was about to suggest you to do. You are right, it seems that the masking is working just fine. In fact, if I run an analogous example I get the same output (set aside class names) and no recursion:

    Log 'com.google.cloud.examples.logging.snippets.AddLoggingHandler' upl=true []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@48eff760]
    Log 'sun.net.www.protocol.http.HttpURLConnection' upl=true []
    Log 'sun.net.www.protocol.http' upl=false []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@48eff760]

    This is really confusing. How are you running the app? Play java? Which version?

  8. gregw commented on Nov 11, 2016

    @gregw
    Author

    I'm currently running from within eclipse IDE using java 1.8.0_40. There is definitely something fishy as if I modify the test to do:

        Logger.getLogger("sun.net.www.protocol.http.HttpURLConnection").info("test");
        log.info("test info");
        log.fine("test fine");
    

    The first log line is correctly masked and does not go infinite, but the second one does???

    Let me try running from command line....

  9. gregw commented on Nov 11, 2016

    @gregw
    Author

    Command line gives me same problem:

    gregw@Tile440: ~/src/gcloud/flex/jetty-runtime/jetty9-base/target/jetty-base (master-68)
    [2130] java -Djava.util.logging.config.file=/tmp/logging.properties -classpath lib/gae/grpc-google-common-protos-0.1.0.jar:lib/gae/google-api-client-appengine-1.21.0.jar:lib/gae/aopalliance-1.0.jar:lib/gae/protobuf-java-util-3.0.0.jar:lib/gae/google-http-client-1.21.0.jar:lib/gae/gson-2.3.1.jar:lib/gae/javax.inject-1.jar:lib/gae/jdo2-api-2.3-eb.jar:lib/gae/grpc-protobuf-1.0.1.jar:lib/gae/grpc-context-1.0.1.jar:lib/gae/google-cloud-logging-0.5.1.jar:lib/gae/grpc-core-1.0.1.jar:lib/gae/google-auth-library-oauth2-http-0.3.1.jar:lib/gae/grpc-auth-1.0.1.jar:lib/gae/netty-codec-http-4.1.3.Final.jar:lib/gae/google-oauth-client-servlet-1.21.0.jar:lib/gae/google-api-client-1.21.0.jar:lib/gae/grpc-netty-1.0.1.jar:lib/gae/google-http-client-jackson2-1.19.0.jar:lib/gae/gax-0.0.21.jar:lib/gae/httpcore-4.0.1.jar:lib/gae/netty-codec-http2-4.1.3.Final.jar:lib/gae/grpc-stub-1.0.1.jar:lib/gae/httpclient-4.0.1.jar:lib/gae/google-cloud-core-0.5.1.jar:lib/gae/google-api-client-servlet-1.21.0.jar:lib/gae/guice-4.0.jar:lib/gae/netty-buffer-4.1.3.Final.jar:lib/gae/commons-codec-1.3.jar:lib/gae/joda-time-2.9.2.jar:lib/gae/transaction-api-1.1.jar:lib/gae/jetty9-base-0.1.0-SNAPSHOT.jar:lib/gae/netty-common-4.1.3.Final.jar:lib/gae/google-oauth-client-1.21.0.jar:lib/gae/google-http-client-jdo-1.21.0.jar:lib/gae/google-http-client-jackson-1.21.0.jar:lib/gae/jackson-core-2.1.3.jar:lib/gae/grpc-google-iam-v1-0.1.0.jar:lib/gae/guava-19.0.jar:lib/gae/google-auth-library-credentials-0.3.1.jar:lib/gae/netty-resolver-4.1.3.Final.jar:lib/gae/jackson-core-asl-1.9.11.jar:lib/gae/google-oauth-client-appengine-1.21.0.jar:lib/gae/commons-logging-1.1.1.jar:lib/gae/json-20151123.jar:lib/gae/protobuf-java-3.0.0.jar:lib/gae/netty-tcnative-boringssl-static-1.1.33.Fork19.jar:lib/gae/jsr305-1.3.9.jar:lib/gae/google-http-client-appengine-1.21.0.jar:lib/gae/netty-handler-4.1.3.Final.jar:lib/gae/grpc-protobuf-lite-1.0.1.jar:lib/gae/auto-value-1.2.jar:lib/gae/grpc-google-logging-v2-0.1.1.jar:lib/gae/netty-codec-4.1.3.Final.jar:lib/gae/netty-transport-4.1.3.Final.jar:lib/slf4j/slf4j-api-1.7.21.jar:lib/slf4j/slf4j-jdk14-1.7.21.jar:lib/slf4j/jcl-over-slf4j-1.7.21.jar com.google.cloud.runtimes.jetty9.StackDriverLogging
    Log 'com.google.cloud.runtimes.jetty9.StackDriverLogging' upl=true []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@7f63425a]
    Log 'sun.net.www.protocol.http.HttpURLConnection' upl=true []
    Log 'sun.net.www.protocol.http' upl=false []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@7f63425a]
    httpurl log
    log info
    Exception in thread "main" java.lang.StackOverflowError
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
        at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
        at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
        at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:434)
        at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
        at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
        at java.security.Signature$Delegate.chooseProvider(Signature.java:1094)
        at java.security.Signature$Delegate.engineInitSign(Signature.java:1176)
        at java.security.Signature.initSign(Signature.java:527)
        at com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:145)
        at com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:208)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
    ...
    
  10. gregw commented on Nov 11, 2016

    @gregw
    Author

    I upgraded to java 1.8.0_77
    Modified test code to be:

    package com.google.cloud.runtimes.jetty9;
    
    import java.util.Arrays;
    import java.util.logging.Logger;
    
    public class StackDriverLogging {
    
      static void init() throws Exception {
        Logger log = Logger.getLogger(StackDriverLogging.class.getName());
    
        for (Logger l : new Logger[] {log,
            Logger.getLogger("sun.net.www.protocol.http.HttpURLConnection")}) {
          while (l != null) {
            System.err.printf("Log '%s' upl=%b %s%n", l.getName(), l.getUseParentHandlers(),
                Arrays.asList(l.getHandlers()));
            l = l.getParent();
          }
        }
        System.err.println("httpurl log");
        Logger.getLogger("sun.net.www.protocol.http.HttpURLConnection").info("test");
        System.err.println("some.random.log log");
        Logger.getLogger("some.random.log").info("test");
        System.err.println("com.google.cloud.runtimes.jetty9.StackDriverLogging log info");
        log.info("test info");
        System.err.println("com.google.cloud.runtimes.jetty9.StackDriverLogging log fine");
        log.fine("test fine");
        System.err.println("done");
      }
    
      public static void main(String... args) throws Exception {
        init();
      }
    }
    

    and if fails running from command line on the some.random.log:

    [2139] java -Djava.util.logging.config.file=/tmp/logging.properties -classpath $( echo $(find -name '*.jar') | sed 's/ /:/g' ) com.google.cloud.runtimes.jetty9.StackDriverLogging
    Log 'com.google.cloud.runtimes.jetty9.StackDriverLogging' upl=true []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@6e8dacdf]
    Log 'sun.net.www.protocol.http.HttpURLConnection' upl=true []
    Log 'sun.net.www.protocol.http' upl=false []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@6e8dacdf]
    httpurl log
    some.random.log log
    Exception in thread "main" java.lang.StackOverflowError
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:206)
        at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:187)
        at sun.security.jca.ProviderList.getProvider(ProviderList.java:233)
        at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:434)
        at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:376)
        at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:486)
        at java.security.Signature$Delegate.chooseProvider(Signature.java:1094)
        at java.security.Signature$Delegate.engineInitSign(Signature.java:1176)
        at java.security.Signature.initSign(Signature.java:527)
        at com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:145)
        at com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:208)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
    ...
    

    my complete classpath is:

    ./grpc-google-common-protos-0.1.0.jar
    ./google-api-client-appengine-1.21.0.jar
    ./aopalliance-1.0.jar
    ./protobuf-java-util-3.0.0.jar
    ./google-http-client-1.21.0.jar
    ./gson-2.3.1.jar
    ./javax.inject-1.jar
    ./jdo2-api-2.3-eb.jar
    ./grpc-protobuf-1.0.1.jar
    ./grpc-context-1.0.1.jar
    ./google-cloud-logging-0.5.1.jar
    ./grpc-core-1.0.1.jar
    ./google-auth-library-oauth2-http-0.3.1.jar
    ./grpc-auth-1.0.1.jar
    ./netty-codec-http-4.1.3.Final.jar
    ./google-oauth-client-servlet-1.21.0.jar
    ./google-api-client-1.21.0.jar
    ./grpc-netty-1.0.1.jar
    ./google-http-client-jackson2-1.19.0.jar
    ./gax-0.0.21.jar
    ./httpcore-4.0.1.jar
    ./netty-codec-http2-4.1.3.Final.jar
    ./grpc-stub-1.0.1.jar
    ./httpclient-4.0.1.jar
    ./google-cloud-core-0.5.1.jar
    ./google-api-client-servlet-1.21.0.jar
    ./guice-4.0.jar
    ./netty-buffer-4.1.3.Final.jar
    ./commons-codec-1.3.jar
    ./joda-time-2.9.2.jar
    ./transaction-api-1.1.jar
    ./jetty9-base-0.1.0-SNAPSHOT.jar
    ./netty-common-4.1.3.Final.jar
    ./google-oauth-client-1.21.0.jar
    ./google-http-client-jdo-1.21.0.jar
    ./google-http-client-jackson-1.21.0.jar
    ./jackson-core-2.1.3.jar
    ./grpc-google-iam-v1-0.1.0.jar
    ./guava-19.0.jar
    ./google-auth-library-credentials-0.3.1.jar
    ./netty-resolver-4.1.3.Final.jar
    ./jackson-core-asl-1.9.11.jar
    ./google-oauth-client-appengine-1.21.0.jar
    ./commons-logging-1.1.1.jar
    ./json-20151123.jar
    ./protobuf-java-3.0.0.jar
    ./netty-tcnative-boringssl-static-1.1.33.Fork19.jar
    ./jsr305-1.3.9.jar
    ./google-http-client-appengine-1.21.0.jar
    ./netty-handler-4.1.3.Final.jar
    ./grpc-protobuf-lite-1.0.1.jar
    ./auto-value-1.2.jar
    ./grpc-google-logging-v2-0.1.1.jar
    ./netty-codec-4.1.3.Final.jar
    ./netty-transport-4.1.3.Final.jar
    

    with my classes in ./jetty9-base-0.1.0-SNAPSHOT.jar

  11. meltsufin commented on Nov 16, 2016

    @meltsufin
    Contributor

    @gregw I'm unable to reproduce this either. Are you using OpenJDK? Can you create a minimal maven project that reproduces this problem and share it on your GitHub account?

  12. gregw commented on Nov 16, 2016

    @gregw
    Author

    I'm using openJDK build 1.8.0_77-b03.

    I'll build a mvn project to reproduce shortly.

  13. gregw commented on Nov 17, 2016

    @gregw
    Author

    @meltsufin I've created a reproduction in a small project. Because it includes credentials, I have not attached it here but posted it a private channel.

    You can build it with:

    cd stackdriver-loop
    mvn install
    

    And run it from the command line with:

    GOOGLE_APPLICATION_CREDENTIALS=src/main/resources/jetty9-work-6f1d053125d8.json java -Djava.util.logging.config.file=src/main/resources/logging.properties -classpath $( echo $(find target/lib -name '*.jar') | sed 's/ /:/g' ):target/classes com.google.cloud.runtimes.jetty9.StackDriverLogging
    

    The recursive logs appear in n the jetty9-work log console
    The output I get is:

    [2142] GOOGLE_APPLICATION_CREDENTIALS=src/main/resources/jetty9-work-6f1d053125d8.json java -Djava.util.logging.config.file=src/main/resources/logging.properties -classpath $( echo $(find target/lib -name '*.jar') | sed 's/ /:/g' ):target/classes com.google.cloud.runtimes.jetty9.StackDriverLogging
    Log 'com.google.cloud.runtimes.jetty9.StackDriverLogging' upl=true []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@49e4cb85, java.util.logging.ConsoleHandler@2133c8f8]
    Log 'sun.net.www.protocol.http.HttpURLConnection' upl=true []
    Log 'sun.net.www.protocol.http' upl=false []
    Log '' upl=true [com.google.cloud.logging.LoggingHandler@49e4cb85, java.util.logging.ConsoleHandler@2133c8f8]
    Exception in thread "main" java.lang.StackOverflowError
        at java.util.Properties.getProperty(Properties.java:969)
        at sun.net.NetProperties.get(NetProperties.java:90)
        at sun.net.spi.DefaultProxySelector$3.run(DefaultProxySelector.java:224)
        at sun.net.spi.DefaultProxySelector$3.run(DefaultProxySelector.java:208)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.spi.DefaultProxySelector.select(DefaultProxySelector.java:207)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1099)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.DirectCallable.futureCall(DirectCallable.java:58)
        at com.google.api.gax.grpc.ExceptionTransformingCallable.futureCall(ExceptionTransformingCallable.java:66)
        at com.google.api.gax.grpc.RetryingCallable$RetryingResultFuture.issueCall(RetryingCallable.java:222)
        at com.google.api.gax.grpc.RetryingCallable.futureCall(RetryingCallable.java:90)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:242)
        at com.google.api.gax.grpc.UnaryCallable.futureCall(UnaryCallable.java:253)
        at com.google.cloud.logging.spi.DefaultLoggingRpc.write(DefaultLoggingRpc.java:199)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:503)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:499)
        at com.google.cloud.logging.LoggingHandler.write(LoggingHandler.java:351)
        at com.google.cloud.logging.LoggingHandler.flush(LoggingHandler.java:357)
        at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:292)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:788)
        at java.util.logging.LoggingProxyImpl.log(LoggingProxyImpl.java:61)
        at sun.util.logging.LoggingSupport.log(LoggingSupport.java:120)
        at sun.util.logging.PlatformLogger$JavaLoggerProxy.doLog(PlatformLogger.java:610)
        at sun.util.logging.PlatformLogger.fine(PlatformLogger.java:364)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:658)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1293)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
        at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:225)
        at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:97)
        at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:74)
        at io.grpc.auth.ClientAuthInterceptor.getRequestMetadata(ClientAuthInterceptor.java:150)
        at io.grpc.auth.ClientAuthInterceptor.access$100(ClientAuthInterceptor.java:64)
        at io.grpc.auth.ClientAuthInterceptor$1.checkedStart(ClientAuthInterceptor.java:96)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:195)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:47)
        at com.google.api.gax.grpc.HeaderInterceptor$1.start(HeaderInterceptor.java:64)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:273)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:252)
    
  14. gregw commented on Nov 17, 2016

    @gregw
    Author

    Note also that I just deployed a LoggingHandler in the gcloud flex environment, with logging only at INFO and observed a similar loop on start up:

    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:738)
    [INFO] GCLOUD:  at java.util.logging.Logger.doLog(Logger.java:765)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:850)
    [INFO] GCLOUD:  at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:340)
    [INFO] GCLOUD:  at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:260)
    [INFO] GCLOUD:  at com.google.api.gax.grpc.ServiceApiSettings$Builder$3.getOrBuildChannel(ServiceApiSettings.java:349)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.<init>(ConfigServiceV2Api.java:155)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.create(ConfigServiceV2Api.java:145)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.DefaultLoggingRpc.<init>(DefaultLoggingRpc.java:140)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:68)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:62)
    [INFO] GCLOUD:  at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:491)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingImpl.<init>(LoggingImpl.java:96)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:43)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:38)
    [INFO] GCLOUD:  at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:478)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.getLogging(LoggingHandler.java:282)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.AsyncLoggingHandler.write(AsyncLoggingHandler.java:101)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.flush(LoggingHandler.java:365)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.publish(LoggingHandler.java:298)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:738)
    [INFO] GCLOUD:  at java.util.logging.Logger.doLog(Logger.java:765)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:850)
    [INFO] GCLOUD:  at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:340)
    [INFO] GCLOUD:  at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:260)
    [INFO] GCLOUD:  at com.google.api.gax.grpc.ServiceApiSettings$Builder$3.getOrBuildChannel(ServiceApiSettings.java:349)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.<init>(ConfigServiceV2Api.java:155)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.create(ConfigServiceV2Api.java:145)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.DefaultLoggingRpc.<init>(DefaultLoggingRpc.java:140)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:68)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:62)
    [INFO] GCLOUD:  at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:491)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingImpl.<init>(LoggingImpl.java:96)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:43)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:38)
    [INFO] GCLOUD:  at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:478)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.getLogging(LoggingHandler.java:282)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.AsyncLoggingHandler.write(AsyncLoggingHandler.java:101)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.flush(LoggingHandler.java:365)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.publish(LoggingHandler.java:298)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:738)
    [INFO] GCLOUD:  at java.util.logging.Logger.doLog(Logger.java:765)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:850)
    [INFO] GCLOUD:  at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:340)
    [INFO] GCLOUD:  at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:260)
    [INFO] GCLOUD:  at com.google.api.gax.grpc.ServiceApiSettings$Builder$3.getOrBuildChannel(ServiceApiSettings.java:349)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.<init>(ConfigServiceV2Api.java:155)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.v2.ConfigServiceV2Api.create(ConfigServiceV2Api.java:145)
    [INFO] GCLOUD:  at com.google.cloud.logging.spi.DefaultLoggingRpc.<init>(DefaultLoggingRpc.java:140)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:68)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingRpcFactory.create(LoggingOptions.java:62)
    [INFO] GCLOUD:  at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:491)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingImpl.<init>(LoggingImpl.java:96)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:43)
    [INFO] GCLOUD:  at com.google.cloud.logging.LoggingOptions$DefaultLoggingFactory.create(LoggingOptions.java:38)
    [INFO] GCLOUD:  at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:478)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.getLogging(LoggingHandler.java:282)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.AsyncLoggingHandler.write(AsyncLoggingHandler.java:101)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.flush(LoggingHandler.java:365)
    [INFO] GCLOUD:  at com.google.cloud.runtimes.jetty9.LoggingHandler.publish(LoggingHandler.java:298)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:738)
    [INFO] GCLOUD:  at java.util.logging.Logger.doLog(Logger.java:765)
    [INFO] GCLOUD:  at java.util.logging.Logger.log(Logger.java:850)
    [INFO] GCLOUD:  at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:340)
    

    Note this is using my copy/paste/modify of LoggingHandler #1329.

  15. 29 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

api: loggingIssues related to the Cloud Logging API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions