Skip to content

Performance tests in pub/sub show that the publisher is using 50-100x more CPU than subscriber #3149

New issue
New issue
Closed
Closed
Performance tests in pub/sub show that the publisher is using 50-100x more CPU than subscriber#3149
Assignees
pongad
Labels
api: pubsubIssues related to the Pub/Sub API.performancetype: questionRequest for information or clarification. Not an issue.

Description

@j256
j256
opened on Apr 13, 2018

I've been running performance tests with Pub/Sub trying to find the right combination of threads, memory, buffer sizes, etc.. I'm seeing that the Publisher taking ~500% CPU versus 5-8% for the Subscriber running on the same server, simultaneously handling the same message throughput. Does that surprise anyone? In looking at the running threads I see that a number of them have the following stack trace in the RSA/SSHA256 JWT signing code. I suspect that this is to a large degree the real limitation of the Publisher's performance.

Anyone else seeing this? Any recommendations on how to work around this? Is there something I can do to the transport layer turn off these or reduce their usage? Maybe GPRC configuration tweaks?

Thanks. This is using v0.38 on a Linux box running Java 7 outside of google's cloud. Would running on Google's computer engine instances?

BigInteger.oddModPow(BigInteger.java:2716) 
java.math.BigInteger.modPow(BigInteger.java:2459) 
sun.security.rsa.RSACore.crtCrypt(RSACore.java:183) 
sun.security.rsa.RSACore.rsa(RSACore.java:122) 
sun.security.rsa.RSASignature.engineSign(RSASignature.java:175) 
java.security.Signature$Delegate.engineSign(Signature.java:1207) 
java.security.Signature.sign(Signature.java:579) 
com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:147) 
com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637) 
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials.getJwtAccess(ServiceAccountJwtAccessCredentials.java:300) 
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials.getRequestMetadata(ServiceAccountJwtAccessCredentials.java:267) 
com.google.auth.Credentials.blockingGetToCallback(Credentials.java:103) 
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials.getRequestMetadata(ServiceAccountJwtAccessCredentials.java:251) 
io.grpc.auth.GoogleAuthLibraryCallCredentials.applyRequestMetadata(GoogleAuthLibraryCallCredentials.java:90) 
io.grpc.internal.CallCredentialsApplyingTransportFactory$CallCredentialsApplyingTransport.newStream(CallCredentialsApplyingTransportFactory.java:91) 
io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:242) 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:387) 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:679) 
io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32) 
com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:95) 
io.grpc.stub.ClientCalls.startCall(ClientCalls.java:293) 
io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:268) 
io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:177) 
com.google.pubsub.v1.PublisherGrpc$PublisherFutureStub.publish(PublisherGrpc.java:538) 
com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:333) 
com.google.cloud.pubsub.v1.Publisher.access$000(Publisher.java:90) 
com.google.cloud.pubsub.v1.Publisher$1.run(Publisher.java:255) 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
java.util.concurrent.FutureTask.run(FutureTask.java:266) 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
java.lang.Thread.run(Thread.java:745)

Activity

  1. j256
    changed the title [-]Performance tests in pub/sub show that the publisher is registering 5-10x more CPU than subscriber[/-] [+]Performance tests in pub/sub show that the publisher is registering 50-100x more CPU than subscriber[/+] on Apr 13, 2018
  2. j256
    changed the title [-]Performance tests in pub/sub show that the publisher is registering 50-100x more CPU than subscriber[/-] [+]Performance tests in pub/sub show that the publisher is using 50-100x more CPU than subscriber[/+] on Apr 13, 2018
  3. pongad
    self-assigned this
    on Apr 18, 2018
  4. pongad
    added
    type: questionRequest for information or clarification. Not an issue.
    api: pubsubIssues related to the Pub/Sub API.
    performance
    on Apr 18, 2018

  5. pongad commented on Apr 18, 2018

    @pongad
    pongad
    on Apr 18, 2018
    Contributor

    Yes, this is extremely surprising. When we load-tested pubsub client, we usually observe that subscribers take a little more CPU than publisher. Our load tests run in GCE though. Could you share how you're authenticating the client?

  6. j256 commented on Apr 24, 2018

    @j256
    j256
    on Apr 24, 2018
    Author

    I'm not sure what you mean. I'm using something like:

    credentialsProvider =
	FixedCredentialsProvider.create(GoogleCredentials.fromStream(jsonFileInputStream))
...
Publisher.Builder publisherBuilder = Publisher.newBuilder(googleTopicName);
publisherBuilder.setCredentialsProvider(credentialsProvider);
publisherBuilder.setRetrySettings(buildRetrySettings());
publisherBuilder.setBatchingSettings(buildBatchingSettings());
// thread settings
publisherBuilder.setExecutorProvider(InstantiatingExecutorProvider.newBuilder()
		.setExecutorThreadCount(configuration.getPublisherThreadCount()).build());
publisher = publisherBuilder.build();

    What else do you need me to show? Are there authentication settings that I'm missing or otherwise doing incorrectly @pongad ? Thanks.

  7. hmigneron commented on Apr 26, 2018

    @hmigneron
    hmigneron
    on Apr 26, 2018

    Just found this ticket after submitting my own. I am seeing similar results inside google's cloud (GKE) as I pointed out in #3194

    My authentication method is similar to yours. Our base docker image is centos 7 if that makes a difference. We saw similar results with 0.39.0-beta. We also have dependencies to

    "io.grpc:grpc-netty:1.10.0"
"io.grpc:grpc-protobuf:1.10.0"
"io.grpc:grpc-stub:1.10.0"

    in our project. I mention this because the fact that we use gRPC/netty has had an impact before on separate issues.

  8. pongad commented on Apr 26, 2018

    @pongad
    pongad
    on Apr 26, 2018
    Contributor

    I think I found the issue. From the stack trace, you seem to be using JWT for auth. Until recently JWT tokens aren't cached so you were doing expensive auth negotiation every single call to publish RPC. The subscribe side wasn't affected because it uses long-running streaming call.

    This was fixed in googleapis/google-auth-library-java@664754e on the auth side. We'll upgrade our library in this repo.

  9. pongad
    closed this as completedin #3197on Apr 26, 2018
  10. pongad
    added a commit that references this issue on Apr 26, 2018

    bump auth library version (#3197)

    87b19d6

  11. j256 commented on Apr 26, 2018

    @j256
    j256
    on Apr 26, 2018
    Author

    Wow. Huge win with 0.38.0 for example. I still see the Publisher use more CPU than the Subscriber but the difference now is like ~3x instead of 50-100x. Throughput seems to have increased so that my program is now hitting other limits. Thanks much @pongad.

    For those who don't want to wait, it's an easy pom.xml exclusion stanza. Before you ask, the only difference that I can see between 0.9.0 and 0.9.1 is the addition of the JWT token caching.

    <dependency>	
	<groupId>com.google.cloud</groupId>	
	<artifactId>google-cloud-pubsub</artifactId>	
	<version>0.##.0-beta</version>	
	<exclusions>	
		<exclusion>	
			<groupId>com.google.auth</groupId>	
			<artifactId>google-auth-library-credentials</artifactId>	
		</exclusion>	
		<exclusion>	
			<groupId>com.google.auth</groupId>	
			<artifactId>google-auth-library-oauth2-http</artifactId>	
		</exclusion>	
	</exclusions>
</dependency>	
<dependency>	
	<groupId>com.google.auth</groupId>	
	<artifactId>google-auth-library-credentials</artifactId>	
	<version>0.9.1</version>	
</dependency>	
<dependency>	
	<groupId>com.google.auth</groupId>	
	<artifactId>google-auth-library-oauth2-http</artifactId>	
	<version>0.9.1</version>	
</dependency>

  12. hmigneron commented on Apr 27, 2018

    @hmigneron
    hmigneron
    on Apr 27, 2018

    Thank you both, the improvement is HUGE for us as well with this solution.

  13. theacodes
    unassigned
    pongad
    on Sep 28, 2018
  14. JustinBeckwith
    assigned
    pongad
    on Feb 1, 2021
  15. chingor13
    added a commit that references this issue on Feb 20, 2026

    chore(main): release 3.19.1-SNAPSHOT (#3149)

    968df89
  16. suztomo
    added a commit that references this issue on Mar 9, 2026

    chore(deps): update dependency com.google.cloud:google-cloud-bigquery…

    c57748d
  17. lqiu96
    added a commit that references this issue on Mar 20, 2026

    chore(main): release 3.19.1-SNAPSHOT (#3149)

    f020564
  18. chingor13
    added 2 commits that reference this issue on Mar 24, 2026

    build(deps): update dependency org.apache.maven.plugins:maven-project…

    3a0b114

    chore: add property for keep-transaction-alive (#3149)

    0f7bfc2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: pubsubIssues related to the Pub/Sub API.performancetype: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions