Skip to content

Query timeout when going through HTTP Proxy since 1.4.4 #548

Open

Description

@eledhwen

Hi 👋

We're having issues with our duckdb-java setup after bumping the patch version from 1.4.2 to 1.4.4 (so technically the issue could be post-1.4.3).

Our setup is as follows:

  • Java app in a docker container deployed on K8S
  • cluster is air-gapped so we package the plugins (namely httpfs) in the docker image and deactivate remote plugin download
  • queries are done against an S3-like server, through an HTTP proxy

Behaviour before 1.4.4 was fine, basic queries without filtering took a few seconds to run.

After 1.4.4 we observe the following:

  • it works seemingly as well as before when querying the S3 server directly (same data, outside K8S)
  • it times out when querying the S3 server through the HTTP proxy

The error is:

Caused by: java.sql.SQLException: IO Error: Connection timed out error for HTTP GET to '/my-bucket/?encoding-type=url&list-type=2&prefix=some-prefix

LINE 1: SELECT COUNT(1) FROM read_parquet("s3://my-bucket/some-prefix...

Has something changed on that front?

Perhaps it's a question more fit for https://github.com/duckdb/duckdb, but since it's a duckdb-java setup I figured it's better to start here.

Activity

  1. staticlibs commented on Jan 29, 2026

    @staticlibs
    Member

    Hi, thanks for the report! Yes, there were a number of HTTP fixes applied in 1.4.4, I will raise this internally.

    If you can provide more details on proxy setup - this may be helpful!

  2. carlopi commented on Jan 29, 2026

    @carlopi
    Member

    I can have a look at what might have been the difference, but to gather more informations (and potentially solve the issue), could you retry with both explcitly setting:

    SET httpfs_client_implementation = curl;
    --- do queries
    

    or

    SET httpfs_client_implementation = httplib;
    --- do queries
    

    They should take different codepaths, and might result in different behaviour, that might be interesting to understand what's going on.

    There is also a:

    SET http_timeout =  100; --- seconds
    

    option, that might also help, you can find out what's the current setting (of both settings) via:

    SELECT current_setting('http_timeout');
    
  3. staticlibs commented on Jan 29, 2026

    @staticlibs
    Member

    @eledhwen

    I cannot reproduce this on 1.4.4.0 with a default Squid proxy server setup and with the following query to S3:

    try (Connection conn = DriverManager.getConnection("jdbc:duckdb:");
         Statement stmt = conn.createStatement()) {
        stmt.execute("SET http_proxy = '127.0.0.1:3128'");
        try (ResultSet rs = stmt.executeQuery("SELECT * FROM 's3://alexkasko-test-bucket-1/iris.parquet'")) {
            while (rs.next()) {
                System.out.println(rs.getString(5));
            }
        }
    }

    Any more details about the problematic env may be helpful.

  4. eledhwen commented on Jan 29, 2026

    @eledhwen
    Author

    I had already tried setting an http_timeout to 60, without seeing a difference ; I suspect this has to do with proxy settings (which I don't control) that cap the transaction time anyway.

    Here is what I use when I initialize the duckdb session:

    SET autoinstall_known_extensions=false;
    SET autoload_known_extensions=false;
    SET custom_extension_repository='file:///dev/null';
    SET extension_directory='/path/to/extensions';
    INSTALL httpfs; LOAD httpfs;
    CREATE OR REPLACE SECRET s3_config (
      TYPE s3,
      PROVIDER config,
      URL_STYLE 'path',
      KEY_ID 'my_key',
      SECRET 'my_secret',
      REGION 'my_region',
      ENDPOINT 'my_endpoint',
      USE_SSL false
    );
    CREATE OR REPLACE SECRET http_proxy (
      TYPE http,
      PROVIDER config,
      HTTP_PROXY 'my_host:my_port'
    );
    

    I won't have the time to test the suggested httpfs_client_implementation option today, but that sounds promising indeed.

  5. eledhwen commented on Apr 7, 2026

    @eledhwen
    Author

    For what it's worth, I just tested again with both curl and httplib on 1.5.1, behaviour seems identical :

    • still timing out, even after raising the timeout to 120sec
    • still working from outside the http proxy.
    • whole setup around it is constant (same parquet files on the same S3)
  6. eledhwen commented on Apr 14, 2026

    @eledhwen
    Author

    Any pointers @staticlibs @carlopi ?

    I can confidently say that I get the issue with only the DuckDB java version being changed, but unfortunately I can't share more info on how that specific proxy is setup.

    I tried the httpfs_client_implementation already, is there any other config I should look into that may be impacted by further duckdb releases?

  7. staticlibs commented on Apr 14, 2026

    @staticlibs
    Member

    @eledhwen

    In general we need some kind of a reproducer to investigate this deeper. I would suggest to look at SSL certificates usage first in your env, because this part is notoriously error prone with proxies. Also there is another issue reported recently that sounds similar to this one - duckdb/duckdb#22067

  8. eledhwen commented on Sep 7, 2026

    @eledhwen
    Author

    FWIW: it's very likely something to do with SSL and the http client implementation change.

    What we've done on our end to circumvent the issue is:

    • changing to httpfs_client_implementation=curl
    • adding a global http_proxy=${our-proxy-host-and-port} in addition to our existing http secret creation logic <- that was the part that eluded me a few months ago, the secret was apparently ignored/misused when using curl

    I'll leave the issue open as technically we're still circumventing an issue and haven't identified the proper way to do it as compared to the older 1.4.x version, but we have a working version for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions