You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Query timeout when going through HTTP Proxy since 1.4.4 #548
We're having issues with our duckdb-java setup after bumping the patch version from 1.4.2 to 1.4.4 (so technically the issue could be post-1.4.3).
Our setup is as follows:
Java app in a docker container deployed on K8S
cluster is air-gapped so we package the plugins (namely httpfs) in the docker image and deactivate remote plugin download
queries are done against an S3-like server, through an HTTP proxy
Behaviour before 1.4.4 was fine, basic queries without filtering took a few seconds to run.
After 1.4.4 we observe the following:
it works seemingly as well as before when querying the S3 server directly (same data, outside K8S)
it times out when querying the S3 server through the HTTP proxy
The error is:
Caused by: java.sql.SQLException: IO Error: Connection timed out error for HTTP GET to '/my-bucket/?encoding-type=url&list-type=2&prefix=some-prefix
LINE 1: SELECT COUNT(1) FROM read_parquet("s3://my-bucket/some-prefix...
Has something changed on that front?
Perhaps it's a question more fit for https://github.com/duckdb/duckdb, but since it's a duckdb-java setup I figured it's better to start here.
I can have a look at what might have been the difference, but to gather more informations (and potentially solve the issue), could you retry with both explcitly setting:
SET httpfs_client_implementation = curl;
--- do queries
or
SET httpfs_client_implementation = httplib;
--- do queries
They should take different codepaths, and might result in different behaviour, that might be interesting to understand what's going on.
There is also a:
SET http_timeout = 100; --- seconds
option, that might also help, you can find out what's the current setting (of both settings) via:
I had already tried setting an http_timeout to 60, without seeing a difference ; I suspect this has to do with proxy settings (which I don't control) that cap the transaction time anyway.
Here is what I use when I initialize the duckdb session:
SET autoinstall_known_extensions=false;
SET autoload_known_extensions=false;
SET custom_extension_repository='file:///dev/null';
SET extension_directory='/path/to/extensions';
INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE SECRET s3_config (
TYPE s3,
PROVIDER config,
URL_STYLE 'path',
KEY_ID 'my_key',
SECRET 'my_secret',
REGION 'my_region',
ENDPOINT 'my_endpoint',
USE_SSL false
);
CREATE OR REPLACE SECRET http_proxy (
TYPE http,
PROVIDER config,
HTTP_PROXY 'my_host:my_port'
);
I won't have the time to test the suggested httpfs_client_implementation option today, but that sounds promising indeed.
I can confidently say that I get the issue with only the DuckDB java version being changed, but unfortunately I can't share more info on how that specific proxy is setup.
I tried the httpfs_client_implementation already, is there any other config I should look into that may be impacted by further duckdb releases?
In general we need some kind of a reproducer to investigate this deeper. I would suggest to look at SSL certificates usage first in your env, because this part is notoriously error prone with proxies. Also there is another issue reported recently that sounds similar to this one - duckdb/duckdb#22067
FWIW: it's very likely something to do with SSL and the http client implementation change.
What we've done on our end to circumvent the issue is:
changing to httpfs_client_implementation=curl
adding a global http_proxy=${our-proxy-host-and-port} in addition to our existing http secret creation logic <- that was the part that eluded me a few months ago, the secret was apparently ignored/misused when using curl
I'll leave the issue open as technically we're still circumventing an issue and haven't identified the proper way to do it as compared to the older 1.4.x version, but we have a working version for now.
Activity
staticlibs commentedon Jan 29, 2026
Hi, thanks for the report! Yes, there were a number of HTTP fixes applied in 1.4.4, I will raise this internally.
If you can provide more details on proxy setup - this may be helpful!
carlopi commentedon Jan 29, 2026
I can have a look at what might have been the difference, but to gather more informations (and potentially solve the issue), could you retry with both explcitly setting:
or
They should take different codepaths, and might result in different behaviour, that might be interesting to understand what's going on.
There is also a:
option, that might also help, you can find out what's the current setting (of both settings) via:
staticlibs commentedon Jan 29, 2026
@eledhwen
I cannot reproduce this on
1.4.4.0with a default Squid proxy server setup and with the following query to S3:Any more details about the problematic env may be helpful.
eledhwen commentedon Jan 29, 2026
I had already tried setting an
http_timeoutto60, without seeing a difference ; I suspect this has to do with proxy settings (which I don't control) that cap the transaction time anyway.Here is what I use when I initialize the duckdb session:
I won't have the time to test the suggested
httpfs_client_implementationoption today, but that sounds promising indeed.eledhwen commentedon Apr 7, 2026
For what it's worth, I just tested again with both
curlandhttplibon1.5.1, behaviour seems identical :eledhwen commentedon Apr 14, 2026
Any pointers @staticlibs @carlopi ?
I can confidently say that I get the issue with only the DuckDB java version being changed, but unfortunately I can't share more info on how that specific proxy is setup.
I tried the
httpfs_client_implementationalready, is there any other config I should look into that may be impacted by further duckdb releases?staticlibs commentedon Apr 14, 2026
@eledhwen
In general we need some kind of a reproducer to investigate this deeper. I would suggest to look at SSL certificates usage first in your env, because this part is notoriously error prone with proxies. Also there is another issue reported recently that sounds similar to this one - duckdb/duckdb#22067
eledhwen commentedon Sep 7, 2026
FWIW: it's very likely something to do with SSL and the http client implementation change.
What we've done on our end to circumvent the issue is:
httpfs_client_implementation=curlhttp_proxy=${our-proxy-host-and-port}in addition to our existinghttpsecret creation logic <- that was the part that eluded me a few months ago, the secret was apparently ignored/misused when usingcurlI'll leave the issue open as technically we're still circumventing an issue and haven't identified the proper way to do it as compared to the older
1.4.xversion, but we have a working version for now.