fix: Support pool_maxsize parameter for concurrent connections #89
Conversation
UnixHTTPConnectionPool was not passing maxsize to the parent
HTTPConnectionPool, causing it to always use the default of 1.
This resulted in "Connection pool is full, discarding connection"
warnings when making concurrent requests through a Unix socket.
Changes:
- UnixHTTPConnectionPool now accepts maxsize parameter and passes
it to parent class
- UnixAdapter now accepts pool_maxsize parameter (default 1 for
backwards compatibility) and passes it to UnixHTTPConnectionPool
This allows users to configure larger connection pools for
concurrent request scenarios:
adapter = UnixAdapter(pool_maxsize=10)
session.mount('http+unix://', adapter)
|
Looks good to me, and thanks for the great commit message and PR! |
webknjaz
left a comment
webknjaz
left a comment
There was a problem hiding this comment.
I see at least one more in _new_conn()
|
_new_conn() creates individual connections, not pools. maxsize is a pool-level setting passed to HTTPConnectionPool.init(). The parent _new_conn() |
|
Ah, right. Could you add a test for the change? |
Looking into This is out of the scope here but I'd still block merging on having at least some regression test coverage. |
| class UnixHTTPConnectionPool(urllib3.connectionpool.HTTPConnectionPool): | ||
|
|
||
| def __init__(self, socket_path, timeout=60): | ||
| def __init__(self, socket_path, timeout=60, maxsize=1, **kwargs): |
There was a problem hiding this comment.
The default value upstream is 10: docker/docker-py@be7d0f0#diff-25eefff144d019c2dcb7e072a6a380825e67c192831eef3d59ca5ea2ca65aa17.
@mupuf should we mimic that?
There was a problem hiding this comment.
Ideally, the original version should have mirrored the upstream default... but this ship has sailed now and we don't want to regress on someone's code just for purity.
Let's start an issue about things that we want to fix for the next major version? There we could address your comment above about redefining private method.
3 participants
Summary
UnixHTTPConnectionPoolwas not passingmaxsizeto the parentHTTPConnectionPool, causing it to always use the default of 1. This resulted in noisy "Connection pool is full, discarding connection" warnings when making concurrent requests through a Unix socket.Changes
UnixHTTPConnectionPoolnow acceptsmaxsizeparameter and passes it to parent classUnixAdapternow acceptspool_maxsizeparameter (default 1 for backwards compatibility) and passes it toUnixHTTPConnectionPoolUsage
This allows users to configure larger connection pools for concurrent request scenarios:
Backwards Compatibility
Default value of
pool_maxsize=1preserves existing behavior. Only users who explicitly set a larger pool size will see different behavior.