[Feature Request] Introduce url() function for HTTP/HTTPS requests from SQL

[EdwardArchive]

Background & Motivation

We propose adding a url() scalar function that enables executing HTTP/HTTPS requests directly from SQL queries. This function would allow users to:

• Send webhook notifications (e.g., Slack, Discord) triggered by query results
• Integrate with external REST APIs for data enrichment
• Enable real-time event-driven workflows from within SQL

Use Cases

1. Alerting: Send Slack notifications when anomaly detection queries find issues
2. Data Enrichment: Call external APIs to augment query results
3. Webhook Integration: Trigger external workflows based on data changes

---

Proposed Interface

-- Basic GET request
SELECT url('https://api.example.com/data');

-- With configuration (method, headers, body)
SELECT url(
    'https://hooks.slack.com/services/xxx',
    '{"method": "POST", "headers": {"Content-Type": "application/json"}, "body": {"text": "Hello"}}'
);

Return Value

• Returns VARCHAR containing the HTTP response body
• Returns JSON object with status, body, headers for detailed response handling

Function Signature Options

-- Option A: Single function with JSON config
url(url VARCHAR, config VARCHAR) -> VARCHAR

-- Option B: Separate functions per method
http_get(url VARCHAR, headers VARCHAR) -> VARCHAR
http_post(url VARCHAR, body VARCHAR, headers VARCHAR) -> VARCHAR

Discussion needed: Which interface style is preferred?

---

Security Considerations

This function introduces potential SSRF (Server-Side Request Forgery) risks. After investigating other systems:

Database	Approach
ClickHouse	remote_url_allow_hosts configuration
Snowflake	Network Rules + External Access Integration
Databricks	Network Policies with Allowed Domains (FQDN)
DuckDB	enable_external_access variable
PostgreSQL (pgsql-http)	Function-level permission control

Proposed Security Approach

Phase 1 (Initial Implementation):

• Admin-configurable variable to enforce HTTPS with certificate validation
• Rely on infrastructure-level controls (Security Groups, Network Firewalls) for host restrictions

Future Consideration:

• Connection/Integration object similar to Databricks (if needed for SaaS/managed deployments)

---

Open Questions

1. Response size limits?
2. Timeout configuration?
3. Rate limiting considerations?

---

References

• ClickHouse url() table function: docs
• PostgreSQL pgsql-http: GitHub
• Snowflake External Functions: docs

[alvin-celerdata]

@EdwardArchive

1. for the function name, how about using http_request?
2. For now, StarRocks support Named arguments, it is okay to have multiple parameters in one function and users can Named arguments to call the function.
3. If we support URL in the first place, it is hard to change it in the future without breaking compatibility. So I will suggest that we need to consider how we can avoid SSRF in the first place.

[EdwardArchive]

• for the function name, how about using http_request?
• For now, StarRocks support Named arguments, it is okay to have multiple parameters in one function and users can Named arguments to call the function.
• If we support URL in the first place, it is hard to change it in the future without breaking compatibility. So I will suggest that we need to consider how we can avoid SSRF in the first place.

@alvin-celerdata Thanks for the suggestion!

1. Function Name
   At first I considered both options, but yes, I think http_request is clearer.

2. Multiple Parameters
   When I planned to make it, I only saw multiple parameters used in table functions, so I thought it was a rare case.
   If not, then I believe supporting multiple parameters with named arguments is the best way to handle this.

3. SSRF Safeguards
   I plan to build SSRF-related safeguards into my approach. For example:

Setting	Type	Default	Description
url_host_allowlist	string	""	Comma-separated list of allowed hosts
url_host_allowlist_regexp	string	""	Comma-separated list of allowed host regex
url_block_private_networks	bool	true	Block private IPs/localhost automatically
url_block_cloud_metadata	bool	true	Block cloud metadata endpoints

However, I think managing cloud metadata endpoints may be difficult to maintain or update
(I only searched briefly, but apart from this link (https://gist.github.com/jhaddix/78cece26c91c6263653f31ba453e273b), it seems difficult to find other resources. This one is not official information, so it doesn’t seem very reliable).
OR combine data from this source

• AWS IMDS Docs
• Azure IMDS Docs
• GCP Metadata Docs
• OCI IMDS Docs
• and else (like Alibaba Cloud)

Therefore, I believe url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks are more practical to implement.

My concern is that managing url_host_allowlist and url_host_allowlist_regexp could become very challenging if users create long lists of entries.

Do you have any good ideas for handling this more efficiently?

And does this approach provide the right level of SSRF protection?

[alvin-celerdata]

Multiple Parameters
When I planned to make it, I only saw multiple parameters used in table functions, so I thought it was a rare case.
If not, then I believe supporting multiple parameters with named arguments is the best way to handle this.

If this is not available for non-table functions, we can support it, because this is the right way in the long-run.

SSRF Safeguards

after first glance of Databricks solution, it ask users to create a connection beforehand.
In this way, users can use this function with pre-defined endpoints, so this will reduce SSRF risks.
In Snowflake, users should create a network policy before accessing any external address
And we can move forward in the same way.

[EdwardArchive]

Multiple Parameters
When I planned to make it, I only saw multiple parameters used in table functions, so I thought it was a rare case.
If not, then I believe supporting multiple parameters with named arguments is the best way to handle this.

If this is not available for non-table functions, we can support it, because this is the right way in the long-run.

SSRF Safeguards

after first glance of Databricks solution, it ask users to create a connection beforehand. In this way, users can use this function with pre-defined endpoints, so this will reduce SSRF risks. In Snowflake, users should create a network policy before accessing any external address And we can move forward in the same way.

Thanks for the idea,

In a simple way, we can reduce SSRF risks by using parameters such as url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks.

For the long term, the Databricks solution with connection configuration can provide a very secure way to manage credentials. However, without strong encryption, this approach may still be risky(storing may be real-world id and password) . The advantage is that the configuration can be reused across all HTTP requests.

In conclusion, applying url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks can effectively help reduce the risk of SSRF attacks

[alvin-celerdata]

In conclusion, applying url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks can effectively help reduce the risk of SSRF attacks

I'm wondering if we moved forward with this solution at the beginning, how can we achieve backward compatibility when we want to switch to the long-term solution?

[EdwardArchive]

In conclusion, applying url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks can effectively help reduce the risk of SSRF attacks

I'm wondering if we moved forward with this solution at the beginning, how can we achieve backward compatibility when we want to switch to the long-term solution?

Hi @alvin-celerdata,

I have noticed that non-table functions currently do not support named parameters, and I believe this is an area we should consider improving.

Regarding SSRF attacks, I think the proposed properties maintain full backward compatibility, since they are simply parameters applied when an HTTP request is executed.

As for the use of CONNECTION objects, after further consideration, I believe they would be over-engineering for this specific use case. Databricks CONNECTION is designed to manage all external access—not only HTTP, but also federation queries (PostgreSQL, Redshift, MySQL, Hive Metastore, REST catalogs, etc.).

In StarRocks, the External Catalog already fulfills this role for database federation. Creating a separate HTTP-only CONNECTION would:

• Break architectural consistency
• Duplicate concepts already handled by External Catalog
• Introduce unnecessary complexity for a single function

For key security, I think using save_secret() and get_secret() functions would be a better approach. This way, we can maintain backward compatibility(or is the the connection object - one time using we don't need this, but some one using this SELECT Query on TASK side then I'll need this).

An example function call could look like this:

SELECT http_request(
    url => 'https://anywere.you.want/',
    method => 'POST',
    path => 'chat.postMessage',      -- append on base path
    body => '{"channel": "C123"}',
    username => get_secret('slack_user_name'),
    password => get_secret('slack_pssword)
);

I hope my idea makes sense.
thanks

[alvin-celerdata]

@EdwardArchive
First of all, thanks for the further investigation, but I want to align with your thoughts.

I have noticed that non-table functions currently do not support named parameters, and I believe this is an area we should consider improving.

We get aligned on this point, and we definitely need to improve it.

As for the use of CONNECTION objects, after further consideration, I believe they would be over-engineering for this specific use case. Databricks CONNECTION is designed to manage all external access—not only HTTP, but also federation queries (PostgreSQL, Redshift, MySQL, Hive Metastore, REST catalogs, etc.).

For this part, I have some different ideas on this design. For Databricks, people can only access the endpoint, which is defined by CREATE CONNECTION. In this way, no one else can use the http_request function to access any malicious URL. What they can access is what all CONNECTIONS are defined.
And in this way, they avoid the SSRF.

And if my understanding is correct, and our http_request implementation can accept any url input, then we can't avoid SSRF without additional configuration. If this is correct, my suggestion was to limit the function argument to a predefined endpoint rather than any URL.

[EdwardArchive]

@EdwardArchive First of all, thanks for the further investigation, but I want to align with your thoughts.

I have noticed that non-table functions currently do not support named parameters, and I believe this is an area we should consider improving.

We get aligned on this point, and we definitely need to improve it.

As for the use of CONNECTION objects, after further consideration, I believe they would be over-engineering for this specific use case. Databricks CONNECTION is designed to manage all external access—not only HTTP, but also federation queries (PostgreSQL, Redshift, MySQL, Hive Metastore, REST catalogs, etc.).

For this part, I have some different ideas on this design. For Databricks, people can only access the endpoint, which is defined by . In this way, no one else can use the function to access any malicious URL. What they can access is what all CONNECTIONS are defined. And in this way, they avoid the SSRF.CREATE CONNECTION``http_request

And if my understanding is correct, and our implementation can accept any input, then we can't avoid SSRF without additional configuration. If this is correct, my suggestion was to limit the function argument to a predefined endpoint rather than any URL.http_request``url

Thanks @alvin-celerdata,

I believe the parameters url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks should have default values:

• url_block_private_networks (default: true) – blocks the following IP ranges:
  -- 10.0.0.0/8
  -- 172.16.0.0/16
  --192.168.0.0/24
• url_host_allowlist (default: "")
• url_host_allowlist_regexp (default: "")

If both url_host_allowlist and url_host_allowlist_regexp are set to an empty string, users will not be able to make any HTTP requests by default.

If a request to a specific path such as https://httpbin.org is required, the user would need to explicitly add either:

• 18.214.194.42 (IP address), or
• httpbin.org.*

With this approach, I believe we can more easily mitigate SSRF risks.

[alvin-celerdata]

@EdwardArchive

I believe the parameters url_host_allowlist, url_host_allowlist_regexp, and url_block_private_networks should have default values:

Are these configurations in fe.conf?
If they are frontend configurations, users need to restart their frontend processes to apply the new configuration.

[EdwardArchive]

@alvin-celerdata

No, this configuration will be handled as a FE Dynamic config.
It will be controlled through the statement ADMIN SET CONFIG, so I would prefer not to introduce additional changes here.

like ADMIN SET FRONTEND CONFIG ("http_request_ssl_verification_required" = "true"); or elase

[alvin-celerdata]

@EdwardArchive

But there are still some shortcomings for dynamic configuration

1. Users need to execute this command for every Frontend Instance.
2. Users need to change the configuration file of each Frontend to persist the changes.

And creating a CONNECTION to achieve the above operations with just a single SQL statement.

This is good, but it doesn't mean we should go this way right away.

But I just came up with an idea: how about we introduce a default_connection concept that can be set in fe.conf, like our default_volume before we introduce CREATE VOLUME?
And then the http_request will only accept connection as its argument, rather than any URL.

And in the future, we can introduce the ability to create or drop CONNECTION or API dynamically.

[EdwardArchive]

@alvin-celerdata
Thank you for the thoughtful suggestion.
I agree that the CONNECTION abstraction is a good idea for the future.

However, even with CONNECTION, I believe cluster-wide security policies
(like url_host_allowlist, url_block_private_networks) would still
need to be managed separately via ADMIN SET CONFIG, as these are
admin-enforced controls that should apply regardless of how connections
are defined. Please let me know if you see a different approach.

If there's significant demand for encrypted credential storage and
connection reusability for http_request, I think introducing a
CONNECTION object would make sense as a follow-up enhancement.

This would also enable backward compatibility through named parameter
functions, which I think would be a logical next step.

for example

CONNECTION Object Design

CREATE CONNECTION my_api (
    type = 'HTTP',                          -- Type (HTTP, gRPC, extensible)
    url = 'https://api.example.com',
    method = 'POST',
    headers = '{"Authorization": "Bearer xxx"}',
    timeout_ms = 30000,
    ssl_verify = true,
    username = 'user',
    password = 'secret'                     -- Stored securely
);

Compatibility with Existing http_request

• Option A: Switch to CONNECTION-only

  • Deprecate the existing approach
    SELECT http_request(url => 'https://...'); --> No longer supported

  • Use CONNECTION exclusively

  SELECT http_request(connection => 'my_api');   
  SELECT http_request(connection => 'my_api', body => '{"data":1}');  -- Partial override
  

• Option B: Support Both

  • Continue supporting the existing approach
    SELECT http_request(url => 'https://...'); --> Backward compatibility

  • Also support CONNECTION
    SELECT http_request(connection => 'my_api'); --> New approach

Thanks

[alvin-celerdata]

• Option B: Support Both

  • Continue supporting the existing approach
    SELECT http_request(url => 'https://...'); --> Backward compatibility

  • Also support CONNECTION
    SELECT http_request(connection => 'my_api'); --> New approach

It makes sense. We can introduce a configuration to enable/disable the url argument in this function.