.Net: [RestApi] Dynamic payload generation with namespaced payload parameters (#2467)

### Motivation and Context

The dynamic payload creation was requested by a few internal teams. It
allows SK RestAPI functionality consumer code to only supply arguments
for parameters specified in the OpenAPI payload metadata for PUT and
POST operations, while the rest – payload constriction, data type
conversion, parameter namespacing, etc. – is handled by SK RestAPI
functionality.

### Description

#### Context
Today, the only way to pass a payload for a PUT or POST RestAPI
operation is by adding the `payload` argument to the context variables
collection before executing the operation. The `payload` argument, for
operations requiring an 'application/json' payload, is expected to be a
valid JSON with all the necessary properties initialized. This approach
is not convenient for SK consumers who only want to provide the required
payload parameters, similar to how they handle query string and header
parameters, without having to construct the payload themselves.

#### Dynamic creation of payload
By default, dynamic payload creation is disabled to ensure backward
compatibility. To enable it, the `BuildOperationPayloadDynamically`
property of the `OpenApiSkillExecutionParameters` execution parameters
should be set to `true` when importing the AI plugin:

```csharp
var plugin = await kernel.ImportAIPluginAsync("<skill name>", new Uri("<chatGPT-plugin>"), new OpenApiSkillExecutionParameters(httpClient) { BuildOperationPayloadDynamically = true });
```

So, if a RestAPI operation payload schema requires payload like the
following:
```json
{
	"value": "secret-value",
	"attributes": {
		"enabled": true
	}
}
```

To dynamically build it, the consumer code needs to register the
following arguments in the context variables collection:

```csharp
var contextVariables = new ContextVariables();
contextVariables.Set("value", "secret-value");
contextVariables.Set("enabled", true);
```

#### Payload parameter namespacing
When building payload dynamically, and a RestAPI operation requires the
'application/json' content type, it's possible that the content may have
properties with the identical names, see example below, at different
levels. In such cases, SK can't resolve their values unambiguously from
a flat list of arguments unless there's a namespacing mechanism that
allows the distinction of those properties from one another.

The namespacing mechanism relies on prefixing parameter names with their
parent parameter name, separated by dots. So, the 'namespaced' parameter
names should be used when adding arguments to the context variables
collection. For example, consider this JSON:

```json
{ 
  "upn": "<sender upn>", 
  "receiver": {
    "upn": "<receiver upn>"
  },
  "cc": {
    "upn": "<cc upn>"
  }
}
```
It contains `upn` properties at different levels. The argument
registration for the parameters (property values) will look like:
```csharp
var contextVariables = new ContextVariables();
contextVariables.Set("upn", "<sender-upn-value>");
contextVariables.Set("receiver.upn", "<receiver-upn-value>");
contextVariables.Set("cc.upn", "<cc-upn-value>");
```

The namespacing mechanism is disabled by default for backward
compatibility and can be easily enabled by setting the value `true` to
the `NamespacePayloadParameters` property of the
`OpenApiSkillExecutionParameters` execution parameters when importing
the AI plugin:

```csharp
var plugin = await kernel.ImportAIPluginAsync("<skill name>", new Uri("<chatGPT-plugin>"), new OpenApiSkillExecutionParameters(httpClient) { NamespacePayloadParameters = true });
```

#### Parameter arguments data type conversion
Currently, the SK context variables collection only supports string
variables.This means that in order to set or add a primitive or complex
type variable, it must first be converted or serialized into a string.
However, this behavior causes issues when the RestAPI functionality
sends HTTP requests with an 'application/json' payload. For instance, a
property like enabled might have the string value `"true"` instead of
the intended Boolean value `true`.

To mitigate this problem, the RestAPIOperationRunner class iterates over
the operation payload metadata. During this process, it converts all
string arguments to their corresponding types as specified in the
metadata. This ensures accurate type representation within the operation
payload.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>

Author: SergeyMenshykh

docs/decisions/006-open-api-dynamic-payload-and-namespaces.md

@@ -0,0 +1,79 @@
+---
+# These are optional elements. Feel free to remove any of them.
+status: proposed
+date: 2023-08-15
+deciders: shawncal
+consulted:
+informed:
+---
+# Dynamic payload building for PUT and POST RestAPI operations and parameter namespacing
+
+## Context and Problem Statement
+Currently, the SK OpenAPI does not allow the dynamic creation of payload/body for PUT and POST RestAPI operations, even though all the required metadata is available. One of the reasons the functionality was not fully developed originally, and eventually removed is that JSON payload/body content of PUT and POST RestAPI operations might contain properties with identical names at various levels. It was not clear how to unambiguously resolve their values from the flat list of context variables. Another reason the functionality has not been added yet is that the 'payload' context variable, along with RestAPI operation data contract schema(OpenAPI, JSON schema, Typings?) should have been sufficient for LLM to provide fully fleshed-out JSON payload/body content without the need to build it dynamically.
+
+<!-- This is an optional element. Feel free to remove. -->
+## Decision Drivers
+* Create a mechanism that enables the dynamic construction of the payload/body for PUT and POST RestAPI operations.
+* Develop a mechanism(namespacing) that allows differentiation of payload properties with identical names at various levels for PUT and POST RestAPI operations.
+* Aim to minimize breaking changes and maintain backward compatibility of the code as much as possible.
+
+## Considered Options
+* Enable the dynamic creation of payload and/or namespacing by default.
+* Enable the dynamic creation of payload and/or namespacing based on configuration.
+
+## Decision Outcome
+Chosen option: "Enable the dynamic creation of payload and/or namespacing based on configuration". This option keeps things compatible, so the change won't affect any SK consumer code. Additionally, it lets SK consumer code easily control both mechanisms, turning them on or off based on the scenario.
+
+## Additional details
+
+### Enabling dynamic creation of payload
+In order to enable the dynamic creation of payloads/bodies for PUT and POST RestAPI operations, please set the `EnableDynamicPayload` property of the `OpenApiSkillExecutionParameters` execution parameters to `true` when importing the AI plugin:
+
+```csharp
+var plugin = await kernel.ImportAIPluginAsync("<skill name>", new Uri("<chatGPT-plugin>"), new OpenApiSkillExecutionParameters(httpClient) { EnableDynamicPayload = true });
+```
+
+To dynamically construct a payload for a RestAPI operation that requires payload like this:
+```json
+{
+	"value": "secret-value",
+	"attributes": {
+		"enabled": true
+	}
+}
+```
+
+Please register the following arguments in context variables collection:
+
+```csharp
+var contextVariables = new ContextVariables();
+contextVariables.Set("value", "secret-value");
+contextVariables.Set("enabled", true);
+```
+
+### Enabling namespacing
+To enable namespacing, set the `EnablePayloadNamespacing` property of the `OpenApiSkillExecutionParameters` execution parameters to `true` when importing the AI plugin:
+
+```csharp
+var plugin = await kernel.ImportAIPluginAsync("<skill name>", new Uri("<chatGPT-plugin>"), new OpenApiSkillExecutionParameters(httpClient) { EnablePayloadNamespacing = true });
+```
+Remember that the namespacing mechanism depends on prefixing parameter names with their parent parameter name, separated by dots. So, use the 'namespaced' parameter names when adding arguments for them to the context variables. Let's consider this JSON:
+
+```json
+{ 
+  "upn": "<sender upn>", 
+  "receiver": {
+    "upn": "<receiver upn>"
+  },
+  "cc": {
+    "upn": "<cc upn>"
+  }
+}
+```
+It contains `upn` properties at different levels. The the argument registration for the parameters(property values) will look like:
+```csharp
+var contextVariables = new ContextVariables();
+contextVariables.Set("upn", "<sender-upn-value>");
+contextVariables.Set("receiver.upn", "<receiver-upn-value>");
+contextVariables.Set("cc.upn", "<cc-upn-value>");
+```

dotnet/samples/KernelSyntaxExamples/Example21_ChatGPTPlugins.cs

@@ -22,7 +22,7 @@ private static async Task RunChatGptPluginAsync()
         using HttpClient httpClient = new();
 
         //Import a ChatGPT plugin via URI
-        var skill = await kernel.ImportAIPluginAsync("<skill name>", new Uri("<chatGPT-plugin>"), new OpenApiSkillExecutionParameters(httpClient));
+        var skill = await kernel.ImportAIPluginAsync("<skill name>", new Uri("<chatGPT-plugin>"), new OpenApiSkillExecutionParameters(httpClient) { EnableDynamicPayload = true });
 
         //Add arguments for required parameters, arguments for optional ones can be skipped.
         var contextVariables = new ContextVariables();

dotnet/src/Skills/Skills.OpenAPI/Extensions/KernelAIPluginExtensions.cs

@@ -43,21 +43,21 @@ public static async Task<IDictionary<string, ISKFunction>> ImportAIPluginAsync(
         Verify.ValidSkillName(skillName);
 
 #pragma warning disable CA2000 // Dispose objects before losing scope. No need to dispose the Http client here. It can either be an internal client using NonDisposableHttpClientHandler or an external client managed by the calling code, which should handle its disposal.
-        var internalHttpClient = HttpClientProvider.GetHttpClient(kernel.Config, executionParameters?.HttpClient, kernel.LoggerFactory);
+        var httpClient = HttpClientProvider.GetHttpClient(kernel.Config, executionParameters?.HttpClient, kernel.LoggerFactory);
 #pragma warning restore CA2000
 
         var pluginContents = await LoadDocumentFromFilePath(
             kernel,
             filePath,
             executionParameters,
-            internalHttpClient,
+            httpClient,
             cancellationToken).ConfigureAwait(false);
 
         return await CompleteImport(
             kernel,
             pluginContents,
             skillName,
-            internalHttpClient,
+            httpClient,
             executionParameters,
             cancellationToken).ConfigureAwait(false);
     }
@@ -82,21 +82,21 @@ public static async Task<IDictionary<string, ISKFunction>> ImportAIPluginAsync(
         Verify.ValidSkillName(skillName);
 
 #pragma warning disable CA2000 // Dispose objects before losing scope. No need to dispose the Http client here. It can either be an internal client using NonDisposableHttpClientHandler or an external client managed by the calling code, which should handle its disposal.
-        var internalHttpClient = HttpClientProvider.GetHttpClient(kernel.Config, executionParameters?.HttpClient, kernel.LoggerFactory);
+        var httpClient = HttpClientProvider.GetHttpClient(kernel.Config, executionParameters?.HttpClient, kernel.LoggerFactory);
 #pragma warning restore CA2000
 
         var pluginContents = await LoadDocumentFromUri(
             kernel,
             uri,
             executionParameters,
-            internalHttpClient,
+            httpClient,
             cancellationToken).ConfigureAwait(false);
 
         return await CompleteImport(
             kernel,
             pluginContents,
             skillName,
-            internalHttpClient,
+            httpClient,
             executionParameters,
             cancellationToken).ConfigureAwait(false);
     }
@@ -121,7 +121,7 @@ public static async Task<IDictionary<string, ISKFunction>> ImportAIPluginAsync(
         Verify.ValidSkillName(skillName);
 
 #pragma warning disable CA2000 // Dispose objects before losing scope. No need to dispose the Http client here. It can either be an internal client using NonDisposableHttpClientHandler or an external client managed by the calling code, which should handle its disposal.
-        var internalHttpClient = HttpClientProvider.GetHttpClient(kernel.Config, executionParameters?.HttpClient, kernel.LoggerFactory);
+        var httpClient = HttpClientProvider.GetHttpClient(kernel.Config, executionParameters?.HttpClient, kernel.LoggerFactory);
 #pragma warning restore CA2000
 
         var pluginContents = await LoadDocumentFromStream(kernel, stream).ConfigureAwait(false);
@@ -130,7 +130,7 @@ public static async Task<IDictionary<string, ISKFunction>> ImportAIPluginAsync(
             kernel,
             pluginContents,
             skillName,
-            internalHttpClient,
+            httpClient,
             executionParameters,
             cancellationToken).ConfigureAwait(false);
     }
@@ -141,7 +141,7 @@ private static async Task<IDictionary<string, ISKFunction>> CompleteImport(
         IKernel kernel,
         string pluginContents,
         string skillName,
-        HttpClient internalHttpClient,
+        HttpClient httpClient,
         OpenApiSkillExecutionParameters? executionParameters,
         CancellationToken cancellationToken)
     {
@@ -160,7 +160,7 @@ private static async Task<IDictionary<string, ISKFunction>> CompleteImport(
             kernel,
             skillName,
             executionParameters,
-            internalHttpClient,
+            httpClient,
             pluginContents,
             cancellationToken).ConfigureAwait(false);
     }
@@ -169,7 +169,7 @@ private static async Task<IDictionary<string, ISKFunction>> LoadSkill(
         IKernel kernel,
         string skillName,
         OpenApiSkillExecutionParameters? executionParameters,
-        HttpClient internalHttpClient,
+        HttpClient httpClient,
         string pluginJson,
         CancellationToken cancellationToken)
     {
@@ -179,7 +179,12 @@ private static async Task<IDictionary<string, ISKFunction>> LoadSkill(
         {
             var operations = await parser.ParseAsync(documentStream, executionParameters?.IgnoreNonCompliantErrors ?? false, cancellationToken).ConfigureAwait(false);
 
-            var runner = new RestApiOperationRunner(internalHttpClient, executionParameters?.AuthCallback, executionParameters?.UserAgent);
+            var runner = new RestApiOperationRunner(
+                httpClient,
+                executionParameters?.AuthCallback,
+                executionParameters?.UserAgent,
+                executionParameters?.EnableDynamicPayload ?? false,
+                executionParameters?.EnablePayloadNamespacing ?? false);
 
             var skill = new Dictionary<string, ISKFunction>();
 
@@ -189,7 +194,7 @@ private static async Task<IDictionary<string, ISKFunction>> LoadSkill(
                 try
                 {
                     logger.LogTrace("Registering Rest function {0}.{1}", skillName, operation.Id);
-                    var function = kernel.RegisterRestApiFunction(skillName, runner, operation, executionParameters?.ServerUrlOverride, cancellationToken);
+                    var function = kernel.RegisterRestApiFunction(skillName, runner, operation, executionParameters, cancellationToken);
                     skill[function.Name] = function;
                 }
                 catch (Exception ex) when (!ex.IsCriticalException())
@@ -208,7 +213,7 @@ private static async Task<string> LoadDocumentFromUri(
         IKernel kernel,
         Uri uri,
         OpenApiSkillExecutionParameters? executionParameters,
-        HttpClient internalHttpClient,
+        HttpClient httpClient,
         CancellationToken cancellationToken)
     {
         using var requestMessage = new HttpRequestMessage(HttpMethod.Get, uri.ToString());
@@ -218,7 +223,7 @@ private static async Task<string> LoadDocumentFromUri(
             requestMessage.Headers.UserAgent.Add(ProductInfoHeaderValue.Parse(executionParameters!.UserAgent));
         }
 
-        using var response = await internalHttpClient.SendAsync(requestMessage, cancellationToken).ConfigureAwait(false);
+        using var response = await httpClient.SendAsync(requestMessage, cancellationToken).ConfigureAwait(false);
         response.EnsureSuccessStatusCode();
 
         return await response.Content.ReadAsStringAsync().ConfigureAwait(false);
@@ -228,7 +233,7 @@ private static async Task<string> LoadDocumentFromFilePath(
         IKernel kernel,
         string filePath,
         OpenApiSkillExecutionParameters? executionParameters,
-        HttpClient internalHttpClient,
+        HttpClient httpClient,
         CancellationToken cancellationToken)
     {
         var pluginJson = string.Empty;
@@ -293,18 +298,22 @@ private static bool TryParseAIPluginForUrl(string gptPluginJson, out string? ope
     /// <param name="skillName">Skill name.</param>
     /// <param name="runner">The REST API operation runner.</param>
     /// <param name="operation">The REST API operation.</param>
-    /// <param name="serverUrlOverride">Optional override for REST API server URL if user input required</param>
+    /// <param name="executionParameters">Skill execution parameters.</param>
     /// <param name="cancellationToken">The cancellation token.</param>
     /// <returns>An instance of <see cref="SKFunction"/> class.</returns>
     private static ISKFunction RegisterRestApiFunction(
         this IKernel kernel,
         string skillName,
         RestApiOperationRunner runner,
         RestApiOperation operation,
-        Uri? serverUrlOverride = null,
+        OpenApiSkillExecutionParameters? executionParameters,
         CancellationToken cancellationToken = default)
     {
-        var restOperationParameters = operation.GetParameters(serverUrlOverride);
+        var restOperationParameters = operation.GetParameters(
+            executionParameters?.ServerUrlOverride,
+            executionParameters?.EnableDynamicPayload ?? false,
+            executionParameters?.EnablePayloadNamespacing ?? false
+        );
 
         var logger = kernel.LoggerFactory is not null ? kernel.LoggerFactory.CreateLogger(nameof(KernelAIPluginExtensions)) : NullLogger.Instance;
 

dotnet/src/Skills/Skills.OpenAPI/Extensions/OpenApiSkillExecutionParameters.cs

@@ -37,7 +37,22 @@ public class OpenApiSkillExecutionParameters
     /// <summary>
     /// Optional user agent header value.
     /// </summary>
-    public string? UserAgent { get; set; }
+    public string UserAgent { get; set; }
+
+    /// <summary>
+    /// Determines whether the operation payload is constructed dynamically based on operation payload metadata.
+    /// If false, the operation payload must be provided via the 'payload' context variable.
+    /// </summary>
+    public bool EnableDynamicPayload { get; set; }
+
+    /// <summary>
+    /// Determines whether payload parameter names are augmented with namespaces.
+    /// Namespaces prevent naming conflicts by adding the parent parameter name as a prefix, separated by dots.
+    /// For instance, without namespaces, the 'email' parameter for both the 'sender' and 'receiver' parent parameters
+    /// would be resolved from the same 'email' argument, which is incorrect. However, by employing namespaces,
+    /// the parameters 'sender.email' and 'sender.receiver' will be correctly resolved from arguments with the same names.
+    /// </summary>
+    public bool EnablePayloadNamespacing { get; set; }
 
     /// <summary>
     /// Initializes a new instance of the <see cref="OpenApiSkillExecutionParameters"/> class.
@@ -49,17 +64,25 @@ public class OpenApiSkillExecutionParameters
     /// <param name="ignoreNonCompliantErrors">A flag indicating whether to ignore non-compliant errors or not
     /// If set to true, the operation execution will not throw exceptions for non-compliant documents.
     /// Please note that enabling this option may result in incomplete or inaccurate execution results.</param>
+    /// <param name="enableDynamicOperationPayload">Determines whether the operation payload is constructed dynamically based on operation payload metadata.
+    /// If false, the operation payload must be provided via the 'payload' context variable.</param>
+    /// <param name="enablePayloadNamespacing">Determines whether payload parameter names are augmented with namespaces.
+    /// Namespaces prevent naming conflicts by adding the parent parameter name as a prefix, separated by dots.</param>
     public OpenApiSkillExecutionParameters(
         HttpClient? httpClient = null,
         AuthenticateRequestAsyncCallback? authCallback = null,
         Uri? serverUrlOverride = null,
-        string? userAgent = Telemetry.HttpUserAgent,
-        bool ignoreNonCompliantErrors = false)
+        string userAgent = Telemetry.HttpUserAgent,
+        bool ignoreNonCompliantErrors = false,
+        bool enableDynamicOperationPayload = false,
+        bool enablePayloadNamespacing = false)
     {
         this.HttpClient = httpClient;
         this.AuthCallback = authCallback;
         this.ServerUrlOverride = serverUrlOverride;
         this.UserAgent = userAgent;
         this.IgnoreNonCompliantErrors = ignoreNonCompliantErrors;
+        this.EnableDynamicPayload = enableDynamicOperationPayload;
+        this.EnablePayloadNamespacing = enablePayloadNamespacing;
     }
 }