self hosted models (#63899)

This PR is stacked on top of all the prior work @chrsmith has done for
shuffling configuration data around; it implements the new "Self hosted
models" functionality.

## Configuration

Configuring a Sourcegraph instance to use self-hosted models basically
involves adding some configuration like this to the site config (if you
set `modelConfiguration`, you are opting in to the new system which is
in early access):

```
  // Setting this field means we are opting into the new Cody model configuration system.
  "modelConfiguration": {
    // Disable use of Sourcegraph's servers for model discovery
    "sourcegraph": null,

    // Create two model providers
    "providerOverrides": [
      {
        // Our first model provider "mistral" will be a Huggingface TGI deployment which hosts our
        // mistral model for chat functionality.
        "id": "mistral",
        "displayName": "Mistral",
        "serverSideConfig": {
          "type": "huggingface-tgi",
          "endpoints": [{"url": "https://mistral.example.com/v1"}]
        },
      },
      {
        // Our second model provider "bigcode" will be a Huggingface TGI deployment which hosts our
        // bigcode/starcoder model for code completion functionality.
        "id": "bigcode",
        "displayName": "Bigcode",
        "serverSideConfig": {
          "type": "huggingface-tgi",
          "endpoints": [{"url": "http://starcoder.example.com/v1"}]
        }
      }
    ],

    // Make these two models available to Cody users
    "modelOverridesRecommendedSettings": [
      "mistral::v1::mixtral-8x7b-instruct",
      "bigcode::v1::starcoder2-7b"
    ],

    // Configure which models Cody will use by default
    "defaultModels": {
      "chat": "mistral::v1::mixtral-8x7b-instruct",
      "fastChat": "mistral::v1::mixtral-8x7b-instruct",
      "codeCompletion": "bigcode::v1::starcoder2-7b"
    }
  }
```

More advanced configurations are possible, the above is our blessed
configuration for today.

## Hosting models

Another major component of this work is starting to build up
recommendations around how to self-host models, which ones to use, how
to configure them, etc.

For now, we've been testing with these two on a machine with dual A100s:

* Huggingface TGI (this is a Docker container for model inference, which
provides an OpenAI-compatible API - and is widely popular)
* Two models:
* Starcoder2 for code completion; specifically `bigcode/starcoder2-15b`
with `eetq` 8-bit quantization.
* Mixtral 8x7b instruct for chat; specifically
`casperhansen/mixtral-instruct-awq` which uses `awq` 4-bit quantization.

This is our 'starter' configuration. Other models - specifically other
starcoder 2, and mixtral instruct models - certainly work too, and
higher parameter versions may of course provide better results.

Documentation for how to deploy Huggingface TGI, suggested configuration
and debugging tips - coming soon.

## Advanced configuration

As part of this effort, I have added a quite extensive set of
configuration knobs to to the client side model configuration (see `type
ClientSideModelConfigOpenAICompatible` in this PR)

Some of these configuration options are needed for things to work at a
basic level, while others (e.g. prompt customization) are not needed for
basic functionality, but are very important for customers interested in
self-hosting their own models.

Today, Cody clients have a number of different _autocomplete provider
implementations_ which tie model-specific logic to enable autocomplete,
to a provider. For example, if you use a GPT model through Azure OpenAI,
the autocomplete provider for that is entirely different from what you'd
get if you used a GPT model through OpenAI officially. This can lead to
some subtle issues for us, and so it is worth exploring ways to have a
_generalized autocomplete provider_ - and since with self-hosted models
we _must_ address this problem, these configuration knobs fed to the
client from the server are a pathway to doing that - initially just for
self-hosted models, but in the future possibly generalized to other
providers.

## Debugging facilities

Working with customers in the past to use OpenAI-compatible APIs, we've
learned that debugging can be quite a pain. If you can't see what
requests the Sourcegraph backend is making, and what it is getting
back.. it can be quite painful to debug.

This PR implements quite extensive logging, and a `debugConnections`
flag which can be turned on to enable logging of the actual request
payloads and responses. This is critical when a customer is trying to
add support for a new model, their own custom OpenAI API service, etc.

## Robustness

Working with customers in the past, we also learned that various parts
of our backend `openai` provider were not super robust. For example, [if
more than one message was present it was a fatal
error](https://github.com/sourcegraph/sourcegraph/blob/main/internal/completions/client/openai/openai.go#L305),
or if the SSE stream yielded `{"error"}` payloads, they would go
ignored. Similarly, the SSE event stream parser we use is heavily
tailored towards [the exact response
structure](https://github.com/sourcegraph/sourcegraph/blob/main/internal/completions/client/openai/decoder.go#L15-L19)
which OpenAI's official API returns, and is therefor quite brittle if
connecting to a different SSE stream.

For this work, I have _started by forking_ our
`internal/completions/client/openai` - and made a number of major
improvements to it to make it more robust, handle errors better, etc.

I have also replaced the usage of a custom SSE event stream parser -
which was not spec compliant and brittle - with a proper SSE event
stream parser that recently popped up in the Go community:
https://github.com/tmaxmax/go-sse

My intention is that after more extensive testing, this new
`internal/completions/client/openaicompatible` provider will be more
robust, more correct, and all around better than
`internal/completions/client/openai` (and possibly the azure one) so
that we can just supersede those with this new `openaicompatible` one
entirely.

## Client implementation

Much of the work done in this PR is just "let the site admin configure
things, and broadcast that config to the client through the new model
config system."

Actually getting the clients to respect the new configuration, is a task
I am tackling in future `sourcegraph/cody` PRs.

## Test plan

1. This change currently lacks any unit/regression tests, that is a
major noteworthy point. I will follow-up with those in a future PR.
* However, these changes are **incredibly** isolated, clearly only
affecting customers who opt-in to this new self-hosted models
configuration.
* Most of the heavy lifting (SSE streaming, shuffling data around) is
done in other well-tested codebases.
2. Manual testing has played a big role here, specifically:
* Running a dev instance with the new configuration, actually connected
to Huggingface TGI deployed on a remote server.
* Using the new `debugConnections` mechanism (which customers would use)
to directly confirm requests are going to the right places, with the
right data and payloads.
* Confirming with a new client (changes not yet landed) that
autocomplete and chat functionality work.

Can we use more testing? Hell yeah, and I'm going to add it soon. Does
it work quite well and have small room for error? Also yes.

## Changelog

Cody Enterprise: added a new configuration for self-hosting models.
Reach out to support if you would like to use this feature as it is in
early access.

---------

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>

Author: Stephen Gutekanst

cmd/frontend/internal/modelconfig/siteconfig.go

@@ -173,13 +173,18 @@ func convertServerSideProviderConfig(cfg *schema.ServerSideProviderConfig) *type
 				Endpoint:    v.Endpoint,
 			},
 		}
+	} else if v := cfg.HuggingfaceTgi; v != nil {
+		return &types.ServerSideProviderConfig{
+			OpenAICompatible: &types.OpenAICompatibleProviderConfig{
+				Endpoints:         convertOpenAICompatibleEndpoints(v.Endpoints),
+				EnableVerboseLogs: v.EnableVerboseLogs,
+			},
+		}
 	} else if v := cfg.Openaicompatible; v != nil {
-		// TODO(slimsag): self-hosted-models: map this to OpenAICompatibleProviderConfig in the future
 		return &types.ServerSideProviderConfig{
-			GenericProvider: &types.GenericProviderConfig{
-				ServiceName: types.GenericServiceProviderOpenAI,
-				AccessToken: v.AccessToken,
-				Endpoint:    v.Endpoint,
+			OpenAICompatible: &types.OpenAICompatibleProviderConfig{
+				Endpoints:         convertOpenAICompatibleEndpoints(v.Endpoints),
+				EnableVerboseLogs: v.EnableVerboseLogs,
 			},
 		}
 	} else if v := cfg.Sourcegraph; v != nil {
@@ -194,13 +199,57 @@ func convertServerSideProviderConfig(cfg *schema.ServerSideProviderConfig) *type
 	}
 }
 
+func convertOpenAICompatibleEndpoints(configEndpoints []*schema.OpenAICompatibleEndpoint) []types.OpenAICompatibleEndpoint {
+	var endpoints []types.OpenAICompatibleEndpoint
+	for _, e := range configEndpoints {
+		endpoints = append(endpoints, types.OpenAICompatibleEndpoint{
+			URL:         e.Url,
+			AccessToken: e.AccessToken,
+		})
+	}
+	return endpoints
+}
+
 func convertClientSideModelConfig(v *schema.ClientSideModelConfig) *types.ClientSideModelConfig {
 	if v == nil {
 		return nil
 	}
-	return &types.ClientSideModelConfig{
-		// We currently do not have any known client-side model configuration.
+	cfg := &types.ClientSideModelConfig{}
+	if o := v.Openaicompatible; o != nil {
+		cfg.OpenAICompatible = &types.ClientSideModelConfigOpenAICompatible{
+			StopSequences:                   o.StopSequences,
+			EndOfText:                       o.EndOfText,
+			ContextSizeHintTotalCharacters:  intPtrToUintPtr(o.ContextSizeHintTotalCharacters),
+			ContextSizeHintPrefixCharacters: intPtrToUintPtr(o.ContextSizeHintPrefixCharacters),
+			ContextSizeHintSuffixCharacters: intPtrToUintPtr(o.ContextSizeHintSuffixCharacters),
+			ChatPreInstruction:              o.ChatPreInstruction,
+			EditPostInstruction:             o.EditPostInstruction,
+			AutocompleteSinglelineTimeout:   uint(o.AutocompleteSinglelineTimeout),
+			AutocompleteMultilineTimeout:    uint(o.AutocompleteMultilineTimeout),
+			ChatTopK:                        float32(o.ChatTopK),
+			ChatTopP:                        float32(o.ChatTopP),
+			ChatTemperature:                 float32(o.ChatTemperature),
+			ChatMaxTokens:                   uint(o.ChatMaxTokens),
+			AutoCompleteTopK:                float32(o.AutoCompleteTopK),
+			AutoCompleteTopP:                float32(o.AutoCompleteTopP),
+			AutoCompleteTemperature:         float32(o.AutoCompleteTemperature),
+			AutoCompleteSinglelineMaxTokens: uint(o.AutoCompleteSinglelineMaxTokens),
+			AutoCompleteMultilineMaxTokens:  uint(o.AutoCompleteMultilineMaxTokens),
+			EditTopK:                        float32(o.EditTopK),
+			EditTopP:                        float32(o.EditTopP),
+			EditTemperature:                 float32(o.EditTemperature),
+			EditMaxTokens:                   uint(o.EditMaxTokens),
+		}
+	}
+	return cfg
+}
+
+func intPtrToUintPtr(v *int) *uint {
+	if v == nil {
+		return nil
 	}
+	ptr := uint(*v)
+	return &ptr
 }
 
 func convertServerSideModelConfig(cfg *schema.ServerSideModelConfig) *types.ServerSideModelConfig {
@@ -213,6 +262,12 @@ func convertServerSideModelConfig(cfg *schema.ServerSideModelConfig) *types.Serv
 				ARN: v.Arn,
 			},
 		}
+	} else if v := cfg.Openaicompatible; v != nil {
+		return &types.ServerSideModelConfig{
+			OpenAICompatible: &types.ServerSideModelConfigOpenAICompatible{
+				APIModel: v.ApiModel,
+			},
+		}
 	} else {
 		panic(fmt.Sprintf("illegal state: %+v", v))
 	}
@@ -262,19 +317,14 @@ func convertModelCapabilities(capabilities []string) []types.ModelCapability {
 //
 // It would specify these equivalent options for them under `modelOverrides`:
 var recommendedSettings = map[types.ModelRef]types.ModelOverride{
-	"bigcode::v1::starcoder2-3b":          recommendedSettingsStarcoder2("bigcode::v1::starcoder2-3b", "Starcoder2 3B", "starcoder2-3b"),
 	"bigcode::v1::starcoder2-7b":          recommendedSettingsStarcoder2("bigcode::v1::starcoder2-7b", "Starcoder2 7B", "starcoder2-7b"),
 	"bigcode::v1::starcoder2-15b":         recommendedSettingsStarcoder2("bigcode::v1::starcoder2-15b", "Starcoder2 15B", "starcoder2-15b"),
-	"mistral::v1::mistral-7b":             recommendedSettingsMistral("mistral::v1::mistral-7b", "Mistral 7B", "mistral-7b"),
 	"mistral::v1::mistral-7b-instruct":    recommendedSettingsMistral("mistral::v1::mistral-7b-instruct", "Mistral 7B Instruct", "mistral-7b-instruct"),
-	"mistral::v1::mixtral-8x7b":           recommendedSettingsMistral("mistral::v1::mixtral-8x7b", "Mixtral 8x7B", "mixtral-8x7b"),
-	"mistral::v1::mixtral-8x22b":          recommendedSettingsMistral("mistral::v1::mixtral-8x22b", "Mixtral 8x22B", "mixtral-8x22b"),
 	"mistral::v1::mixtral-8x7b-instruct":  recommendedSettingsMistral("mistral::v1::mixtral-8x7b-instruct", "Mixtral 8x7B Instruct", "mixtral-8x7b-instruct"),
 	"mistral::v1::mixtral-8x22b-instruct": recommendedSettingsMistral("mistral::v1::mixtral-8x22b", "Mixtral 8x22B", "mixtral-8x22b-instruct"),
 }
 
 func recommendedSettingsStarcoder2(modelRef, displayName, modelName string) types.ModelOverride {
-	// TODO(slimsag): self-hosted-models: tune these further based on testing
 	return types.ModelOverride{
 		ModelRef:     types.ModelRef(modelRef),
 		DisplayName:  displayName,
@@ -285,15 +335,18 @@ func recommendedSettingsStarcoder2(modelRef, displayName, modelName string) type
 		Tier:         types.ModelTierEnterprise,
 		ContextWindow: types.ContextWindow{
 			MaxInputTokens:  8192,
-			MaxOutputTokens: 4000,
+			MaxOutputTokens: 4096,
+		},
+		ClientSideConfig: &types.ClientSideModelConfig{
+			OpenAICompatible: &types.ClientSideModelConfigOpenAICompatible{
+				StopSequences: []string{"<|endoftext|>", "<file_sep>"},
+				EndOfText:     "<|endoftext|>",
+			},
 		},
-		ClientSideConfig: nil,
-		ServerSideConfig: nil,
 	}
 }
 
 func recommendedSettingsMistral(modelRef, displayName, modelName string) types.ModelOverride {
-	// TODO(slimsag): self-hosted-models: tune these further based on testing
 	return types.ModelOverride{
 		ModelRef:     types.ModelRef(modelRef),
 		DisplayName:  displayName,
@@ -304,9 +357,10 @@ func recommendedSettingsMistral(modelRef, displayName, modelName string) types.M
 		Tier:         types.ModelTierEnterprise,
 		ContextWindow: types.ContextWindow{
 			MaxInputTokens:  8192,
-			MaxOutputTokens: 4000,
+			MaxOutputTokens: 4096,
+		},
+		ClientSideConfig: &types.ClientSideModelConfig{
+			OpenAICompatible: &types.ClientSideModelConfigOpenAICompatible{},
 		},
-		ClientSideConfig: nil,
-		ServerSideConfig: nil,
 	}
 }

cmd/frontend/internal/modelconfig/siteconfig_completions.go

@@ -160,8 +160,8 @@ func getProviderConfiguration(siteConfig *conftypes.CompletionsConfig) *types.Se
 			Endpoint:    siteConfig.Endpoint,
 		}
 
-		// For all the other types of providers you can define in the site configuration, we
-		// just use a generic config. Rather than creating one for Anthropic, Fireworks, Google, etc.
+		// For all the other types of providers you can define in the legacy "completions" site configuration,
+		// we just use a generic config. Rather than creating one for Anthropic, Fireworks, Google, etc.
 		// We'll add those when needed, when we expose the newer style configuration in the site-config.
 	default:
 		serverSideConfig.GenericProvider = &types.GenericProviderConfig{

deps.bzl

@@ -6237,6 +6237,13 @@ def go_dependencies():
         sum = "h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=",
         version = "v0.6.1",
     )
+    go_repository(
+        name = "com_github_tmaxmax_go_sse",
+        build_file_proto_mode = "disable_global",
+        importpath = "github.com/tmaxmax/go-sse",
+        sum = "h1:pPpTgyyi1r7vG2o6icebnpGEh3ebcnBXqDWkb7aTofs=",
+        version = "v0.8.0",
+    )
     go_repository(
         name = "com_github_tmc_dot",
         build_file_proto_mode = "disable_global",

go.mod

@@ -318,6 +318,7 @@ require (
 	github.com/sourcegraph/sourcegraph/lib v0.0.0-20240524140455-2589fef13ea8
 	github.com/sourcegraph/sourcegraph/lib/managedservicesplatform v0.0.0-00010101000000-000000000000
 	github.com/sourcegraph/sourcegraph/monitoring v0.0.0-00010101000000-000000000000
+	github.com/tmaxmax/go-sse v0.8.0
 	github.com/vektah/gqlparser/v2 v2.4.5
 	github.com/vvakame/gcplogurl v0.2.0
 	go.opentelemetry.io/collector/config/confighttp v0.103.0

go.sum

@@ -2410,6 +2410,8 @@ github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFA
 github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=
 github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=
 github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=
+github.com/tmaxmax/go-sse v0.8.0 h1:pPpTgyyi1r7vG2o6icebnpGEh3ebcnBXqDWkb7aTofs=
+github.com/tmaxmax/go-sse v0.8.0/go.mod h1:HLoxqxdH+7oSUItjtnpxjzJedfr/+Rrm/dNWBcTxJFM=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80 h1:nrZ3ySNYwJbSpD6ce9duiP+QkD3JuLCcWkdaehUS/3Y=
 github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80/go.mod h1:iFyPdL66DjUD96XmzVL3ZntbzcflLnznH0fr99w5VqE=

internal/completions/client/BUILD.bazel

@@ -17,6 +17,7 @@ go_library(
         "//internal/completions/client/fireworks",
         "//internal/completions/client/google",
         "//internal/completions/client/openai",
+        "//internal/completions/client/openaicompatible",
         "//internal/completions/tokenusage",
         "//internal/completions/types",
         "//internal/httpcli",

internal/completions/client/client.go

@@ -10,6 +10,7 @@ import (
 	"github.com/sourcegraph/sourcegraph/internal/completions/client/fireworks"
 	"github.com/sourcegraph/sourcegraph/internal/completions/client/google"
 	"github.com/sourcegraph/sourcegraph/internal/completions/client/openai"
+	"github.com/sourcegraph/sourcegraph/internal/completions/client/openaicompatible"
 	"github.com/sourcegraph/sourcegraph/internal/completions/tokenusage"
 	"github.com/sourcegraph/sourcegraph/internal/completions/types"
 	"github.com/sourcegraph/sourcegraph/internal/httpcli"
@@ -64,6 +65,11 @@ func getAPIProvider(modelConfigInfo types.ModelConfigInfo) (types.CompletionsCli
 		return client, errors.Wrap(err, "getting api provider")
 	}
 
+	// OpenAI Compatible
+	if openAICompatibleCfg := ssConfig.OpenAICompatible; openAICompatibleCfg != nil {
+		return openaicompatible.NewClient(httpcli.UncachedExternalClient, *tokenManager), nil
+	}
+
 	// The "GenericProvider" is an escape hatch for a set of API Providers not needing any additional configuration.
 	if genProviderCfg := ssConfig.GenericProvider; genProviderCfg != nil {
 		token := genProviderCfg.AccessToken

internal/completions/client/openaicompatible/BUILD.bazel

@@ -0,0 +1,20 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "openaicompatible",
+    srcs = [
+        "openaicompatible.go",
+        "types.go",
+    ],
+    importpath = "github.com/sourcegraph/sourcegraph/internal/completions/client/openaicompatible",
+    visibility = ["//:__subpackages__"],
+    deps = [
+        "//internal/completions/tokenizer",
+        "//internal/completions/tokenusage",
+        "//internal/completions/types",
+        "//internal/modelconfig/types",
+        "//lib/errors",
+        "@com_github_sourcegraph_log//:log",
+        "@com_github_tmaxmax_go_sse//:go-sse",
+    ],
+)