LLM

LLM Options

The name field is an arbitrary string descriptor used only for administrative identification. Choose a name that reflects the purpose of the record. This field has no bearing on the configuration or settings determined by this scaffold.

The default checkbox, if checked specifies this LLM Option will be used for all portals without an explicit LLM option defined.

The temperature field is a number between 0 and 1 (0.8 default). Increasing the temperature value will make the model answer more creatively.

The chatbot name field is used to set the name of the chatbot, default is Romeo George.

The chatbot avatar field allows the operator to upload a custom image to be as the chatbots avatar in chatbox window.

The apply guardrails checkbox applies guardrails against harmful, unethical, racist, sexist, toxic, etc. conversations. This is applied after any custom instructions and is enabled by default.

The custom instructions field allows the operator to provide the LLM with a full set of custom system instructions.

The initial greeting field allows the operator to specify a specific greeting the chatbox will use to introduce itself when a user initiates a new chat.

The default llm model drop down is used to specify the default LLM modem this LLM will use.

The llm models field allows the operator to select all the models that can be used with this LLM Option.

The admin roles field sets the admin roles that will use this LLM option, selecting any roles here will remove them from being selected in another LLM option.

The operator portal field selects which Operator Portals are assigned to this LLM Option. Note that an admin role match is prioritized over an operator portal match. Associating a record removes it from other options.

The landing portals field sets which splash/landing portals use this LLM option. Associating a record removes it from the other options.

The alow anonymous chats field, if checked allows users to chat via the portal without being logged in with an account.

LLM Workers

The adapter field specifies the adapter to be used, Ollama is selected by default.

The run locally checkbox tells the system to run a local server for the specfied backend (adapter) and optionally can be made available to specific WAN targets and or policies.

The host field is used to specify the IP or FQDN of the host providing the API interface for the designated backend.

The port field specifies the port to be used for communication.

The timeout field sets the amount of time the system should wait for the LLM worker to respond.

The online checkbox, if checked indicates that the LLM worker is online and ready to process reqquests, unchecking this field will make the LLM worker unavailable.

The llm model drop down specifies which LLM Model this worker will use by default.

The llm models field specfies which LLM Models this worker is allowed to use.

The use for embeddings field if checked, designates the worker that will be used to generate embeddings for context lookup. Enabling will deactive embedding for other workers.

LLM Models

The name field is the name of the model as recognized by the LLM Workers running it. This name needs to match the model name this field is not arbitrary.

The url NEED Clarification here, does it pull the model to the system and the URL is where that model can be downloaded from.

The formatter specifies which LLM model to use to format the requests that are sent to the LLM Worker Needs clarification

The context window sets the size of the set of information that is relevant for answering questions. Setting a larger context window can allow for more detailed and comprehensive answers. What limits this

The embedding dimensions field sets the scope for how many different features/variables are being considered, different models require varying amounts of data.

The quantization level field is used to set how many bits are being used per word when encoding text data. The higher this value the more detail can be gained from the information/data.

LLM Sources

The source field lets you choose the file to upload that will be used as an embeded source.

The visibilty field sets which users can access this information, if set to admins only, the source will only be referenced if you are logged in as an admin. Admins and Users allows both admins and client users to receieve information from this source, and setting it to anonymous allows any user interacting with the LLM to recieve this information regardless of being logged in or not. The anonymous setting should only be used if the client would be interacting with the chatbot without a login sessions, ie from the splash portal.

LLM Embeddings

An entry in the embedding scaffold represents data the LLM can pull from to answer client questions. The source shows where teh data is from, updated reflects the last time this information was updated, the llm model used and the dimensionality.

LLM Prompts

This scaffold is a list of all the prompts sent to the LLM from the clients.

LLM Requests

This scaffold is a list of all the prompts sent to the LLM from the clients, it will list which llm model was used, the LLM worker, when the task was started, when it completed, and how long it took to complete the response.

Chats

The chats scaffold is a history of the chats that have been initiated on the system.

TODO, ADD Glossary of terms if we think its a good idea.

LLM Setup Example

In this example the hardware is a pc with a 3090 graphics card, WAN + certificate is configured, no other configuration has been done.

Navigate to Services::LLM

Create a new LLM Worker.

Give the record a name, in this case since it will be running locally on the system using Ollama I will use the name Local Ollama.

The adapter field should be set to Ollama, and the run locally checkbox should be checked.

The default port value of 11434 should be used, and timeout can be left at 30 seconds.

Add any WAN targets that should be allowed to communicate on this port and/or any polcies that should be allowed. Being that there is no other configuration currently on this system I will select the default policy, if this were a live deployment I would need to add any client policies that will have access to the chatbot.

Leave the online checkbox, checked.

We do not have any llm models yet so we will leave those fields blank. In this demo we will also be using this worker for the embedding so the use for embeddings checkbox should be checked.

Click Create.

To pull a new model click the pull model link and enter the name of the model to be fetched.

Here we will pull the latest llama3 model, then click submit.

Repeat for each desired model.

Edit the LLM Worker created previously and now we can select the default model to use with this worker as well as specify other models the worker can use.

After selecting the default LLM model and any additional models click create.

Next enable embedding, for this we will use the nomic-embed-text:latest model. Edit the nomic LLM worker.

The embedding dimensions field is required when uisng a model for embedding valid values are as follows: 512, 768, 1024. Default is 768 and will be used if the field is blank. Check the use for embeddings checkbox and click update. NOTE It will not start generating LLM embeddings until we create the LLM Option which brings us to the next step.

Create a new LLM Option.

Give the record a name, since this will be the default LLM Option I will call it default, and check the default box below the name field. If desired you can enter a name for the chatbot, and upload a custom avatar.

Be default apply guardrails is checked, for the purpose of this demo it will remain checked.

I will not be changing the custom instructions or the the initial greeting at this time.

Select the default LLM model, for this we will be using llama3:latest, I will select the other models as well.

In the Provisioning section select which admin roles that will use this option set. If there are any operator portals or splash/landing portals that should use this LLM option they can be selected at this time as well. If the goal is to allow anyone to access the chatbot without a login session check allow anonymous chats. Primarily this would be checked if you intend to have the chatbot on the splash portal before authentication.

Click create.

This will then start generating the LLM Embeddings to be used as resources for chat responses. The time it will take to generate the embeddings depends on hardware.

The Chat is now available for use in the admin gui.