> ## Documentation Index
> Fetch the complete documentation index at: https://runpod-b18f5ded-public-endpoints.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Google'S Gemma Model

This tutorial walks you through running Google's Gemma model using Runpod's vLLM Worker. Throughout this tutorial, you'll learn to set up a Serverless Endpoint with a gated large language model (LLM).

## Prerequisites

Before diving into the deployment process, gather the necessary tokens and accepting Google's terms. This step ensures that you have access to the model and are in compliance with usage policies.

* [Hugging Face access token](https://huggingface.co/settings/tokens)
* [Accepting Google's terms of service](https://huggingface.co/google/gemma-7b)

The next section will guide you through Setting up your Serverless Endpoint with Runpod.

## Get started

To begin, we'll deploy a vLLM Worker as a Serverless Endpoint. Runpod simplifies the process of running large language models, offering an alternative to the more complex Docker and Kubernetes deployment methods.

Follow these steps in the Runpod Serverless console to create your Endpoint.

1. Log in to the [Runpod Serverless console](https://www.console.runpod.io/serverless).

2. Select **+ New Endpoint**.

3. Provide the following:

   i. Endpoint name.

   ii. Select a GPU.

   iii. Configure the number of Workers.

   iv. (optional) Select **FlashBoot**.

   v. Enter the vLLM Worker image: `runpod/worker-vllm:stable-cuda11.8.0` or `runpod/worker-vllm:stable-cuda12.1.0`.

   vi. Specify enough storage for your model.

   vii. Add the following environment variables:

   a. `MODEL_NAME`: `google/gemma-7b-it`.

   b. `HF_TOKEN`: your Hugging Face API token for private models.

4. Select **Deploy**.

Once the Endpoint initializes, you can send a request to your [Endpoint](/serverless/endpoints/send-requests). You've now successfully deployed your model, a significant milestone in utilizing Google's Gemma model. As we move forward, the next section will focus on interacting with your model.

## Interact with your model

With the Endpoint up and running, it's time to leverage its capabilities by sending requests to interact with the model. This section demonstrates how to use OpenAI APIs to communicate with your model.

In this example, you'll create a Python chat bot using the `OpenAI` library; however, you can use any programming language and any library that supports HTTP requests.

Here's how to get started:

Use the `OpenAI` class to interact with the model. The `OpenAI` class takes the following parameters:

* `base_url`: The base URL of the Serverless Endpoint.
* `api_key`: Your Runpod API key.

```javascript theme={"system"}
from openai import OpenAI
import os

client = OpenAI(
    base_url=os.environ.get("RUNPOD_BASE_URL"),
    api_key=os.environ.get("RUNPOD_API_KEY"),
)
```

<Info>
  Set your environment variables `RUNPOD_BASE_URL` and `RUNPOD_API_KEY` to your Runpod API key and base URL. Your `RUNPOD_BASE_URL` will be in the form of:

  ```
  https://api.runpod.ai/v2/${RUNPOD_ENDPOINT_ID}/openai/v1
  ```

  Where `${RUNPOD_ENDPOINT_ID}` is the ID of your Serverless Endpoint.
</Info>

Next, you can use the `client` to interact with the model. For example, you can use the `chat.completions.create` method to generate a response from the model.

Provide the following parameters to the `chat.completions.create` method:

* `model`: `The model name`.
* `messages`: A list of messages to send to the model.
* `max_tokens`: The maximum number of tokens to generate.
* `temperature`: The randomness of the generated text.
* `top_p`: The cumulative probability of the generated text.
* `max_tokens`: The maximum number of tokens to generate.

```python theme={"system"}
messages = [
    {
        "role": "assistant",
        "content": "Hello, I'm your assistant. How can I help you today?",
    }
]


def display_chat_history(messages):
    for message in messages:
        print(f"{message['role'].capitalize()}: {message['content']}")


def get_assistant_response(messages):
    r = client.chat.completions.create(
        model="google/gemma-7b-it",
        messages=[{"role": m["role"], "content": m["content"]} for m in messages],
        temperature=0.7,
        top_p=0.8,
        max_tokens=100,
    )
    response = r.choices[0].message.content
    return response


while True:
    display_chat_history(messages)

    prompt = input("User: ")
    messages.append({"role": "user", "content": prompt})

    response = get_assistant_response(messages)
    messages.append({"role": "assistant", "content": response})
```

Congratulations! You've successfully set up a Serverless Endpoint and interacted with Google's Gemma model. This tutorial has shown you the essentials of deploying a model on Runpod and creating a simple application to communicate with it. You've taken important steps towards integrating large language models into your projects.