In this tutorial, you will learn how to integrate Google's Gemini AI Model into R. Google AI has an official Python package and documentation for the Gemini API but R users need not feel let down by the lack of official documentation for this API as this tutorial will provide you with the necessary information to get started using Gemini API in R.
Steps to Integrate Gemini into R
To use Google's Gemini via API in R, please follow these steps.
- Step 1 : Get API Key -
You can access the Gemini API by visiting this link : Google AI Studio. Once you have access, you can create an API key by clicking on
Create API Key
button. Copy and save your API key for future reference.Please note that the Gemini API is currently available for free. In the future, there may be a cost involved in using the Gemini API. Check out the pricing page here.
- Step 2 : Install the Required Libraries -
Before we can start using Gemini AI Model in R, we need to install the necessary libraries. The two libraries we will be using are
httr
andjsonlite
. The "httr" library allows us to post our question and fetch response with Gemini API, while the "jsonlite" library helps to convert R object to JSON format.To install these libraries, you can use the following code in R:
install.packages("httr") install.packages("jsonlite")
- Step 3 : Generate Content Based on Prompt -
The following function generates a response from the Gemini Model based on your question (prompt).
library(httr) library(jsonlite) # Function gemini <- function(prompt, temperature=0.5, max_output_tokens=1024, api_key=Sys.getenv("GEMINI_API_KEY"), model = "gemini-1.5-flash-latest") { if(nchar(api_key)<1) { api_key <- readline("Paste your API key here: ") Sys.setenv(GEMINI_API_KEY = api_key) } model_query <- paste0(model, ":generateContent") response <- POST( url = paste0("https://generativelanguage.googleapis.com/v1beta/models/", model_query), query = list(key = api_key), content_type_json(), encode = "json", body = list( contents = list( parts = list( list(text = prompt) )), generationConfig = list( temperature = temperature, maxOutputTokens = max_output_tokens ) ) ) if(response$status_code>200) { stop(paste("Error - ", content(response)$error$message)) } candidates <- content(response)$candidates outputs <- unlist(lapply(candidates, function(candidate) candidate$content$parts)) return(outputs) } prompt <- "R code to remove duplicates using dplyr." cat(gemini(prompt))
When you run the function above first time, it will ask you to enter your API Key. It will save the API Key in
GEMINI_API_KEY
environment variable so it won't ask for API key when you run the function next time.Sys.setenv( )
is to store API Key whereasSys.getenv( )
is to pull the stored API Key.
```r
# Remove duplicate rows from a data frame
df <- data.frame(
x = c(1, 2, 3, 4, 5, 1, 2, 3, 4, 5),
y = c("a", "b", "c", "d", "e", "a", "b", "c", "d", "e")
)
# Using `distinct()`
df %>% distinct()
```
- Prompt: Prompt means a question you want to ask. It is also called search query.
- Tokens: Tokens are subwords or words. For example, the word "lower" splits into two tokens: "low" and "er".
- Temperature: It is the model parameter which is used to fine tune the response. It lies between 0 and 1. If you set value of temperature close to 0, it means model to generate response which has highest probability. A value closer to 1 will produce responses that are more creative or random.
- Max output tokens: It is the model parameter which defines the maximum number of tokens that can be generated in the response.
How to Handle Image as Input
To handle image as input, we can use the gemini-1.5-flash-latest
model. It helps to describe image. You can ask any question related to the image. Make sure to install "base64enc" library.
library(httr) library(jsonlite) library(base64enc) # Function gemini_vision <- function(prompt, image, temperature=0.5, max_output_tokens=4096, api_key=Sys.getenv("GEMINI_API_KEY"), model = "gemini-1.5-flash-latest") { if(nchar(api_key)<1) { api_key <- readline("Paste your API key here: ") Sys.setenv(GEMINI_API_KEY = api_key) } model_query <- paste0(model, ":generateContent") response <- POST( url = paste0("https://generativelanguage.googleapis.com/v1beta/models/", model_query), query = list(key = api_key), content_type_json(), encode = "json", body = list( contents = list( parts = list( list( text = prompt ), list( inlineData = list( mimeType = "image/png", data = base64encode(image) ) ) ) ), generationConfig = list( temperature = temperature, maxOutputTokens = max_output_tokens ) ) ) if(response$status_code>200) { stop(paste("Error - ", content(response)$error$message)) } candidates <- content(response)$candidates outputs <- unlist(lapply(candidates, function(candidate) candidate$content$parts)) return(outputs) } gemini_vision(prompt = "Describe what people are doing in this image", image = "https://upload.wikimedia.org/wikipedia/commons/a/a7/Soccer-1490541_960_720.jpg")
Shiny App to Explain Image using Gemini API
You can create an interactive Shiny app that describes an image uploaded by user.
library(shiny) Sys.setenv(GEMINI_API_KEY = "xxxxxxxxxxx") ui <- fluidPage( mainPanel( fluidRow( fileInput( inputId = "imgFile", label = "Select image to upload", ), textInput( inputId = "prompt", label = "Prompt", placeholder = "Enter Your Query" ), actionButton("submit", "Talk to Gemini"), textOutput("response") ), imageOutput(outputId = "myimage") ) ) server <- function(input, output) { observeEvent(input$imgFile, { path <- input$imgFile$datapath output$myimage <- renderImage({ list( src = path ) }, deleteFile = FALSE) }) observeEvent(input$submit, { output$response <- renderText({ gemini_vision(input$prompt, input$imgFile$datapath) }) }) } shinyApp(ui = ui, server = server)
Question and Answering
Suppose you have some documents. You want to make a system where people can ask questions about these documents and a chatbot will answer based on what they ask.
make_prompt <- function(query, relevant_passage) { escaped <- gsub("'", "", gsub('"', "", gsub("\n", " ", relevant_passage))) prompt <- sprintf("You are a helpful and informative bot that answers questions using text from the reference passage included below. \ Be sure to respond in a complete sentence, being comprehensive, including all relevant background information. \ However, you are talking to a non-technical audience, so be sure to break down complicated concepts and \ strike a friendly and conversational tone. \ If the passage is irrelevant to the answer, you may ignore it. QUESTION: '%s' PASSAGE: '%s' ANSWER: ", query, escaped) return(prompt) } passage <- "Title: Is AI a Threat to Content Writers?\n Author: Deepanshu Bhalla\nFull article:\n Both Open source and commercial generative AI models have made content writing easy and quick. Now you can create content in a few mins which used to take hours."
query <- "Who is the author of this article?" prompt = make_prompt(query, passage) cat(gemini(prompt))
Deepanshu Bhalla is the author of this article.
query <- "Summarize this article" prompt = make_prompt(query, passage) cat(gemini(prompt))
This article discusses whether AI is a threat to content writers. It argues that both open source and commercial generative AI models have made content writing easier and quicker, and that this could potentially put content writers out of a job.
Chat Functionality
There are many use cases where it is important for a Chatbot to remember your previous questions in order to answer subsequent questions. For example, if you ask a question like "What is 2+2?" and then follow up with the another question : "What is the square of it?", it should understand your query and respond accordingly.
library(httr) library(jsonlite) chat_gemini <- function(prompt, temperature=0.5, api_key=Sys.getenv("GEMINI_API_KEY"), model="gemini-1.5-flash-latest") { if(nchar(api_key)<1) { api_key <- readline("Paste your API key here: ") Sys.setenv(GEMINI_API_KEY = api_key) } model_query <- paste0(model, ":generateContent") # Add new message chatHistory <<- append(chatHistory, list(list(role = 'user', parts = list( list(text = prompt) )))) response <- POST( url = paste0("https://generativelanguage.googleapis.com/v1beta/models/", model_query), query = list(key = api_key), content_type_json(), body = toJSON(list( contents = chatHistory, generationConfig = list( temperature = temperature ) ), auto_unbox = T)) if(response$status_code>200) { chatHistory <<- chatHistory[-length(chatHistory)] stop(paste("Status Code - ", response$status_code)) } else { answer <- content(response)$candidates[[1]]$content$parts[[1]]$text chatHistory <<- append(chatHistory, list(list(role = 'model', parts = list(list(text = answer))))) } return(answer) }
chatHistory <- list() cat(chat_gemini("2+2")) cat(chat_gemini("square of it")) cat(chat_gemini("add 3 to result"))
> chatHistory <- list() > cat(chat_gemini(prompt="3+5")) 3+5 is equal to 8. > cat(chat_gemini(prompt="square of it")) The square of 8 is 64. > cat(chat_gemini(prompt="Add 2 to it")) 64 + 2 = 66.
How to Generate Text Embeddings
In this section, we will show you how to use the Gemini API to generate text embeddings. This will help you search through a list of documents and ask questions about a certain topic.
In the example below, we have three documents on AI. We want to identify the most relevant document based on a question about the AI topic.
embedding_gemini <- function(prompt, api_key=Sys.getenv("GEMINI_API_KEY"), model = "text-embedding-004") { if(nchar(api_key)<1) { api_key <- readline("Paste your API key here: ") Sys.setenv(GEMINI_API_KEY = api_key) } model_query <- paste0(model, ":embedContent") response <- POST( url = paste0("https://generativelanguage.googleapis.com/v1beta/models/", model_query), query = list(key = api_key), content_type_json(), encode = "json", body = list( model = paste0("models/",model), content = list( parts = list( list(text = prompt) )) ) ) if(response$status_code>200) { stop(paste("Status Code - ", response$status_code)) } return(unlist(content(response))) } DOCUMENT1 = "AI is like a smart helper in healthcare. It can find problems early by looking at lots of information, help doctors make plans just for you, and even make new medicines faster." DOCUMENT2 = "AI needs to be open and fair. Sometimes, it can learn things that aren't right. We need to be careful and make sure it's fair for everyone. If AI makes a mistake, someone needs to take responsibility." DOCUMENT3 = "AI is making school exciting. It can make learning fit you better, help teachers make fun lessons, and show when you need more help." df <- data.frame(Text = c(DOCUMENT1, DOCUMENT2, DOCUMENT3)) # Get the embeddings of each text embedding_out <- list() for(i in 1:nrow(df)) { result <- embedding_gemini(prompt = df[i,"Text"]) embedding_out[[i]] <- result } # Identify Most relevant document query <- "AI can generate misleading results many times." scores_query <- embedding_gemini(query) # Calculate the dot products dot_products <- sapply(embedding_out, function(x) sum(x * scores_query)) # Find the index of the maximum dot product to view the most relevant document idx <- which.max(dot_products) df$Text[idx]
[1] "AI needs to be open and fair. Sometimes, it can learn things that aren't right. We need to be careful and make sure it's fair for everyone. If AI makes a mistake, someone needs to take responsibility."
The following code returns all the AI models available through the Gemini API.
library(httr) library(jsonlite) models <- GET( url = "https://generativelanguage.googleapis.com/v1beta/models", query = list(key = Sys.getenv("GEMINI_API_KEY"))) lapply(content(models)[["models"]], function(model) c(description = model$description, displayName = model$displayName, name = model$name, method = model$supportedGenerationMethods[1]))
[[1]] [[1]]$description [1] "A legacy text-only model optimized for chat conversations" [[1]]$displayName [1] "PaLM 2 Chat (Legacy)" [[1]]$name [1] "models/chat-bison-001" [[1]]$method [1] "generateMessage" [[2]] [[2]]$description [1] "A legacy model that understands text and generates text as an output" [[2]]$displayName [1] "PaLM 2 (Legacy)" [[2]]$name [1] "models/text-bison-001" [[2]]$method [1] "generateText" [[3]] [[3]]$description [1] "Obtain a distributed representation of a text." [[3]]$displayName [1] "Embedding Gecko" [[3]]$name [1] "models/embedding-gecko-001" [[3]]$method [1] "embedText" [[4]] [[4]]$description [1] "The best model for scaling across a wide range of tasks. This is the latest model." [[4]]$displayName [1] "Gemini 1.0 Pro Latest" [[4]]$name [1] "models/gemini-1.0-pro-latest" [[4]]$method [1] "generateContent" [[5]] [[5]]$description [1] "The best model for scaling across a wide range of tasks" [[5]]$displayName [1] "Gemini 1.0 Pro" [[5]]$name [1] "models/gemini-1.0-pro" [[5]]$method [1] "generateContent" [[6]] [[6]]$description [1] "The best model for scaling across a wide range of tasks" [[6]]$displayName [1] "Gemini 1.0 Pro" [[6]]$name [1] "models/gemini-pro" [[6]]$method [1] "generateContent" [[7]] [[7]]$description [1] "The best model for scaling across a wide range of tasks. This is a stable model that supports tuning." [[7]]$displayName [1] "Gemini 1.0 Pro 001 (Tuning)" [[7]]$name [1] "models/gemini-1.0-pro-001" [[7]]$method [1] "generateContent" [[8]] [[8]]$description [1] "The best image understanding model to handle a broad range of applications" [[8]]$displayName [1] "Gemini 1.0 Pro Vision" [[8]]$name [1] "models/gemini-1.0-pro-vision-latest" [[8]]$method [1] "generateContent" [[9]] [[9]]$description [1] "The best image understanding model to handle a broad range of applications" [[9]]$displayName [1] "Gemini 1.0 Pro Vision" [[9]]$name [1] "models/gemini-pro-vision" [[9]]$method [1] "generateContent" [[10]] [[10]]$description [1] "Mid-size multimodal model that supports up to 2 million tokens" [[10]]$displayName [1] "Gemini 1.5 Pro Latest" [[10]]$name [1] "models/gemini-1.5-pro-latest" [[10]]$method [1] "generateContent" [[11]] [[11]]$description [1] "Mid-size multimodal model that supports up to 2 million tokens" [[11]]$displayName [1] "Gemini 1.5 Pro 001" [[11]]$name [1] "models/gemini-1.5-pro-001" [[11]]$method [1] "generateContent" [[12]] [[12]]$description [1] "Mid-size multimodal model that supports up to 2 million tokens" [[12]]$displayName [1] "Gemini 1.5 Pro" [[12]]$name [1] "models/gemini-1.5-pro" [[12]]$method [1] "generateContent" [[13]] [[13]]$description [1] "Mid-size multimodal model that supports up to 2 million tokens" [[13]]$displayName [1] "Gemini 1.5 Pro Experimental 0801" [[13]]$name [1] "models/gemini-1.5-pro-exp-0801" [[13]]$method [1] "generateContent" [[14]] [[14]]$description [1] "Mid-size multimodal model that supports up to 2 million tokens" [[14]]$displayName [1] "Gemini 1.5 Pro Experimental 0827" [[14]]$name [1] "models/gemini-1.5-pro-exp-0827" [[14]]$method [1] "generateContent" [[15]] [[15]]$description [1] "Fast and versatile multimodal model for scaling across diverse tasks" [[15]]$displayName [1] "Gemini 1.5 Flash Latest" [[15]]$name [1] "models/gemini-1.5-flash-latest" [[15]]$method [1] "generateContent" [[16]] [[16]]$description [1] "Fast and versatile multimodal model for scaling across diverse tasks" [[16]]$displayName [1] "Gemini 1.5 Flash 001" [[16]]$name [1] "models/gemini-1.5-flash-001" [[16]]$method [1] "generateContent" [[17]] [[17]]$description [1] "Fast and versatile multimodal model for scaling across diverse tasks" [[17]]$displayName [1] "Gemini 1.5 Flash 001 Tuning" [[17]]$name [1] "models/gemini-1.5-flash-001-tuning" [[17]]$method [1] "generateContent" [[18]] [[18]]$description [1] "Fast and versatile multimodal model for scaling across diverse tasks" [[18]]$displayName [1] "Gemini 1.5 Flash" [[18]]$name [1] "models/gemini-1.5-flash" [[18]]$method [1] "generateContent" [[19]] [[19]]$description [1] "Fast and versatile multimodal model for scaling across diverse tasks" [[19]]$displayName [1] "Gemini 1.5 Flash Experimental 0827" [[19]]$name [1] "models/gemini-1.5-flash-exp-0827" [[19]]$method [1] "generateContent" [[20]] [[20]]$description [1] "Fast and versatile multimodal model for scaling across diverse tasks" [[20]]$displayName [1] "Gemini 1.5 Flash 8B Experimental 0827" [[20]]$name [1] "models/gemini-1.5-flash-8b-exp-0827" [[20]]$method [1] "generateContent" [[21]] [[21]]$description [1] "Obtain a distributed representation of a text." [[21]]$displayName [1] "Embedding 001" [[21]]$name [1] "models/embedding-001" [[21]]$method [1] "embedContent" [[22]] [[22]]$description [1] "Obtain a distributed representation of a text." [[22]]$displayName [1] "Text Embedding 004" [[22]]$name [1] "models/text-embedding-004" [[22]]$method [1] "embedContent" [[23]] [[23]]$description [1] "Model trained to return answers to questions that are grounded in provided sources, along with estimating answerable probability." [[23]]$displayName [1] "Model that performs Attributed Question Answering." [[23]]$name [1] "models/aqa" [[23]]$method [1] "generateAnswer"
Share Share Tweet