Build a Voice Powered Virtual Assistant in AppSheet with Gemini

Manual mobile data entry is a bottleneck to productivity. Learn how to replace slow, error-prone tapping with an intelligent, voice-powered virtual assistant.

Introduction: The End of Manual Mobile Data Entry

In the world of enterprise applications, the final frontier is often the “last mile” of data collection—the point where a frontline worker interacts with a mobile device. For decades, this interaction has been defined by tapping on small glass screens, a process that is slow, error-prone, and fundamentally at odds with the dynamic environments where real work happens. We’ve optimized databases, streamlined cloud infrastructure, and automated backend processes, yet we still ask our most active employees to stop what they’re doing and type. This paradigm is a bottleneck to productivity and a source of poor-quality data.

This article introduces a new paradigm. We will move beyond simple data entry and build an intelligent, voice-powered virtual assistant directly inside an AI-Powered Invoice Processor application. By harnessing the power of large language models, we can transform natural language commands into structured data, effectively eliminating the need for manual typing. Prepare to build an application that doesn’t just listen but understands and acts.

The Challenge: Frontline Workers and Inefficient Typing

Imagine a field service technician inspecting equipment, a warehouse manager conducting an inventory count, or a construction site supervisor logging a safety incident. These are not desk jobs. These frontline workers operate in environments where their hands are occupied, they might be wearing gloves, and their attention needs to be on the physical task at hand, not on a smartphone screen.

Forcing them to use a traditional mobile app introduces significant friction:

• Context Switching: The mental and physical effort of stopping a task, removing gloves, pulling out a device, navigating to the right screen, and typing information breaks concentration and slows down operations.

• Data Entry Errors: Small keyboards and “fat-finger” mistakes lead to typos, incomplete records, and inconsistent data, which downstream processes and analytics must then contend with.

• Reduced Adoption: When an application is cumbersome to use in the field, workers will often resort to workarounds—jotting notes on paper to be entered later (or never), leading to data latency and loss.

This inefficiency is not a minor inconvenience; it’s a critical operational drag that impacts data integrity, employee productivity, and the overall agility of the business.

The Solution: A Voice-to-Action Virtual Assistant in AMA Patient Referral and Anesthesia Management System

The solution is not merely voice-to-text dictation, which still requires the user to navigate to the correct field before speaking. We are building a true voice-to-action system. This is a conversational interface where the user can state their intent in a single, natural phrase, and the application intelligently executes the entire task.

Consider the difference:

• Old Way: Tap to open the app. Tap to navigate to the “New Incident” form. Tap the “Description” field. Type “Broken ladder”. Tap the “Location” field. Type “Warehouse B”. Tap “Save”.

• New Way: Tap a single microphone button. Say, “Log a new safety issue for a broken ladder in warehouse B.” The system processes the command, populates the correct fields with the extracted information, and creates the record automatically.

This hands-free, conversational approach is a game-changer. It allows workers to capture high-quality, structured data in real-time without ever breaking their workflow. By embedding this capability within AppSheetway Connect Suite, we leverage a powerful low-code platform to build and deploy these sophisticated solutions at unprecedented speed.

Overview of the Tech Stack: OSD App Clinical Trial Management, Gemini, and [AI Powered Cover Letter [Automated Job Creation in Real Time Jobber and Google Sheets Integration from Gmail](https://votuduc.com/Automated-Job-Creation-in-Jobber-from-Gmail-p115606) Engine](https://votuduc.com/AI-Powered-Cover-Letter-Automated Quote Generation and Delivery System for Jobber-Engine-p111092)

This powerful solution is made possible by the seamless integration of three core Google Cloud technologies, each playing a distinct and vital role.

• AppSheet: This is our application front-end and data management layer. As a premier no-code/low-code platform, AppSheet allows us to rapidly build the mobile user interface, define the data structure (using [Automated Web Scraping with [Multilingual Text-to-Speech Tool with SocialSheet Streamline Your Social Media Posting 123](https://votuduc.com/Multilingual-Text-to-Speech-Tool-with-Google-Workspace-p809282)](https://votuduc.com/Automated-Web-Scraping-with-Google-Sheets-p292968) as our database), and manage user access. It will provide the microphone button that captures the user’s voice and initiates the workflow by calling a webhook.

• Gemini: This is the AI “brain” of our operation. We will use the Gemini Pro model via its API. Its role is to perform Natural Language Understanding (NLU). It will receive the raw text of the user’s command (e.g., “Add a new task to fix the conveyor belt in sector 4”) and, based on our instructions, extract the key entities and return them as structured JSON data (e.g., {"action": "new_task", "item": "conveyor belt", "location": "sector 4"}).

• Genesis Engine AI Powered Content to Video Production Pipeline: This is the serverless “connective tissue” that orchestrates the entire process. We will deploy a simple Apps Script project as a web app, which serves as the secure webhook endpoint for AppSheet. This script will receive the text from AppSheet, format and send the request to the Gemini API, parse the structured JSON response from Gemini, and then use the AppSheet API to insert the new, structured data directly back into our application’s data source.

Architecting the Voice-Powered Workflow

Before we start wiring things together and writing code, let’s zoom out and understand the architecture. A solid plan is the difference between a robust, scalable system and a brittle one that breaks with the slightest change. Our goal is to create a seamless flow from a user’s spoken command in AppSheet to a concrete action in our database, with Google’s ecosystem acting as the connective tissue.

High-Level System Diagram: From Voice to Database Action

At its core, this is an event-driven system. The “event” is a user saving a new voice recording. Everything else is a reaction to that initial event. Think of it as a digital Rube Goldberg machine where each component triggers the next in a precise sequence.

Here’s the journey our data will take:

1. User Input (AppSheet): The user records a voice command and saves it within the AppSheet app.

2. File Storage & Trigger (Google Drive): AppSheet automatically uploads the audio file to a designated folder in Google Drive. This file drop is the crucial trigger for our entire backend process.

3. Orchestration ([Architecting Multi Tenant AI Workflows in Building Modular Agentic Apps Script with Gemini Function Calling](https://votuduc.com/architecting-multi-tenant-ai-workflows-in-google-apps-script-p-20260321290501)): A script, listening for new files in that specific Drive folder, wakes up. It grabs the new audio file.

4. AI Processing (Gemini API): The Apps Script sends the audio data to the Gemini API. Gemini performs two key tasks: transcribing the audio to text and then analyzing that text to determine the user’s intent and extract key information.

5. Action Execution (AppSheet API): Armed with structured data from Gemini (e.g., “add a task named ‘Buy milk’”), the Apps Script makes a call to the AppSheet API to add, update, or delete a record in the corresponding table.

6. Data Sync (AppSheet & Database): The AppSheet platform processes the API call, updates the underlying data source (e.g., Google Sheets), and the change is reflected back in the user’s app.

Visually, the flow looks like this:

User Voice -> AppSheet UI -> Google Drive (File Drop) -> Apps Script (Trigger) -> Gemini API (Process) -> Apps Script (Parse) -> AppSheet API (Action) -> Database

Now, let’s break down the role of each component in this chain.

Step 1: Capturing Audio within the AppSheet Interface

This is our user’s entry point and, thankfully, the most straightforward part of the setup. AppSheet has native support for capturing various file types, including audio.

We’ll configure a table with a column of type File. When a user interacts with a form for this table, AppSheet presents a native interface to either upload a file or create a new one. For our use case, the user will tap “New,” which opens the device’s default voice memo recorder. They speak their command—“Add a new project for the Acme client due next Friday”—and hit save.

Upon saving the form, AppSheet handles the magic of packaging this recording (often as an .m4a file) and uploading it directly to your app’s default data folder in Google Drive. The path usually looks something like /AppSheet/data/YourAppName-12345/Files/. This automatic, reliable file placement is the lynchpin of our entire workflow.

Step 2: Storing and Triggering from Google Drive

Google Drive is more than just a cloud file cabinet for our audio clips; it’s our event bus. We are leveraging a specific folder as a “hot folder.” Any file that lands here is considered a new command to be processed.

This is where the handoff from the no-code world of AppSheet to our custom-coded backend begins. We will set up a trigger in Automating Technical Debt Audits in Apps Script with AI Agents that specifically monitors this folder. When AppSheet completes its upload and a new audio file appears, Google Drive emits a signal. Our Apps Script is configured to listen for this exact signal (an on change or create event), which immediately executes our main processing function, passing along information about the newly created file.

Step 3: Orchestrating Logic with Google Apps Script

Think of Google Apps Script as the central nervous system of this operation. It’s the “glue” that connects the user-facing app (AppSheet) with the powerful AI brain (Gemini). It’s a serverless JavaScript platform that lives within the Google ecosystem, giving it privileged and easy access to services like Drive, Sheets, and external APIs.

Our script will be responsible for:

1. Receiving the Trigger: It activates the moment a new file is detected in our target Drive folder.

2. Reading the File: It uses the event information to identify and read the raw binary data of the audio file.

3. Calling the Gemini API: It formats the audio data into a valid request and sends it to the Gemini API endpoint for processing.

4. Parsing the Response: It receives the structured JSON response from Gemini, which contains the transcription and, more importantly, the identified intent and entities.

5. Constructing the AppSheet API Call: Based on the parsed intent, it builds the correct request for the AppSheet API. For example, if the intent is ADD_ROW, it will format a JSON payload with the column values extracted from the voice command.

6. Executing the Action: It sends this request to the AppSheet API, effectively telling the app to perform a database operation on the user’s behalf.

Step 4: Leveraging Gemini for Transcription and Intent Recognition

This is where the real intelligence of our virtual assistant comes from. We aren’t just doing simple speech-to-text. We are performing Natural Language Understanding (NLU) to turn ambiguous human language into structured, machine-readable commands. Gemini will handle two critical functions in a single API call.

1. Transcription: First, the model processes the raw audio bytes sent by our Apps Script and converts them into a text string. The raw output is simply a transcript: "Add a new project for the Acme client due next Friday".

2. Intent Recognition & Entity Extraction: This is the game-changer. We won’t just ask Gemini for the transcript; we will provide it with a carefully engineered prompt that asks it to act as an NLU engine. We’ll instruct it to analyze the transcript and return a structured JSON object that identifies two things:

• Intent: The overall goal of the user’s command. For example, CREATE_PROJECT, UPDATE_TASK_STATUS, or QUERY_INVENTORY.

• Entities: The specific pieces of data, or parameters, needed to fulfill that intent. For the example above, the entities would be project_name: "Acme client" and due_date: "next Friday".

The final output from Gemini that our Apps Script receives won’t be a simple string, but rather a clean JSON object like this:

{

"intent": "CREATE_PROJECT",

"entities": {

"projectName": "Acme client",

"dueDate": "2023-10-27"

}

}

This structured data is unambiguous. It eliminates the guesswork and allows our Apps Script to reliably know exactly which AppSheet API endpoint to call and what data to send.

Prerequisites and Initial Setup

Before we dive into the magic of AI, we need to lay the essential groundwork. This setup process creates the necessary connections between our user-facing AppSheet app, the powerful Gemini model in Google Cloud, and the Google Apps Script that will act as our intelligent intermediary. Let’s get our digital hands dirty.

Configuring Your AppSheet App for Audio Input

First, we need a way for our users to speak to the assistant. In AppSheet, the most straightforward way to capture audio is by using the File column type. When a user interacts with this column on a mobile device, AppSheet will natively provide an option to record audio directly from the microphone.

Here’s how to set it up:

1. Navigate to Your Data Table: In the AppSheet editor, go to the Data section and select the table you’ll use to manage interactions (e.g., a table named Prompts).

2. Add a New Column: In the Columns tab for your selected table, click + Add Column.

3. Define the Audio Column:

• Column Name: Give it a clear name, like AudioInput.

• Type: This is the most important step. Change the column type to File.

• File Folder Path (Recommended): To keep your Google Drive organized, specify a folder path where the audio recordings will be stored. For example, you can enter AudioRecordings/. AppSheet will automatically create this folder in your app’s default data directory if it doesn’t exist.

Step-by-Step Implementation Guide

With the architecture defined, it’s time to dive into the code. This guide breaks down the process into four distinct parts, starting from the moment AppSheet saves an audio file to the final update in your data source. We’ll be working primarily within a single Google Apps Script function that will act as the intelligent backend for our AppSheet application.

Part 1: Writing the Apps Script Function to Process Audio Files

Our journey begins in Google Apps Script. The first task is to create a function that can be called by AppSheet, receive information about the recorded audio file, and prepare it for the Gemini API. AppSheet typically saves audio recordings to your Google Drive, and our script will need to access this file.

The core steps are:

1. Create a web app function (like doPost(e)) that can be triggered by an AppSheet Automated Work Order Processing for UPS.

2. Extract the file ID of the audio recording from the request sent by AppSheet.

3. Use the DriveApp service to locate and read the audio file.

4. Encode the audio file’s content into a Base64 string, which is the format required by the Gemini API for file data.

Let’s create the initial structure of our script.

// This function will be the endpoint for our AppSheet webhook.

function doPost(e) {

// 1. Parse the incoming request from AppSheet

const requestBody = JSON.parse(e.postData.contents);

const audioFileId = requestBody.fileId; // We'll configure AppSheet to send the file ID here.

const userEmail = requestBody.userEmail; // It's good practice to pass the user context.

if (!audioFileId) {

return ContentService.createTextOutput(JSON.stringify({ "status": "error", "message": "File ID not provided." })).setMimeType(ContentService.MimeType.JSON);

}

try {

// 2. Access the audio file from Google Drive

const audioFile = DriveApp.getFileById(audioFileId);

const audioBytes = audioFile.getBlob().getBytes();

// 3. Encode the audio file to Base64

const audioBase64 = Utilities.encodeBase64(audioBytes);

const audioMimeType = audioFile.getMimeType();

// In the next parts, we will pass audioBase64 and audioMimeType to the Gemini API.

// For now, let's just confirm we've processed it.

Logger.log(`Successfully processed file ${audioFile.getName()} with MIME type ${audioMimeType}`);

// Placeholder for the next steps

// const geminiResponse = callGeminiAPI(audioBase64, audioMimeType);

// updateDataSource(geminiResponse);

return ContentService.createTextOutput(JSON.stringify({ "status": "success", "message": "Audio processed." })).setMimeType(ContentService.MimeType.JSON);

} catch (error) {

Logger.log(`Error processing audio file: ${error.toString()}`);

return ContentService.createTextOutput(JSON.stringify({ "status": "error", "message": `Failed to process audio file: ${error.toString()}` })).setMimeType(ContentService.MimeType.JSON);

}

}

This snippet provides a robust starting point. It handles incoming requests, validates the input, and uses a try...catch block for error handling. The key takeaway is the conversion of the raw audio file into a Base64 encoded string, preparing it for a web request.

Part 2: Authenticating and Calling the Gemini API

Now that we have the audio data in the correct format, we can make the call to the Gemini API. This involves constructing a proper HTTP POST request using Google Apps Script’s UrlFetchApp service.

Before you begin: You must obtain an API key for the Gemini API. You can generate one from Google AI Studio. As a best practice, we’ll store this key in Apps Script’s “Project Properties” to avoid hardcoding it directly in our script.

1. In the Apps Script editor, go to Project Settings (the gear icon ⚙️).

2. Scroll down to Script Properties and add a new property.

3. Set the Property name to GEMINI_API_KEY and the Value to the key you just generated.

The following code demonstrates how to build and send the request to the Gemini 1.5 Pro model, which is excellent at understanding context from various modalities, including audio.

function callGeminiAPI(audioBase64, audioMimeType, prompt) {

// 1. Retrieve the API key securely from Project Properties

const API_KEY = PropertiesService.getScriptProperties().getProperty('GEMINI_API_KEY');

const API_URL = `https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-latest:generateContent?key=${API_KEY}`;

// 2. Construct the request payload

// The payload contains our prompt and the audio data.

const payload = {

"contents": [

{

"parts": [

{ "text": prompt }, // The instructional prompt (covered in Part 3)

{

"inlineData": {

"mimeType": audioMimeType,

"data": audioBase64

}

}

]

}

],

"generationConfig": {

"responseMimeType": "application/json", // We explicitly ask for a JSON response

"temperature": 0.2, // Lower temperature for more deterministic, structured output

"maxOutputTokens": 2048

}

};

// 3. Configure the HTTP request options

const options = {

'method': 'post',

'contentType': 'application/json',

'payload': JSON.stringify(payload),

'muteHttpExceptions': true // Important for capturing detailed error messages

};

// 4. Execute the API call

try {

const response = UrlFetchApp.fetch(API_URL, options);

const responseCode = response.getResponseCode();

const responseBody = response.getContentText();

if (responseCode === 200) {

// The response from Gemini is nested, so we need to parse it to get to the core content.

const parsedData = JSON.parse(responseBody);

const geminiContent = parsedData.candidates[0].content.parts[0].text;

return JSON.parse(geminiContent); // We expect the content itself to be a JSON string

} else {

Logger.log(`Gemini API Error: ${responseCode} - ${responseBody}`);

throw new Error(`Gemini API request failed with status ${responseCode}.`);

}

} catch (error) {

Logger.log(`Failed to call Gemini API: ${error.toString()}`);

throw error; // Re-throw the error to be caught by the main function

}

}

This function is the heart of our integration. It securely handles the API key, builds a precise payload with our audio data and a text prompt, and makes the call. Critically, we’ve set responseMimeType to application/json and muteHttpExceptions to true for more predictable and debuggable behavior.

Part 3: [Prompt Engineering for Reliable Autonomous Workspace Agents for Reliable Autonomous Workspace Agents](https://votuduc.com/prompt-engineering-for-reliable-autonomous-workspace-agents-p-20260319404106) for Accurate Intent Extraction

Simply sending an audio file to Gemini is not enough. We need to guide the model to understand the context of our application and provide a response in a structured format that our script can easily parse. This is where prompt engineering becomes essential.

Our prompt is the text instruction we send along with the audio. A well-crafted prompt ensures we get back a clean, predictable JSON object, not just a conversational sentence.

Consider an inventory management app where a user might say, “Add fifteen kilograms of Fuji apples to the stock.” We want Gemini to extract three key pieces of information:

• Intent: add_stock

• Entities: itemName (Fuji apples) and quantity (15)

Here is a powerful, example-driven (few-shot) prompt that establishes the rules for the AI:

function getSystemPrompt() {

const prompt = `

You are an intelligent assistant for an inventory management application.

Your task is to analyze the user's voice command and extract their intent and the relevant entities.

The possible intents are:

- "add_stock": When the user wants to add items to the inventory.

- "remove_stock": When the user wants to remove items from inventory.

- "check_inventory": When the user wants to know the quantity of an item.

- "unknown": If the user's intent cannot be determined from the list above.

The possible entities are:

- "itemName": The name of the product.

- "quantity": The numerical quantity of the item.

You MUST respond ONLY with a valid JSON object. Do not include any other text, explanations, or markdown formatting.

Here are some examples:

User audio: "Please add 24 boxes of nitrile gloves."

Your JSON response:

{

"intent": "add_stock",

"entities": {

"itemName": "nitrile gloves",

"quantity": 24

}

}

User audio: "How many green widgets do we have left?"

Your JSON response:

{

"intent": "check_inventory",

"entities": {

"itemName": "green widgets",

"quantity": null

}

}

User audio: "Yeah, can you take out five of the large t-shirts."

Your JSON response:

{

"intent": "remove_stock",

"entities": {

"itemName": "large t-shirts",

"quantity": 5

}

}

User audio: "What's the weather like today?"

Your JSON response:

{

"intent": "unknown",

"entities": {}

}

Now, analyze the following user command.

`;

return prompt;

}

This prompt is effective because it:

1. Defines the Role: “You are an intelligent assistant for an inventory management application.”

2. Specifies the Task: “extract their intent and the relevant entities.”

3. Constrains the Output: “You MUST respond ONLY with a valid JSON object.” This is the most crucial instruction for automation.

4. Provides Examples (Few-Shot Learning): It shows the model exactly what a good input-output pair looks like, dramatically improving its accuracy and adherence to the desired format.

Part 4: Parsing the Gemini Response and Updating Your Data Source

The final step is to take the structured JSON response from Gemini and perform the corresponding action in our data source (e.g., a Google Sheet). This involves parsing the JSON, using a switch statement to route the logic based on the intent, and using the SpreadsheetApp service to manipulate the data.

Let’s integrate all the pieces into our final doPost(e) function.

// Assumes your Google Sheet has columns: "ItemName", "Quantity"

function doPost(e) {

const requestBody = JSON.parse(e.postData.contents);

const audioFileId = requestBody.fileId;

const userEmail = requestBody.userEmail;

if (!audioFileId) {

return createJsonResponse({ "status": "error", "message": "File ID not provided." });

}

try {

// Part 1: Process Audio

const audioFile = DriveApp.getFileById(audioFileId);

const audioBytes = audioFile.getBlob().getBytes();

const audioBase64 = Utilities.encodeBase64(audioBytes);

const audioMimeType = audioFile.getMimeType();

// Part 3: Get Prompt

const prompt = getSystemPrompt();

// Part 2: Call Gemini

const geminiResponse = callGeminiAPI(audioBase64, audioMimeType, prompt);

Logger.log(`Parsed Gemini Response: ${JSON.stringify(geminiResponse)}`);

// Part 4: Parse and Update Data Source

const intent = geminiResponse.intent;

const entities = geminiResponse.entities;

let resultMessage = "";

const ss = SpreadsheetApp.openById("YOUR_SPREADSHEET_ID"); // <-- IMPORTANT: Add your Sheet ID

const sheet = ss.getSheetByName("Inventory"); // <-- IMPORTANT: Use your sheet name

const dataRange = sheet.getDataRange();

const values = dataRange.getValues();

switch (intent) {

case "add_stock":

if (entities.itemName && entities.quantity) {

resultMessage = updateStock(sheet, values, entities.itemName, entities.quantity);

} else {

resultMessage = "Could not add stock. Item name or quantity was missing.";

}

break;

case "remove_stock":

if (entities.itemName && entities.quantity) {

// Pass quantity as a negative number to subtract

resultMessage = updateStock(sheet, values, entities.itemName, -entities.quantity);

} else {

resultMessage = "Could not remove stock. Item name or quantity was missing.";

}

break;

case "check_inventory":

// This is a read-only action. You could implement logic to find and return the quantity.

resultMessage = "Inventory check functionality is not yet implemented.";

break;

default: // "unknown" or any other case

resultMessage = "Sorry, I could not understand your command. Please try again.";

break;

}

return createJsonResponse({ "status": "success", "message": resultMessage, "data": geminiResponse });

} catch (error) {

Logger.log(`Error in doPost: ${error.toString()}`);

return createJsonResponse({ "status": "error", "message": `An error occurred: ${error.toString()}` });

}

}

function updateStock(sheet, values, itemName, quantityChange) {

const itemNameLower = itemName.toLowerCase();

let itemFound = false;

// Start from row 2 (index 1) to skip header

for (let i = 1; i < values.length; i++) {

const currentItem = values[i][0].toString().toLowerCase(); // Assuming item name is in column A

if (currentItem === itemNameLower) {

const currentQuantity = parseInt(values[i][1]) || 0; // Assuming quantity is in column B

const newQuantity = currentQuantity + quantityChange;

sheet.getRange(i + 1, 2).setValue(newQuantity); // Update the quantity in column B

itemFound = true;

break;

}

}

if (!itemFound && quantityChange > 0) {

// If item not found and we are adding stock, create a new entry

sheet.appendRow([itemName, quantityChange]);

} else if (!itemFound) {

return `Could not update stock. Item "${itemName}" not found in inventory.`;

}

const action = quantityChange > 0 ? "Added" : "Removed";

return `${action} ${Math.abs(quantityChange)} for item "${itemName}".`;

}

// Helper function for consistent JSON responses

function createJsonResponse(obj) {

return ContentService.createTextOutput(JSON.stringify(obj)).setMimeType(ContentService.MimeType.JSON);

}

// (Include the callGeminiAPI and getSystemPrompt functions from above here)

This final script ties everything together. It follows the complete flow: receiving the file, calling Gemini with a structured prompt, and then using the model’s clean JSON output to perform a specific, predictable action on our Google Sheet. The helper function updateStock contains the business logic for finding and modifying inventory records, making the main function clean and readable.

Putting It to the Test: A Real-World Use Case

Theory and setup are one thing, but the real magic happens when we apply this voice-powered engine to a tangible business problem. Let’s move beyond the abstract and build a practical, high-value feature: updating inventory levels hands-free. This is a classic scenario where voice commands can dramatically boost efficiency for users on a warehouse floor, in a stockroom, or out in the field.

Example Scenario: Voice-Based Inventory Updates

Imagine a warehouse manager, Sarah. She’s doing a spot check on inventory, clipboard in one hand, scanner in the other. She notices a pallet of “Heavy-Duty Widgets” is running low. Traditionally, she’d have to put everything down, pull out her phone, navigate to the AppSheet app, find the product, and manually type in the new quantity. It’s a clunky, multi-step process that breaks her workflow.

With our new Gemini-powered voice assistant, the process is transformed. Sarah can simply tap a microphone icon in the app and say:

“Hey assistant, set the quantity for SKU HD-WID-001 to 25 units in the main warehouse.”

Or even more naturally:

“We’ve got 15 more blue widgets, add them to stock.”

The system should be smart enough to understand the intent (adding, setting, or removing stock) and the key entities (the quantity, the product identifier, and perhaps the location). It then executes the update in the background, allowing Sarah to continue her work uninterrupted. This is the power we’re about to unlock.

Code Snippets and Walkthrough

To make this happen, we’ll configure an [Architecting Autonomous Data Entry Apps with AppSheet and Building Self-Correcting Agentic Workflows with Vertex AI](https://votuduc.com/architecting-autonomous-data-entry-apps-with-appsheet-and-vertex-ai-p-20260322535129) that triggers when a voice command is captured, sends it to Gemini for processing, and then uses the structured output to update our database.

Assumptions:

• You have an Inventory table with columns like SKU (the key), ProductName, CurrentQuantity, and Location.

• You have a VoiceCommands table with at least two columns: CommandText (where the transcribed voice input is stored) and Status (to track the outcome). A new row in this table will trigger our automation.

Step 1: The Automation Bot & Gemini Prompt

Create a new Automation Bot that triggers on “Adds only” to the VoiceCommands table. The first step in your process will be a “Call a script” or “Call a webhook” task that sends the command to the Gemini API.

The most critical piece is the prompt you send. You need to instruct Gemini to act as a data parser and return a clean, predictable JSON object.

Task Configuration (Call a Webhook):

• URL: Your Google Cloud Function URL or other endpoint for the Gemini API.

• HTTP Verb: POST

• Body: This is where you craft the prompt.

{

"contents": [{

"parts": [{

"text": "You are a highly intelligent data extraction assistant for an inventory management application. Your task is to analyze the user's voice command and extract specific entities into a structured JSON format. The user's command is: '<<[CommandText]>>'.

From the command, extract the following four entities:

1.  'action': The user's intent. This must be one of three values: 'add', 'remove', or 'set'.

2.  'quantity': The numerical quantity mentioned.

3.  'sku': The product's Stock Keeping Unit (SKU).

4.  'location': The warehouse or storage location.

Rules:

- If an entity cannot be found in the command, its value in the JSON should be null.

- The 'quantity' must be a number, not a string.

- Return ONLY the raw JSON object and no other text, explanations, or markdown formatting.

Example Input: 'add 50 units of product ABC-123 to the back warehouse'

Example Output: {\"action\": \"add\", \"quantity\": 50, \"sku\": \"ABC-123\", \"location\": \"back warehouse\"}"

}]

}]

}

This detailed prompt “teaches” Gemini its role, defines the exact output structure we need, and provides examples, ensuring we get consistent results.

Step 2: Parsing the JSON Response

Gemini will return a JSON string in the body of the webhook response. We need to parse this in AppSheet to use the values. We’ll use a series of “Change data” steps or virtual columns to extract each piece of information.

Let’s assume the result of the webhook call is stored in a column called [GeminiResponse].

• ExtractedAction: INDEX(SPLIT(INDEX(SPLIT([GeminiResponse], "\"action\": \""), 2), "\""), 1)

• ExtractedQuantity: NUMBER(INDEX(SPLIT(INDEX(SPLIT([GeminiResponse], "\"quantity\": "), 2), ","), 1))

• ExtractedSKU: INDEX(SPLIT(INDEX(SPLIT([GeminiResponse], "\"sku\": \""), 2), "\""), 1)

Note: While these expressions work, they are sensitive to spacing. Future AppSheet functions for direct JSON parsing will simplify this immensely. For now, careful string splitting is the way.

Step 3: Conditional Logic and Data Updates

Now that we have our entities, we can act on them. Add a new step to your automation with a conditional branch (IF statement) based on the [ExtractedAction].

Condition: [ExtractedAction] = "add"

If true, run an action called “Add Stock.”

• Action Type: “Execute an action on a set of rows”

• Referenced Table: Inventory

• Referenced Rows: FILTER("Inventory", [SKU] = [_THISROW].[ExtractedSKU])

• Referenced Action: This should be a pre-existing action on the Inventory table that looks like this:

• Name: Update Stock Quantity

• Action: “Data: set the value of some columns in this row”

• Column: CurrentQuantity

• Value: [CurrentQuantity] + [ExtractedQuantity]

You would create similar conditional branches and actions for "remove" (subtracting the quantity) and "set" (setting the quantity directly to the new value).

Best Practices for Error Handling and User Feedback

A system that fails silently is a frustrating one. To create a robust and user-friendly assistant, we must anticipate issues and provide clear feedback.

Error Handling:

1. Parsing Failures: What if Gemini can’t understand the command or returns a malformed response? Your SPLIT() expressions will fail. Wrap your data update steps in a condition that first checks if the key values were extracted successfully.

• Example Condition: AND(ISNOTBLANK([ExtractedAction]), ISNOTBLANK([ExtractedSKU]), ISNOTBLANK([ExtractedQuantity]))

• If this condition is false, the automation should update the Status column of the VoiceCommands table to “Failed: Could not understand command.”

2. **Invalid Data: The user might mention a SKU that doesn’t exist. The FILTER() expression in your data update action will simply find zero rows to act upon. This is a silent failure. To fix this, add a step that checks if the SKU exists before trying to update it.

• Check Step: Use another IF condition: ISBLANK(FILTER("Inventory", [SKU] = [_THISROW].[ExtractedSKU]))

• If true, send a push notification to the user: “Error: SKU <<[ExtractedSKU]>> not found. Please try again.” and stop the bot.

User Feedback:

1. Confirmation is Key: Don’t leave the user guessing if their command worked. The final step in a successful automation branch should be to provide positive feedback.

• Push Notification: Send a notification that confirms the action. This creates a complete feedback loop. For example: "Success! Updated <<LOOKUP([ExtractedSKU], "Inventory", "SKU", "ProductName")>> quantity to <<...>>."

• UI Update: Change the Status of the command in the VoiceCommands table to “Completed.” This provides a visual log within the app of what has been processed.

2. Requesting Clarification: In a more advanced build, if Gemini returns null for a critical entity like quantity, you could have the automation set the command’s Status to “Needs Clarification” and trigger a UI view that asks the user to fill in the missing piece of information.

By implementing this use case with robust error handling and clear feedback, you elevate your AppSheet application from a simple data entry tool to a truly interactive and intelligent assistant.

Conclusion: The Future is Conversational

We’ve journeyed from a simple idea—voice control—to a tangible, working virtual assistant built on a no-code platform. What we’ve created is more than just a clever feature; it’s a glimpse into the future of application design and user interaction. The barrier between human intent and digital action is dissolving, replaced by the most natural interface of all: conversation. By embedding Gemini’s intelligence directly into the fabric of an AppSheet application, we’ve fundamentally changed the relationship with our data from one of manual entry to one of dynamic dialogue.

Recap: The Power of Integrating AI with No-Code Platforms

Let’s take a moment to appreciate the synergy we’ve just harnessed. AppSheet provided the robust, secure, and scalable framework—the bones of our application, connected directly to our data source. Gemini, via its API, provided the cognitive power—the brain that could understand natural language, parse intent, and structure unstructured speech into actionable data.

This combination is a game-changer for a few key reasons:

• Speed & Agility: Building a similar system from scratch would traditionally require a team of specialized developers, extensive coding, and months of work. We accomplished it in a fraction of the time, proving that sophisticated AI-powered solutions are no longer the exclusive domain of large tech corporations.

• Democratization of AI: No-code platforms have already empowered business users to build their own solutions. By integrating powerful models like Gemini, we are now democratizing access to artificial intelligence, allowing the people closest to the business problems to design and deploy intelligent solutions.

• Reliability Meets Flexibility: We get the best of both worlds. AppSheet ensures data integrity, user authentication, and process automation, while Gemini provides a fluid, human-centric front-end that adapts to the user, not the other way around.

Expanding the Possibilities Beyond Simple Data Entry

The voice assistant we built is a powerful proof of concept, but it’s merely the starting point. The true potential is unlocked when you start applying this conversational pattern to other business processes.

Imagine:

• A field service technician arriving on-site, hands full of tools, who can simply say, “Log my arrival at job site 75B, start the clock, and pull up the client’s service history.”

• A warehouse manager conducting a spot-check on inventory who can update stock levels by speaking into their device: “Reduce SKU 8801-B by three units and flag it for reorder.”

• An executive on the go who wants a quick briefing and asks their dashboard app, “What were our total sales in the West region last quarter, and how does that compare to the previous quarter?”

We are moving away from a world where applications are just digital forms and into an era where applications become active partners. They can understand context, anticipate needs, and execute complex, multi-step tasks based on simple, conversational commands. This shift dramatically reduces friction, minimizes training time, and boosts productivity by allowing users to operate at the speed of thought.

Ready to Scale Your Workspace Architecture

The final step is to think beyond a single application and consider this a new architectural layer for your entire digital workspace. The framework we’ve established can be a template. You can build a centralized “Gateway” app that acts as a conversational hub, capable of routing commands and interacting with multiple other AppSheet apps and data sources across your organization.

By doing so, you’re not just building apps; you’re architecting an intelligent, interconnected ecosystem. You are creating a workspace where technology conforms to human communication patterns, enabling a more intuitive, efficient, and ultimately more powerful way of working. The tools are here, the pattern is proven. Now it’s your turn to build the future.