Building a Scalable Multi Agent Orchestrator with GCP PubSub and Gemini

The impulse to build one all-powerful ‘god’ AI is a seductive trap that fails at scale. Discover why the future of robust AI systems lies in specialized agents, not monolithic oracles.

The Challenge of Scaling AI From Monoliths to Specialized Agents

The advent of powerful, general-purpose foundation models like Gemini has fundamentally altered the landscape of software development. The initial impulse is often to build a single, monolithic AI system—a “god” model—tasked with handling every conceivable user request. This approach is seductive in its simplicity: one prompt, one endpoint, one brain to rule them all. However, as anyone who has navigated the shift from monolithic applications to microservices can attest, this simplicity is a facade. When faced with the demands of production scale, complexity, and reliability, the monolithic AI begins to crack under its own weight.

Why a single ‘god’ AI model falls short

Treating a large language model (LLM) as a singular, all-knowing oracle is an anti-pattern for building robust, scalable systems. This approach, while functional for simple demos, introduces significant and often intractable problems in a real-world context.

• Context Window Bloat and Spiraling Costs: To be a jack-of-all-trades, a monolithic model requires an enormous and complex system prompt. This prompt must contain instructions, examples, and tool definitions for every possible task—from analyzing financial data to drafting marketing copy to generating code. This inflates the context window for every single call, leading to higher token consumption and, consequently, higher operational costs and increased latency. You pay the price for the model’s entire potential skillset, even when you only need a fraction of it.

• The “Master of None” Dilemma: A general-purpose prompt, by its very nature, dilutes focus. While a model like Gemini is incredibly capable, its performance on a specific, nuanced task (e.g., interpreting complex legal clauses) will almost always be surpassed by a model given a highly specialized prompt and fine-tuned set of tools for that exact domain. The “god” model provides “good enough” answers for many things but rarely the best answer for anything.

• A Brittle Maintenance Nightmare: How do you update the capabilities of a monolithic AI? Improving its code-generation logic might inadvertently degrade its creative writing performance. Adding a new tool for one function risks confusing the model when it’s attempting another. This entanglement makes iteration slow and risky. It’s the AI equivalent of redeploying an entire enterprise application to fix a typo on the contact page. The system becomes brittle, unpredictable, and a nightmare to debug and enhance.

The power of a multi-agent paradigm: Specialization and scalability

The solution lies in adopting a paradigm that software architecture has embraced for years: breaking down the monolith. We move from a single, overburdened model to a collaborative ecosystem of specialized Architecting AI Agents for the Google Workspace Marketplace. This is the microservices architecture for the generative AI era.

• Expertise through Specialization: Each agent is an expert in a narrow domain. A CodeAnalyzerAgent has a system prompt optimized for syntax, logic, and best practices. A DataQueryAgent is equipped with function-calling tools that interface directly with your databases and BI platforms. This specialization leads to higher accuracy, greater reliability, and more deterministic outcomes. Each agent does one thing and does it exceptionally well.

• Independent Scalability and Resilience: In a multi-agent system, you can scale resources intelligently. If your CustomerSupportAgent is experiencing a surge in traffic, you can scale its underlying infrastructure independently, without affecting the ReportGenerationAgent. This decoupling also builds resilience. The failure or degraded performance of one agent doesn’t cascade and bring down the entire system.

• Agility and Maintainability: Development teams can own, iterate on, and deploy their agents independently. The team responsible for data analysis can refine their agent’s prompts and tools without coordinating with the marketing team’s CreativeCopyAgent. This accelerates development cycles, simplifies testing, and lowers the risk associated with each change.

Introducing our architecture: A Router Agent and its team of Gemini experts

This is where theory meets practice. Our orchestrator is built on this multi-agent philosophy, using a simple yet powerful pattern: a central router directing tasks to a team of specialists.

At the heart of our system is the Router Agent. Its sole purpose is to act as an intelligent switchboard. It receives the initial, often ambiguous, user request and, using a specially crafted prompt and classification logic, determines the user’s core intent. It doesn’t answer the query itself; instead, it decides which specialist agent is best equipped to handle it.

This Router then dispatches the task to one of its Gemini-powered expert agents. While all are built on the same Gemini foundation, each is a unique specialist:

A* DataAnalysisAgent** might use Gemini 1.5 Pro for its massive context window, allowing it to analyze entire CSV files or reports in a single pass.

A* CodeGenerationAgent** could be configured with a specific set of function-calling tools that interface with a Git repository or a CI/CD pipeline.

A* CreativeWritingAgent** might have a system prompt fine-tuned for a specific brand voice and tone, ensuring all marketing copy is consistent.

The Router Agent makes its decision and publishes a message containing the user request and the target agent’s identity to a GCP Pub/Sub topic. This asynchronous handoff is the key to our system’s scalability and resilience, a topic we will dive into deeply in the sections to come. This architecture transforms our AI from a monolithic “know-it-all” into a highly efficient, scalable, and manageable digital workforce.

Solution Architecture: The GCP-Powered Orchestrator

To build a system that is both powerful and resilient, we can’t just stitch services together; we need a deliberate architecture. Our approach leans heavily on serverless, event-driven principles, creating a system that is cost-effective, infinitely scalable, and remarkably decoupled. Each component has a distinct role, working in asynchronous harmony to execute complex, multi-step tasks.

High-Level Diagram: Visualizing the task flow

Before we dive into the nuts and bolts of each service, let’s visualize the entire process. Think of our system not as a single, monolithic application, but as a dynamic, distributed network of specialized workers collaborating on a project.

[Image: A high-level architectural diagram showing the flow of tasks. An initial request enters a “Task Ingest” Pub/Sub topic. This triggers an “Orchestrator” Cloud Function, which analyzes the task and publishes a new, specific message to a “Worker” topic (e.g., code-generation-topic). A specialized “Worker” Cloud Function, subscribed to that topic, is triggered. It calls the Gemini API, performs its job, and publishes the result to a “Results” Pub/Sub topic. The Orchestrator function is also subscribed to the results, allowing it to process the output and dispatch the next task in the sequence.]

The flow, at its core, is a loop of communication and computation:

1. Initiation: A complex task (e.g., “Analyze this dataset, generate a JSON-to-Video Automated Rendering Engine script to visualize it, and then write a summary of the findings”) is submitted as a message to an initial Pub/Sub topic.

2. Dispatch: The central Orchestrator Agent (a Cloud Function) consumes this message. It breaks the task down, determines the very first step required (e.g., “Analyze this dataset”), and publishes a new, targeted message to a specific worker topic.

3. Execution: A specialized Worker Agent (another Cloud Function) subscribed to that topic is instantly triggered. It has one job. It takes the data from the message, formats a request to the Gemini API, and executes its specialized function.

4. Completion & Continuation: Upon receiving a response from Gemini, the worker agent publishes the result to a shared results topic. The Orchestrator Agent, listening on this topic, picks up the result, updates its state, and determines the next step (e.g., “Now, generate a Python script based on this analysis”). It then dispatches another message, and the cycle continues until the final goal is achieved.

GCP Pub/Sub: The asynchronous backbone for decoupling services

Google Cloud Pub/Sub is the heart of our architecture, acting as the central nervous system. It’s more than just a message queue; it’s a globally scalable, real-time messaging service that allows our various agents to communicate without ever knowing about each other’s existence. This is the principle of decoupling, and it’s our key to scalability and resilience.

Why Pub/Sub is critical:

• Service Decoupling: The Orchestrator doesn’t need the IP address or status of the CodeGenerationAgent. It simply publishes a “job request” to the code-generation-topic. Any service subscribed to that topic can pick it up. This means we can update, replace, or add multiple CodeGenerationAgent instances without ever touching the Orchestrator’s code.

• Durability and Resilience: What happens if our CodeGenerationAgent crashes? Without Pub/Sub, the request would be lost. With Pub/Sub, the message remains in the topic’s subscription until it is successfully processed and acknowledged. The system automatically handles retries, ensuring no task is dropped.

• Load Balancing and Scalability: As we publish more tasks, Pub/Sub effortlessly distributes the messages among all available subscribers (our Cloud Functions). If we experience a sudden spike in requests, Pub/Sub absorbs the load, and the serverless functions scale up to meet the demand. We don’t need to provision or manage a fleet of message brokers.

In essence, Pub/Sub allows each agent to focus solely on its task, shouting its results into a reliable, managed void, confident that the right component will hear it at the right time.

Cloud Functions: Serverless compute for reactive, event-driven agents

If Pub/Sub is the nervous system, Cloud Functions are the muscles that perform the actual work. Each “agent” in our system is implemented as a distinct, lightweight Cloud Function. This serverless compute model is a perfect match for our Architecting an Event-Driven Workspace with PubSub Firebase and Gemini.

The advantages of this model:

• **Event-Driven Triggers: Cloud Functions are designed to be reactive. We configure our CodeGenerationAgent function to trigger only when a new message is published to the code-generation-topic. It wakes up, does its job, and shuts down. It doesn’t consume resources while waiting for work.

• Pay-per-Invocation: This reactive nature leads to extreme cost efficiency. We are billed only for the milliseconds our code is actually running. For workflows that have idle periods, this is vastly cheaper than maintaining a fleet of always-on virtual machines.

• Focus on Logic, Not Infrastructure: The “serverless” promise is real. We provide the code for our agent—the logic for processing input and calling the Gemini API—and Google Cloud handles everything else: provisioning, patching, scaling, and security. This lets us focus on building intelligent agents, not managing infrastructure.

• Single Responsibility Principle: Each function can be small and dedicated to a single purpose. This makes our overall system easier to reason about, test, debug, and update. Modifying the DataAnalysisAgent has zero risk of breaking the CodeReviewAgent.

Gemini API: The specialized intelligence behind each worker

Finally, we arrive at the brain of each operation: the Gemini API. While Pub/Sub provides the communication channels and Cloud Functions provide the execution environment, Gemini provides the cognitive power that makes each agent intelligent.

This isn’t about simply plugging into a generic chatbot. The power of this architecture lies in using the Gemini API for specialized tasks:

• Model Specialization: The CodeGenerationAgent might be prompted to think step-by-step and produce clean, commented code. The SummarizationAgent might be given a completely different prompt focused on extracting key entities and condensing information. We can even choose different models (e.g., Gemini 1.5 Pro for its large context window, or a future, more specialized model) for different agents.

• Stateless, Contextual Execution: Each call to the Gemini API is stateless. The Cloud Function is responsible for packaging all necessary context—the user’s query, data from a previous step, specific instructions—into the prompt for each API call. This fits perfectly with the ephemeral nature of serverless functions and ensures that each agent’s task is self-contained and reproducible.

• Structured Data and Function Calling: Modern LLMs like Gemini excel at processing and generating structured data (like JSON). Our DataAnalysisAgent can ask Gemini to return its findings in a specific JSON format, which can then be easily parsed and passed to the next agent in the chain. This programmatic interaction is the foundation of reliable agentic workflows.

Implementation Step-by-Step Guide

Alright, let’s roll up our sleeves and build this thing. Theory is great, but code is better. We’ll use the gcloud CLI for infrastructure setup and Python for our Cloud Functions. Make sure you have the Google Cloud SDK installed and authenticated.

Step 1: Configuring the Pub/Sub Topics and Subscriptions

First, we need to lay the messaging groundwork. Our architecture relies on a few key Pub/Sub topics to decouple our agents.

1. ingress-topic: The single entry point for all new tasks or user requests.

2. router-topic: The topic where our Router Agent publishes its decisions. Worker agents will listen here.

3. results-topic: The topic where worker agents publish their final results.

4. dead-letter-topic: A crucial component for reliability. If a message fails processing multiple times, Pub/Sub will send it here for later inspection, preventing it from getting stuck in an infinite retry loop.

Let’s create them using gcloud:

# Set your project ID

export PROJECT_ID=$(gcloud config get-value project)

# Create the dead-letter topic first

gcloud pubsub topics create dead-letter-topic

# Create the main topics

gcloud pubsub topics create ingress-topic

gcloud pubsub topics create router-topic

gcloud pubsub topics create results-topic

Now for the subscriptions. This is where the magic of content-based routing happens. Our Router Agent will subscribe to the ingress-topic. Each worker, however, will subscribe to the same router-topic but with a unique filter. This filter tells Pub/Sub to only deliver messages with a specific attribute.

Let’s create a subscription for our Router Agent and one for a hypothetical code-generator-agent.

# Create a dead-letter policy for our subscriptions

gcloud pubsub dead-letter-policies create main-dlp \

--dead-letter-topic="projects/${PROJECT_ID}/topics/dead-letter-topic" \

--max-delivery-attempts=5

# Subscription for the Router Agent (listens to all new tasks)

gcloud pubsub subscriptions create router-agent-sub \

--topic=ingress-topic \

--dead-letter-policy=main-dlp

# Subscription for our specialized Code Generator Agent

# Note the --message-filter flag. This is the key to our routing.

gcloud pubsub subscriptions create code-generator-agent-sub \

--topic=router-topic \

--message-filter='attributes.agent_type="code-generator"' \

--dead-letter-policy=main-dlp

With this setup, when a message is published to router-topic with the attribute agent_type set to "code-generator", only the code-generator-agent-sub subscription will receive it. This is incredibly efficient and scalable.

Step 2: Building the ‘Router Agent’ Cloud Function

The Router Agent is the brain of our operation. It’s a Cloud Function that triggers on messages to ingress-topic, uses Gemini to classify the request, and then forwards it to the appropriate worker via the router-topic.

Here’s the Python code for a 2nd Gen Cloud Function.

main.py

import base64

import json

import os

import functions_framework

from google.cloud import pubsub_v1

import vertexai

from vertexai.generative_models import GenerativeModel, Part

# --- Configuration ---

PROJECT_ID = os.environ.get("GCP_PROJECT")

LOCATION = "us-central1"

ROUTER_TOPIC_ID = "router-topic"

# --- Initialize Clients ---

vertexai.init(project=PROJECT_ID, location=LOCATION)

publisher = pubsub_v1.PublisherClient()

model = GenerativeModel("gemini-1.5-flash-001")

router_topic_path = publisher.topic_path(PROJECT_ID, ROUTER_TOPIC_ID)

ROUTING_PROMPT = """

You are an intelligent routing agent. Your task is to analyze a user's request and determine which specialized agent should handle it.

Respond with only the name of the target agent. Your choices are:

- "code-generator": For requests involving writing, explaining, or debugging code.

- "text-summarizer": For requests to summarize long pieces of text.

- "data-analyzer": For requests involving data analysis, trends, or insights from datasets.

- "general-qa": For all other general questions.

User Request:

"{user_request}"

Target Agent:

"""

@functions_framework.cloud_event

def router_agent(cloud_event):

"""

Cloud Function triggered by a message on ingress-topic.

It classifies the request using Gemini and routes it to the correct worker.

"""

# 1. Decode the incoming message

try:

message_data_encoded = cloud_event.data["message"]["data"]

message_data_str = base64.b64decode(message_data_encoded).decode("utf-8")

payload = json.loads(message_data_str)

print(f"Received payload: {payload}")

user_request = payload.get("user_request")

if not user_request:

raise ValueError("Missing 'user_request' in payload")

except (KeyError, json.JSONDecodeError, ValueError) as e:

print(f"Error processing message: {e}")

# Acknowledge the message to prevent retries for malformed data

return

# 2. Use Gemini to classify the request

try:

prompt = ROUTING_PROMPT.format(user_request=user_request)

response = model.generate_content(prompt)

target_agent = response.text.strip()

print(f"Gemini classified request for agent: {target_agent}")

except Exception as e:

print(f"Error calling Gemini API: {e}")

# Nack the message by raising an exception, causing a retry

raise

# 3. Publish the message to the router-topic with the new attribute

try:

# Pass the original payload through, encoded as bytes

future = publisher.publish(

router_topic_path,

data=message_data_str.encode("utf-8"),

# This attribute is what the subscription filter uses!

agent_type=target_agent

)

message_id = future.result()

print(f"Published message {message_id} to {router_topic_path} for agent '{target_agent}'")

except Exception as e:

print(f"Error publishing to Pub/Sub: {e}")

# Nack the message

raise

requirements.txt

functions-framework==3.*

google-cloud-pubsub==2.*

google-cloud-aiplatform==1.*

Deploy it with the following command:

gcloud functions deploy router-agent \

--gen2 \

--runtime=python311 \

--region=us-central1 \

--source=. \

--entry-point=router_agent \

--trigger-topic=ingress-topic

Step 3: Implementing a Specialized ‘Worker Agent’ (e.g., Code-Generator)

Now, let’s build one of the workers. This agent’s only job is to be excellent at generating code. It subscribes to the router-topic with the filter we defined and gets to work when a relevant task arrives.

main.py

import base64

import json

import os

import functions_framework

from google.cloud import pubsub_v1

import vertexai

from vertexai.generative_models import GenerativeModel, Part

# --- Configuration ---

PROJECT_ID = os.environ.get("GCP_PROJECT")

LOCATION = "us-central1"

RESULTS_TOPIC_ID = "results-topic"

# --- Initialize Clients ---

vertexai.init(project=PROJECT_ID, location=LOCATION)

publisher = pubsub_v1.PublisherClient()

model = GenerativeModel("gemini-1.5-pro-001") # Using a more powerful model for the task

results_topic_path = publisher.topic_path(PROJECT_ID, RESULTS_TOPIC_ID)

CODE_GENERATION_PROMPT = """

You are an expert code generation assistant. You write clean, efficient, and well-documented code.

Generate the code based on the following request. Provide only the code block in your response.

Request:

"{user_request}"

"""

@functions_framework.cloud_event

def code_generator_agent(cloud_event):

"""

Cloud Function triggered by a filtered subscription on router-topic.

Generates code using Gemini and publishes the result.

"""

# 1. Decode the incoming message

try:

message_data_encoded = cloud_event.data["message"]["data"]

message_data_str = base64.b64decode(message_data_encoded).decode("utf-8")

payload = json.loads(message_data_str)

print(f"Code Generator received payload: {payload}")

user_request = payload.get("user_request")

correlation_id = payload.get("correlation_id") # Crucial for state management

if not all([user_request, correlation_id]):

raise ValueError("Missing 'user_request' or 'correlation_id' in payload")

except (KeyError, json.JSONDecodeError, ValueError) as e:

print(f"Error processing message: {e}")

return

# 2. Use Gemini to generate the code

try:

prompt = CODE_GENERATION_PROMPT.format(user_request=user_request)

response = model.generate_content(prompt)

generated_code = response.text.strip()

print(f"Gemini generated code for correlation_id: {correlation_id}")

except Exception as e:

print(f"Error calling Gemini API: {e}")

raise

# 3. Publish the result to the results-topic

try:

result_payload = {

"correlation_id": correlation_id,

"status": "SUCCESS",

"result": generated_code

}

result_data = json.dumps(result_payload).encode("utf-8")

future = publisher.publish(results_topic_path, data=result_data)

message_id = future.result()

print(f"Published result {message_id} to {results_topic_path} for {correlation_id}")

except Exception as e:

print(f"Error publishing result to Pub/Sub: {e}")

raise

requirements.txt (same as the router)

functions-framework==3.*

google-cloud-pubsub==2.*

google-cloud-aiplatform==1.*

Deploy this worker, making sure to trigger it from the subscription we created, not the topic directly.

gcloud functions deploy code-generator-agent \

--gen2 \

--runtime=python311 \

--region=us-central1 \

--source=. \

--entry-point=code_generator_agent \

--trigger-topic=router-topic \

--trigger-filter=attributes.agent_type="code-generator"

Note: The gcloud CLI for 2nd Gen functions doesn’t have a direct --trigger-subscription flag. Instead, it creates a new subscription for you with the specified filter. For production, you’d manage the subscription lifecycle explicitly. The command above demonstrates the intent.

Step 4: Managing state and returning results

Our system is asynchronous. The initial caller fires a message and can’t just wait for an HTTP response. So how do we get the result back? The key is a correlation_id.

The client that initiates the request must generate a unique ID (like a UUID) and include it in the initial payload sent to ingress-topic.

// Example initial payload from a client

{

"correlation_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",

"user_request": "Write a python function to calculate the fibonacci sequence"

}

This correlation_id is passed untouched through the Router and to the Worker. When the Worker finishes, it includes this same ID in its message to the results-topic. This allows us to tie the final output back to the original request.

Now, we need a way to consume these results. The best pattern for a web application is to use a state store like Firestore.

1. Create a “Results Handler” Cloud Function: This function’s sole purpose is to listen to the results-topic.

2. Subscribe to results-topic: Create a subscription results-handler-sub for this new function.

3. Write to Firestore: When the handler receives a message, it parses the payload and writes the result to a Firestore collection (e.g., task_results). It should use the correlation_id as the document ID.

Conceptual results_handler/main.py:

# ... (imports for functions_framework, base64, json, firestore)

# Initialize Firestore client

db = firestore.Client()

@functions_framework.cloud_event

def results_handler(cloud_event):

# 1. Decode message from results-topic

# ... (similar decoding logic as other functions)

payload = json.loads(decoded_data)

correlation_id = payload.get("correlation_id")

if not correlation_id:

print("Error: Message missing correlation_id")

return

# 2. Write the entire payload to Firestore

try:

doc_ref = db.collection("task_results").document(correlation_id)

doc_ref.set(payload)

print(f"Result for {correlation_id} stored in Firestore.")

except Exception as e:

print(f"Error writing to Firestore: {e}")

raise

The original client application can now provide a simple REST API endpoint (e.g., another HTTP-triggered Cloud Function) like GET /tasks/{correlation_id}. This endpoint simply reads the document from the task_results collection in Firestore and returns its content. The UI can poll this endpoint until the result is available. This creates a robust, scalable, and fully asynchronous workflow.

Production-Ready Considerations for SREs

Building a functional prototype of our Architecting a Multi Agent Orchestrator Using Google Cloud Pub Sub and Gemini is an exciting milestone. However, transitioning from a “works on my machine” proof-of-concept to a resilient, secure, and cost-effective production system requires a shift in mindset. This is where the principles of Site Reliability Engineering (SRE) come into play. We need to anticipate failure, secure our components, monitor everything, and keep an eye on the bill. Let’s break down the essential considerations for running this system at scale.

Error Handling with Dead-Letter Queues (DLQs)

In a distributed system, failure is not an “if” but a “when.” Messages sent to your agents can fail to be processed for countless reasons: a transient 503 error from the Gemini API, a malformed message payload that your agent’s code can’t parse, a downstream database being temporarily unavailable, or even a bug in an agent’s logic causing an unhandled exception.

If an agent fails to process a message, Pub/Sub’s default behavior is to retry delivery. While this is great for transient issues, it can create a “poison pill” scenario for persistent failures. The same bad message gets redelivered repeatedly, consuming agent resources and potentially blocking the processing of valid messages behind it.

The Solution: Dead-Letter Queues (DLQs)

A Dead-Letter Queue is a simple yet powerful pattern for handling message-processing failures gracefully. You configure your primary subscription to forward any message that fails delivery a certain number of times to a separate Pub/Sub topic, the “dead-letter topic.”

This isolates problematic messages, allowing your main pipeline to continue processing healthy traffic. It also gives you a repository of failed messages that can be inspected, debugged, and potentially reprocessed later.

Implementation Steps:

1. Create the DLQ Topic: This is where the failed messages will be sent.

gcloud pubsub topics create your-agent-inbox-dlq

2. Create a Subscription for the DLQ Topic: This subscription will hold the dead-letter messages for inspection.

gcloud pubsub subscriptions create your-agent-inbox-dlq-sub --topic=your-agent-inbox-dlq

3. Configure the Main Subscription: Update your agent’s primary subscription to use the new DLQ topic after a set number of failed delivery attempts. A value between 5 and 10 is a reasonable starting point.

gcloud pubsub subscriptions update your-agent-inbox-sub \

--dead-letter-topic=your-agent-inbox-dlq \

--max-delivery-attempts=5

Once configured, you must set up an alert on the number of messages in your DLQ subscription (your-agent-inbox-dlq-sub). A rising count of undelivered messages in the DLQ is a critical signal that something is wrong with your agents or the data they are receiving.

Monitoring and Observability with Cloud Monitoring

“If you can’t measure it, you can’t improve it.” To run a reliable system, you need deep visibility into its health and performance. Google Cloud Monitoring provides a rich set of tools for this, integrating seamlessly with Pub/Sub and allowing for custom application metrics.

Key Metrics to Watch:

Your monitoring dashboard should be your single source of truth for system health. Focus on these metrics:

• Pub/Sub Backlog & Latency (The Golden Signals):

• subscription/num_undelivered_messages: This is the most critical metric. A consistently growing backlog means your agents can’t keep up with the ingress rate. This is your primary signal for autoscaling.

• subscription/oldest_unacked_message_age: This tells you the latency of your system. How long is the oldest task waiting to be processed? High values can indicate stuck agents or insufficient processing capacity.

• Pub/Sub Throughput & Errors:

• topic/send_request_count: Measures the rate of new tasks entering the system.

• subscription/deadletter_message_count: Tracks the number of messages sent to your DLQ. Set up an alert for any increase here.

• Agent-Level Custom Metrics (Instrument your code!):

• Gemini API Latency: How long are calls to the LLM taking? This can help identify performance degradation in the model itself.

• Gemini API Error Rate: Track the count of HTTP status codes, especially 429 (Resource Exhausted) for rate limits and 5xx for server errors.

• Task Processing Duration: The wall-clock time from when an agent receives a message to when it ACKs it. This helps you understand your agent’s performance profile.

• LLM Token Usage: Log the number of input and output tokens per task. This is invaluable for performance tuning and cost analysis.

By combining built-in Pub/Sub metrics with custom metrics from your agent’s application code, you can build a comprehensive dashboard in Cloud Monitoring. This allows you to set up intelligent alerting policies that notify you of problems—like a growing backlog or a spike in Gemini API errors—before they impact your users.

Security: IAM Roles and Principle of Least Privilege for Agents

In a multi-agent system, each agent is an independent actor. Granting excessive permissions is a significant security risk. A compromised or buggy agent with broad permissions could access sensitive data, delete resources, or invoke other services maliciously.

The guiding philosophy here is the Principle of Least Privilege (PoLP): every component should have only the minimum permissions required to perform its function.

Implementation with IAM:

1. Dedicated Service Accounts: Never use the default compute service account. Create a unique IAM Service Account for each distinct agent role. For example, a billing-analysis-agent and a data-summarization-agent should have separate service accounts, as their required permissions will differ.

2. Granular IAM Roles: Assign fine-grained IAM roles to these service accounts at the most specific resource level possible.

Let’s consider an agent whose job is to read a request from Pub/Sub, query Gemini, and write a result to a specific Cloud Storage bucket. Its Service Account should have only these roles:

• roles/pubsub.subscriber on projects/your-project/subscriptions/your-agent-inbox-sub

• roles/pubsub.publisher on projects/your-project/topics/your-agent-output-topic

• roles/aiplatform.user on your GCP Project (to access [Building Self Correcting Agentic Workflows with Building Self-Correcting Agentic Workflows with Vertex AI](https://votuduc.com/building-self-correcting-agentic-workflows-with-vertex-ai-p-20260321542526) Gemini endpoints)

• roles/storage.objectCreator on projects/_/buckets/your-specific-results-bucket

This agent has no permission to read from other buckets, delete objects, or access any other GCP service. This tight scoping drastically reduces the “blast radius” if the agent is ever compromised.

Cost Management and Optimization Strategies

LLM-powered systems can become expensive if not managed carefully. The primary cost drivers in our architecture are the Gemini API calls, the compute resources running the agents, and, to a lesser extent, Pub/Sub data volume and Cloud Monitoring/Logging.

Strategies for Optimization:

1. Optimize LLM Interactions:

• Model Selection: Use the right tool for the job. Don’t use the powerful (and expensive) Gemini 1.5 Pro for a simple text classification task if a smaller, faster, and cheaper model would suffice.

• [Prompt Engineering for Reliable Autonomous Workspace Agents for Reliable Autonomous Workspace Agents](https://votuduc.com/prompt-engineering-for-reliable-autonomous-workspace-agents-p-20260319404106): Brevity is key. The more tokens you send in your prompt, the more you pay. Invest time in crafting concise, efficient prompts that yield the desired result with minimum input.

• Intelligent Caching: If agents are likely to receive repetitive requests, implement a caching layer (like Memorystore for Redis). Before calling the Gemini API, check if the exact same request has been processed recently. A cache hit can save you the cost and latency of an LLM call.

2. Optimize Compute Resources:

• Embrace Serverless and Autoscaling: Run your agents on a platform like Cloud Run or GKE Autopilot. These services can automatically scale the number of agent instances based on demand (e.g., using the Pub/Sub backlog size as a scaling metric). Critically, they can scale down to zero when there are no messages to process, eliminating costs for idle compute.

• Subscriber-side Batching: Configure your Pub/Sub client library to pull messages in batches. This allows a single agent instance to process multiple tasks in one go, reducing the overhead of message handling and potentially enabling batched API calls to downstream systems, further improving efficiency.

3. Monitor and Alert on Costs:

• Use Labels: Apply a consistent label (e.g., service: multi-agent-orchestrator) to all your resources—Cloud Run services, Pub/Sub topics, Memorystore instances, etc.

• Filter Billing Reports: In the Cloud Billing console, you can filter your costs by this label. This gives you a precise, real-time view of exactly how much the entire system is costing you.

• Set Billing Alerts: Create a budget for your labeled resources and configure billing alerts to notify you when costs are projected to exceed your threshold. This proactive approach prevents month-end bill shock.

Conclusion and Future Enhancements

We’ve journeyed from a conceptual idea to a functioning, scalable multi-agent orchestrator. By leveraging the asynchronous power of GCP Pub/Sub and the advanced reasoning capabilities of Gemini, we’ve laid the foundation for a system that is both robust and intelligent. But this is just the beginning. Let’s recap what we’ve accomplished and explore the exciting paths that lie ahead.

Recap: The benefits of a decoupled multi-agent system

The architecture we’ve built isn’t just a technical curiosity; it’s a strategic choice that unlocks significant advantages over monolithic designs. By treating each agent as an independent, message-driven service, we’ve gained:

• Massive Scalability: GCP Pub/Sub is designed for planetary-scale event ingestion. As your workload increases, you can scale out your agent services (e.g., Cloud Run instances) independently. A research-intensive task can spin up hundreds of researcher agents without affecting the performance of the code-writing agent.

• Inherent Resilience: Decoupling means no single point of failure. If one agent encounters an error and fails, it doesn’t crash the entire system. The message remains in the Pub/Sub subscription (with proper dead-lettering configured) to be retried or debugged, while other agents continue their work uninterrupted.

• Enhanced Maintainability & Agility: Each agent has a single responsibility. This makes them easier to develop, test, debug, and update. Want to swap out the Gemini model for a fine-tuned version in your “Summarizer” agent? You can do so without touching any other part of the system. This modularity is key to rapid, iterative development.

• Flexibility for Specialization: The true power of a multi-agent system is specialization. Our architecture allows each agent to be an expert in its domain, equipped with specific prompts, tools, and even different versions of the Gemini model best suited for its task.

Next Steps: Expanding the agent pool and building complex workflows

The foundation is solid, and now it’s time to build the skyscraper. The real magic happens when you expand this framework to tackle more complex, multi-step problems.

Here are some ideas to get you started:

1. Expand Your Agent Roster:

Think about the specialized skills you need. You could build:

• A CodeValidator Agent: Subscribes to topics with code snippets and uses static analysis tools or a test framework to validate the code’s correctness.

• A DataVisualizer Agent: Takes structured data (like JSON or CSV) as input and uses libraries like Matplotlib or a call to an external API to generate charts and graphs.

• A WebResearcher Agent: Given a query, it uses Google Search APIs or browser [Automated Job Creation in Real Time Jobber and Google Sheets Integration from Gmail](https://votuduc.com/Automated-Job-Creation-in-Jobber-from-Gmail-p115606) tools to scrape websites, extract relevant information, and publish its findings.

• A GitHubManager Agent: Interacts with the GitHub API to create issues, commit code, or read from a repository.

2. Implement Complex Orchestration Patterns:

Move beyond a single request-response and start chaining agents together to create sophisticated workflows.

• Fan-out/Fan-in: A ProjectManager agent could receive a high-level goal, break it down into five sub-tasks, and publish five separate messages to a SubTask topic. Five Worker agents could pick up these tasks in parallel. A final Aggregator agent could then listen for five corresponding completion messages, assemble the final report, and notify the user.

• Conditional Routing: Introduce a Router agent. Based on the content of a message, this agent’s sole job is to decide which agent should handle the task next. For example, if the output contains code, it routes to the CodeValidator; if it contains a question, it routes back to a Clarification agent.

• State Management: For long-running, multi-step tasks, you’ll need to persist the state. An agent could write its partial result to a Firestore document, identified by a unique correlation_id, before publishing a message for the next agent in the chain. This allows the system to track the progress of a complex job across multiple asynchronous steps.

Your Call to Action

The code and concepts in this article are a launchpad, not a final destination. The most exciting applications of this architecture are the ones you will create.

So, now it’s your turn. Clone the repository, deploy the infrastructure, and start experimenting.

• What is the first new agent you will build? Will it be a creative writer, a meticulous data analyst, or something entirely new?

• How can you adapt this pattern to solve a specific problem you face in your work or personal projects?

• Can you improve the orchestration logic? Perhaps by building a more dynamic routing agent or integrating a formal state machine.

Share your creations, your challenges, and your questions in the comments below or on social media. Fork the repository, build something amazing, and show the world what a truly scalable, intelligent agent system can do. The future is asynchronous, and you now have the blueprint to build it.