Overview
Google Vertex AI Vector Search is a fully managed, high-scale, low-latency solution for finding similar vectors. It supports both exact and approximate nearest neighbor (ANN) search using Google’s ScaNN (Scalable Nearest Neighbors) technology. Vertex AI Vector Search is available in two versions:- Vector Search 2.0: Uses Collections to store Data Objects containing vectors, metadata, and content together. Provides a unified data model with simpler and quicker operations.
- Vector Search 1.0: Uses Indexes deployed to Endpoints with separate document storage in Google Cloud Storage or Datastore.
- Vector Search 2.0 - If you’re using Collections
- Vector Search 1.0 - If you’re using Indexes and Endpoints
Installation
Install the LangChain Google Vertex AI integration:Vector Search 2.0
Vector Search 2.0 uses Collections to store Data Objects. Each Data Object contains vectors, metadata, and content in a unified structure.Prerequisites
-
Google Cloud project with Vertex AI API and Vector Search APIs enabled
- Vector Search Collection created (see Creating a Collection)
-
Appropriate IAM permissions (
Vertex AI Userrole or equivalent)
Creating a Collection (V2)
Before using Vector Search 2.0, you need to create a collection. Here’s how to create a collection compatible with LangChain:- The vector field must be named
"embedding"to match LangChain’s default (or usevector_field_nameparameter) - Only fields defined in
data_schema.propertiescan be used for filtering in V2 - Dimensions should match your embedding model (768 for text-embedding-005)
Initialization
collection_id: Your Vector Search Collection ID (required)api_version: Must be set to"v2"(required)project_id: GCP project ID (required)region: GCP region where Collection exists (required)vector_field_name: Name of the vector field in your Collection schema (default:"embedding")
Adding Documents
Adding Texts
Searching
Basic Similarity Search
Similarity Search with Scores
Search by Vector
Filtering
Vector Search 2.0 uses dict-based query syntax for filtering Data Objects:$eq: Equals$ne: Not equals$lt: Less than$lte: Less than or equal$gt: Greater than$gte: Greater than or equal$and: Logical AND$or: Logical OR$not: Logical NOT
Delete Operations
Delete by IDs
Delete by Metadata Filter
Note: Delete by metadata filter has limitations in the current V2 API. The recommended approach is to:- Use
similarity_searchwith your filter to get document IDs - Delete by IDs
Advanced Features
Vector Search 2.0 offers several advanced search capabilities that go beyond traditional dense vector search.Semantic Search
Semantic search automatically generates embeddings from your query text using Vertex AI models. Your collection must be configured withvertex_embedding_config in the vector schema.
RETRIEVAL_QUERY: For search queries (default)RETRIEVAL_DOCUMENT: For document indexingSEMANTIC_SIMILARITY: For semantic similarity tasksCLASSIFICATION: For classification tasksCLUSTERING: For clustering tasks
Text Search
Text search performs keyword/full-text matching on data fields without using embeddings.semantic_search() or similarity_search() if you need filtering.
Hybrid Search
Hybrid search combines semantic search (with auto-generated embeddings) and text search (keyword matching) using Reciprocal Rank Fusion (RRF) to produce optimally ranked results.- Higher
semantic_weight: Prioritizes semantic understanding - Higher
text_weight: Prioritizes exact keyword matches - Equal weights (default): Balanced results
Custom Vector Field Names
If your Collection schema uses a custom field name for vectors:Additional Resources
- Vector Search 2.0 Overview
- Vector Search 2.0 Collections
- Vector Search 2.0 Data Objects
- Vector Search 2.0 Search Guide
- Vector Search 2.0 Query Guide
- Vector Search 2.0 Migration Guide
Vector Search 1.0
This notebook shows how to use functionality related to theGoogle Cloud Vertex AI Vector Search vector database.
Google Vertex AI Vector Search, formerly known as Vertex AI Matching Engine, provides the industry’s leading high-scale low latency vector database. These vector databases are commonly referred to as vector similarity-matching or an approximate nearest neighbor (ANN) service.Note: LangChain API expects an endpoint and deployed index already created.Index creation time can take upto one hour.
To see how to create an index refer to the section Create Index and deploy it to an Endpoint If you already have an index deployed , skip to Create VectorStore from texts
Create index and deploy it to an endpoint
- This section demonstrates creating a new index and deploying it to an endpoint
Use VertexAIEmbeddings as the embeddings model
Create an empty index
Note : While creating an index you should specify an “index_update_method” from either a “BATCH_UPDATE” or “STREAM_UPDATE”A batch index is for when you want to update your index in a batch, with data which has been stored over a set amount of time, like systems which are processed weekly or monthly. A streaming index is when you want index data to be updated as new data is added to your datastore, for instance, if you have a bookstore and want to show new inventory online as soon as possible. Which type you choose is important, since setup and requirements are different.Refer Official Documentation for more details on configuring indexes