Intelligent Retrieval Filtering

Intelligent retrieval filtering in XpertAI combines structured conditions with vector semantic search in a single query. Engineering, water resources, logistics, policies, and other document collections can live in one knowledge base while file names, logical folders, file types, and business metadata narrow the candidate set before semantic ranking. The feature provides two complementary filter layers:

Fixed filters are configured by an administrator and form a boundary that an agent cannot change.
Agent-generated filters are added when the agent can infer useful conditions from the current question and conversation context.

Every search also remains inside the tenant, organization, knowledge base permission, and enabled-content boundaries. An agent can only narrow a fixed scope; it cannot override or bypass it.

When to use it

Intelligent filtering is especially useful when:

one knowledge base contains documents from several domains or projects;
multiple agents share the same content but must retrieve different scopes;
quoting, compliance, or support applications require structured eligibility rules before semantic matching;
questions frequently include a year, region, format, document status, or category;
operators need to reconstruct why a particular set of chunks was retrieved.

For example, a single construction-cost knowledge base can contain water resources, logistics, and building documents. A water-resources quoting agent can be fixed to metadata.domain = water-resources and metadata.documentStatus = effective. When a user asks for “2025 PDF rate schedules,” the agent can add year, PDF, and file-name conditions for that request.

How the two layers work together

Layer	Configured by	Purpose	Can the agent change it?
Fixed filter	Administrator or workflow designer	Defines business scope, access boundaries, and long-lived constraints	No
Agent-generated filter	Agent	Narrows candidates for the current question	The agent only decides whether to add valid conditions

When the agent cannot determine a reliable condition, it omits the dynamic filter and searches only within the fixed scope. Fixed filters are never exposed as tool arguments.

Effective filter

The effective search scope can be understood as:

tenant / organization / knowledge base boundary
AND enabled documents and chunks
AND fixed filter
AND caller or test filter
AND agent-generated filter

Fixed and agent-generated filters are always combined with AND. Each layer can contain nested AND and OR groups, allowing expressions such as “effective documents in either East China or South China.”

Filterable fields

System fields

Field	Description	Example
`document.fileName`	File name	`Water_Resources_Rate_Schedule_2025.pdf`
`document.folderPath`	Logical folder inside the knowledge base	`Cost/Water-Resources/East-China`
`document.fileExtension`	Normalized extension	`pdf`
`document.mimeType`	MIME type	`application/pdf`
`document.category`	Content category	`text`
`document.sourceType`	Document source type	`local-file`
`document.createdAt`	Document creation time	`2025-01-15T08:00:00Z`
`document.updatedAt`	Document update time	`2025-06-01T12:30:00Z`

document.folderPath is the logical path inside the knowledge base, not a server storage path. The under operator matches the selected folder and its descendants using complete path segments, so water does not accidentally match water-archive.

Folder path values and matching

XpertAI derives a folder path from the knowledge base directory tree. A breadcrumb such as Water Resources > East China has the canonical value Water Resources/East China.

Canonical paths use / between directory names and do not have a leading or trailing slash.
A document stored directly in the knowledge base root has an empty folder path. The recall-test folder picker displays this as / (root).
equals (eq) matches only the exact current folder. Water Resources/East China does not include Water Resources/East China/Archive.
is under (under) matches the selected folder and every descendant. It normalizes leading and trailing slashes, then compares complete path segments.
The recall-test panel lists existing folders and submits their canonical values, avoiding mismatches caused by manually copied breadcrumbs.

Do not use a document’s filePath value as a filter. filePath identifies an internal or external storage object and may contain generated directories, identifiers, or file names. It is deliberately excluded from the public filter field registry and does not have to change when a logical folder is renamed or moved. Folder creation, rename, and move operations update the logical path of the folder and its descendants. The updated attributes are synchronized to PGVector or Milvus without recomputing embeddings. During an upgrade, existing logical paths are rebuilt from parent-child relationships rather than inferred from storage paths.

Business metadata

The metadata schema can declare document-scoped and chunk-scoped fields:

metadata.<field> applies to a document and all its chunks, such as domain, year, region, or document status.
chunk.metadata.<field> applies only to the matching chunk, such as section type, specialty code, or bill-item category.

Fields can be strings, enums, numbers, datetimes, booleans, string arrays, number arrays, or objects. Administrators can also provide a label, description, and allowed enum values. Agents can only use registered fields whose values satisfy the declared type.

Model stable attributes that frequently participate in retrieval as metadata, such as domain, effectiveYear, region, documentStatus, and specialtyCode.

Operators

XpertAI offers operators based on the selected field type, preventing invalid comparisons.

Field type	Common operators
String and enum	equals, not equals, in, not in, contains, does not contain, starts with, ends with, exists
Folder path	String operators plus under, which includes the folder and descendants
Number and datetime	equals, not equals, greater than, greater than or equal, less than, less than or equal, between, in, exists
Boolean	equals, not equals, exists
String or number array	contains, contains any, contains all, is empty, exists
Object	JSON contains, exists

Equality and membership operators use exact matching. Text contains, starts-with, and ends-with comparisons are case-insensitive. Datetimes use ISO-8601 UTC.

Configure a fixed filter

Fixed filters belong to the binding between an agent or workflow node and a knowledge base. They are not global knowledge base policies. This lets several agents safely reuse one knowledge base with different boundaries.

In Studio, open an agent Knowledge Base node or a workflow Knowledge Retrieval node.
Add or select a knowledge base and set the retrieval mode to Vector.
Add fields, operators, and values in Fixed filter.
Create AND or OR groups when required.
Save the configuration and validate it in the test panel.

A fixed value can be a constant or an agent/workflow variable. For example:

metadata.domain equals "water-resources"
AND metadata.documentStatus equals "effective"
AND document.folderPath is under {{input.regionFolder}}

If a fixed variable is missing, has the wrong type, or references an invalid field, retrieval stops before the vector store is queried. XpertAI never ignores a broken fixed boundary and falls back to an unrestricted search.

Allow the agent to filter automatically

Enable Allow agent automatic filtering on the same binding to make all registered system and metadata fields available to the Knowledge Retriever tool, together with their types, enum values, and allowed operators. For the question:

Find the 2025 PDF water-resources rate schedule.

the agent can add:

document.fileExtension in ["pdf"]
AND metadata.effectiveYear equals 2025
AND document.fileName contains "rate schedule"

These conditions only narrow the water-resources and effective-document scope configured by the administrator.

Let the agent discover live filter values

Folder names, file types, and business metadata values are live knowledge base data, so the agent is not expected to guess them from the conversation. When agent automatic filtering is enabled, XpertAI also registers a read-only knowledge filter options tool for that knowledge base. The agent can first call the tool without a field to inspect every discoverable field, its type, allowed operators, and option kind. It can then query a selected field to:

list actual file names, extensions, MIME types, categories, source types, enum values, booleans, and string values;
inspect distinct number and datetime values together with their current minimum and maximum;
expand string-array and number-array metadata into selectable member values;
compare eligible and existing document/chunk counts for fields that primarily support exists;
search values by partial, case-insensitive text, or page through the complete list;
copy a returned value exactly into the Retriever tool’s dynamic filter.

Document-scoped metadata returns document and chunk counts; chunk-scoped metadata is counted against the matching chunks and their source documents. Configured enum values remain visible in the field catalog, while the value query shows which values currently occur inside the effective boundary. Folder paths receive additional handling: the tool lists canonical logical paths, distinguishes the knowledge base root (""), reports direct and descendant document counts, and includes useful ancestor paths for the under operator. For example, a request mentioning “the East China folder” can produce the following sequence:

Inspect filter fields, then query document.folderPath for "East China"
Receive document.folderPath = "Water Resources/East China"
Retrieve with document.folderPath under "Water Resources/East China"

The tool returns knowledge base-relative logical paths only, never server filePath values. Every live value and count is calculated after tenant, organization, knowledge base, enabled-content, and administrator-managed fixed boundaries are applied. Fixed conditions remain hidden, and a missing fixed-filter variable stops option discovery just as it stops retrieval. Empty folders and folders without eligible chunks are omitted; ancestor folders can still be returned because they are valid choices for the under operator.

Explore GraphRAG before retrieving chunks

When GraphRAG is enabled for the bound knowledge base, XpertAI also gives the agent a read-only knowledge graph explorer tool. This tool is useful when the user knows a concept or relationship but does not know the exact terminology used in the documents. Instead of forcing one graph query to produce a final answer, the agent can explore in several small calls:

Call search with the question or a short concept to find semantically related seed entities.
Call neighbors with a returned entity ID to follow relevant relationships for one or two hops.
Call evidence for a promising entity to inspect bounded supporting references.
Repeat the exploration when the relationship is still unclear, or pass suggestedRetrievalQuery to the knowledge Retriever to retrieve the final source chunks and citations.

For example:

search("pump-station supplier")
→ entity GOODSPRINGS

neighbors(entity GOODSPRINGS)
→ GOODSPRINGS -- SUPPLIES --> Pump station equipment

evidence(entity GOODSPRINGS)
→ supporting document/chunk references and a suggested retrieval query

Retriever("pump-station supplier GOODSPRINGS pump station equipment")
→ source chunks and citations used in the answer

The graph tool intentionally uses a compact parameter schema: action, query, entityId, depth, and take. Entity IDs are returned by earlier calls, so the agent does not have to construct nested graph expressions or guess internal identifiers. Graph exploration and filter-option discovery complement each other. The filter-options tool discovers current structured values such as folders, file types, years, and metadata; the graph explorer discovers current entities, names, and relationships extracted from document content. The agent can use both outputs before making the Retriever call. Every graph result is checked against tenant, organization, knowledge base, enabled-content, and fixed-filter boundaries. An entity is returned only when it has eligible chunk evidence in the effective scope; a relationship is returned only when its supporting evidence is also in scope. When a fixed filter is active, global entity descriptions, summaries, aliases, and aggregate counts are not exposed because they may have been built from documents outside that binding. Missing fixed variables fail closed before graph exploration begins. Graph evidence is an exploration clue, not the final answer source. The agent is instructed to finish with the Retriever so normal answers still use full chunks and traceable citations. This auxiliary tool does not enable filtered Graph or Hybrid retrieval mode: the final filtered retrieval remains Vector in the current release. See Knowledge Graph and GraphRAG for graph extraction, indexing, visual curation, retrieval modes, and operational guidance.

Invalid filters and zero results

If a dynamic filter contains an unknown field, wrong value type, or unsupported operator, the complete dynamic filter is discarded. XpertAI searches with the fixed filter and records invalid_dynamic_filter.
If a valid dynamic filter returns no results, the server does not automatically broaden the search.
The tool reports the zero-hit result and indicates that a call without the dynamic filter is possible. The agent decides whether retrying is appropriate.
Disabled documents and chunks are excluded from every retrieval path.

Test and diagnose filtering

The knowledge base test panel and agent debugger show:

the fixed, caller/test, and agent-generated filters;
the effective filter and the source of each condition;
candidate document count, candidate chunk count, and final hit count;
vector backend, filter latency, and vector-search latency;
whether an invalid dynamic filter was discarded and why.

Normal end-user answers continue to show only the answer and citations by default, not internal filtering decisions. Retrieval logs retain normalized conditions, summaries, counts, backend information, and timing for auditing, but never document body text. Access to these logs follows knowledge base permissions.

Support matrix

Capability	Current support
Vector mode	Supported
PGVector	Fully supported
Milvus	Fully supported; Milvus Server 2.6.2 or later is required
Graph-assisted exploration	Supported when GraphRAG is enabled; final filtered chunk retrieval remains Vector
Graph / Hybrid mode	Rejects intelligent filter configuration with an explicit error
Chroma / Weaviate	Rejects the new filter conditions with an explicit error
External knowledge bases	Depends on the provider API; XpertAI does not silently pass through or ignore the new filter DSL

Best practices

Use fixed filters for security boundaries, domain ownership, document status, and project scope.
Enable agent filtering only for reliable fields that commonly appear in user questions.
Prefer enums over free text for controlled business attributes.
Store years as numbers, effective timestamps as datetimes, and flags as booleans instead of converting every value to a string.
Use folders for content organization and metadata for stable business semantics; they can be combined.
Before release, test queries with and without dynamic filters, zero-hit cases, and missing fixed variables.

Continue with Maintaining documents and metadata, Recall testing, and Ways to use a knowledge base.

Default

Create Knowledge Base Via Pipeline

Intelligent Retrieval Filtering

When to use it

How the two layers work together

Effective filter

Filterable fields

System fields

Folder path values and matching

Business metadata

Operators

Configure a fixed filter

Allow the agent to filter automatically

Let the agent discover live filter values

Explore GraphRAG before retrieving chunks

Invalid filters and zero results

Test and diagnose filtering

Support matrix

Best practices

​When to use it

​How the two layers work together

​Effective filter

​Filterable fields

​System fields

​Folder path values and matching

​Business metadata

​Operators

​Configure a fixed filter

​Allow the agent to filter automatically

​Let the agent discover live filter values

​Explore GraphRAG before retrieving chunks

​Invalid filters and zero results

​Test and diagnose filtering

​Support matrix

​Best practices

When to use it

How the two layers work together

Effective filter

Filterable fields

System fields

Folder path values and matching

Business metadata

Operators

Configure a fixed filter

Allow the agent to filter automatically

Let the agent discover live filter values

Explore GraphRAG before retrieving chunks

Invalid filters and zero results

Test and diagnose filtering

Support matrix

Best practices