General
PromptBeginner5 minmarkdown
<h1 align="center">
<a href="https://prompts.chat">
5
This document is the single source of truth for Prometheus metrics exposed by Weaviate. It explains what we measure and why, how to use the metrics, and how we keep the set lean and cost‑effective.
Sign in to like and favorite skills
This document is the single source of truth for Prometheus metrics exposed by Weaviate. It explains what we measure and why, how to use the metrics, and how we keep the set lean and cost‑effective.
docs/metrics.md) is authoritative. Any metric changes (add/modify/deprecate) must be reflected here in the correct section.| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration in ms of a single batch | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of currently ongoing async operations | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of concurrently running query operations | | | ❌ High |
| Number of all requests made | | | ❌ High |
| Duration of queries in milliseconds | | | ❌ High |
| Duration of queries in milliseconds | | | ❌ High |
| Vector dimensions used by read-queries involving vectors | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of currently present segments per shard | | | ❌ High |
| Size of memtable by path | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of currently ongoing async operations | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of records in the queue | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of active vector index tombstones | | | ❌ High |
| Total number of deleted objects that have been cleaned up | | | ❌ High |
| Total number of unexpected tombstones found | | | ❌ High |
| Total number of mutating operations on the vector index | | | ❌ High |
| The size of the vector index | | | ❌ High |
| Total segments in a shard if quantization enabled | | | ❌ High |
| Total dimensions in a shard | | | ❌ High |
| Duration of typical vector index operations (insert, delete) | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Ratio (percentage) of startup progress for a particular component in a shard | | | ❌ High |
| Disk I/O throughput in bytes per second | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of tombstone delete local entrypoint calls | | | ❌ High |
| Total number of tombstone delete global entrypoint calls | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of batches currently running | | | - Low |
| Time of a batch spent in specific portions of the queue | | | - Low |
| Duration of an individual request to the vectorizer | | | - Low |
| Number of tokens in a user-defined batch | | | - Low |
| Number of tokens in an individual request sent to the vectorizer | | | - Low |
| Rate limit stats for the vectorizer | | | - Low |
| Why batch scheduling is repeated | | | - Low |
| Number of requests required to process an entire (user) batch | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of shards per index status | | | - Low |
| Time taken to update shard status in seconds | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of tenants processed | | | - Low |
| Time spent in auto tenant operations | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Unix epoch timestamp of the end of the last tombstone cleanup cycle | | | ❌ High |
| Ratio (percentage) of the progress of the current tombstone cleanup cycle | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of tenant offload operations | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Latency of usage operations in seconds | | | - Low |
| Size of the last uploaded usage file in bytes | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of shards currently loading | | | - Low |
| Number of shards waiting for permit to load | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of replication operations pending processing | | | - Low |
| Number of replication operations currently in progress | | | - Low |
| Number of successfully completed replication operations | | | - Low |
| Number of failed replication operations | | | - Low |
| Number of cancelled replication operations | | | - Low |
| Replication engine running status (0:not running, 1:running) | | | - Low |
| Replication engine producer running status (0:not running, 1:running) | | | - Low |
| Replication engine consumer running status (0:not running, 1:running) | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of active distributed tasks running per namespace | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Time (in seconds) spent serving requests | | | ❌ High |
| Size (in bytes) of the request received | | | ❌ High |
| Size (in bytes) of the response sent | | | ❌ High |
| Current number of inflight requests | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Time (in seconds) spent serving requests | | | ❌ High |
| Size (in bytes) of the request received | | | ❌ High |
| Size (in bytes) of the response sent | | | ❌ High |
| Current number of inflight requests | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Time to apply cluster store FSM state in local node | | | - Low |
| Total failure count of cluster store FSM state apply in local node | | | - Low |
| Current applied index of a raft cluster in local node | | | - Low |
| Current applied index of cluster store FSM in local node | | | - Low |
| Previous applied index of the cluster store FSM in local node | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of collections per node | | | - Low |
| Number of shards per node with corresponding status | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Whether the last loading attempt of runtime config was success | | | - Low |
| Hash value of the currently active runtime configuration | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of queries in milliseconds | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of a sync or async vector index maintenance operation | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of module operations | | | - Low |
| Number of resources tracked by module | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Size of a raw batch request batch in bytes | | | - Low |
| Number of objects in a batch | | | - Low |
| Number of unique tenants referenced in a batch | | | - Low |
| Duration in ms of a single delete batch | | | ❌ High |
| Number of objects processed in a batch | | | ❌ High |
| Number of bytes processed in a batch | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of bitmap buffers used by size | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of bytes written to disk | | | - Low |
| Total number of bytes read from disk | | | - Low |
| Total number of mmap operations | | | - Low |
| Number of entries in /proc/self/maps | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of schema writes (which always involve the leader) | | | - Low |
| Duration of local schema reads that do not involve the leader | | | - Low |
| Duration of schema reads that are passed to the leader | | | - Low |
| Duration of waiting for a schema version to be reached | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of an individual object operation | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Vector dimensions used by read-queries, aggregated across all classes and shards | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of concurrently running goroutines | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of segments per shard in the objects bucket | | | ❌ High |
| Number of segments per shard in the vectors_compressed bucket | | | ❌ High |
| Number of objects/entries of segment by level | | | ❌ High |
| Size of segment by level and unit | | | ❌ High |
| Number of segments by level | | | ❌ High |
| Number of unloaded segments | | | ❌ High |
| Time in ms for a bucket operation to complete | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Disk usage of the queue | | | ❌ High |
| Whether the queue is paused | | | ❌ High |
| Number of queues | | | ❌ High |
| Duration in ms of a single partition processing | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of insert operations added to the vector index queue | | | ❌ High |
| Number of delete operations added to the vector index queue | | | ❌ High |
| Number of threads in use to clean up tombstones | | | ❌ High |
| Unix epoch timestamp of the start of the current tombstone cleanup cycle | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of individual startup operations in ms | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of a backup restore | | | ❌ High |
| Duration restoring class | | | ❌ High |
| Startup phase of a backup restore | | | ❌ High |
| File transfer stage of a backup restore | | | ❌ High |
| File transfer stage of a backup store | | | ❌ High |
| Bucket pause durations | | | - Low |
| Total number of bytes transferred during a backup restore | | | ❌ High |
| Total number of bytes transferred during a backup store | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of shards loaded | | | - Low |
| Number of shards not loaded | | | - Low |
| Number of shards in process of loading | | | - Low |
| Number of shards in process of unloading | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of opened schema transactions | | | - Low |
| Total number of closed schema transactions | | | - Low |
| Mean duration of a tx by status | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total number of tombstone reassign neighbor calls | | | ❌ High |
| Delete list size of tombstones | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of a tokenizer operation | | | - Low |
| Number of tokenizer requests | | | - Low |
| Duration of a tokenizer initialization operation | | | - Low |
| Number of tokens processed | | | - Low |
| Number of tokens processed per request | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Number of module requests to external APIs | | | ❌ High |
| Duration of an individual request to a module external API | | | ❌ High |
| Number of items in a batch | | | ❌ High |
| Size (in bytes) of the request sent to an external API | | | ❌ High |
| Size (in bytes) of the response received from an external API | | | ❌ High |
| Number of tokens in the request sent to an external vectorizer | | | ❌ High |
| Number of single-item external API requests | | | ❌ High |
| Number of batched module requests | | | ❌ High |
| Number of OpenAI errors | | | ❌ High |
| Number of module errors (related to external calls) | | | ❌ High |
| Number of API response statuses | | | ❌ High |
| Number of batch errors | | | ❌ High |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Total bytes fetched during tenant offload operations | | | - Low |
| Total bytes transferred during tenant offload operations | | | - Low |
| Name | Description | Type | Labels | High Cardinality |
|---|---|---|---|---|
| Duration of checksum validation | | | - Low |
| Number of bytes read during checksum validation | | | - Low |
| Name | Description | Type | Labels | Reason | Removed In |
|---|---|---|---|---|---|
| Duration of bloom filter operations | | | Removed due to high CPU cost and synchronization on hot path during segment reads; no demonstrated value | v1.31 (PR #9057) |