Skip to main content

Custom Observability

Use VeloDB Cloud as the storage and SQL analysis layer for application logs and distributed traces. Keep your existing collectors, define tables that match your telemetry, and query the data with SQL or a visualization tool such as Grafana.

Choose this approach when you need control over collection, transformation, table design, or visualization. For a walkthrough that combines a collector, VeloDB Cloud, and Grafana, start with DOG Stack.

Architecture and responsibilities​

A custom observability pipeline has three layers:

Applications and infrastructure
|
v
Collectors and preprocessing that you operate
(OpenTelemetry Collector, Logstash, Filebeat, or Fluent Bit)
|
| Stream Load
v
VeloDB Cloud warehouse
(log and trace tables, indexes, SQL queries)
|
| MySQL protocol
v
Visualization tools that you operate
(Grafana dashboards, trace views, and alerts)
LayerYour choices and responsibilities
Collection and preprocessingInstrument applications, deploy collectors, parse logs, choose trace sampling, protect credentials, and monitor delivery failures.
Storage and analysisCreate tables, choose indexed fields, set retention, load telemetry, and size warehouse compute for your workload. VeloDB Cloud manages the database service.
Visualization and alertingConfigure a SQL-compatible data source, dashboards, trace views, access control, and alert rules. These tools run outside the warehouse.

This architecture applies to SaaS and BYOC warehouses. The network path and endpoints depend on the deployment. Follow Connect to a Warehouse before configuring a collector or visualization tool.

Note:

These guides describe telemetry from applications and infrastructure that you operate. They do not provide access to the managed warehouse's FE or BE hosts, configuration files, or host log directories.

Choose a collection path​

Source or requirementCollection pathStart here
Applications instrumented with OpenTelemetrySend OTLP to an OpenTelemetry Collector that includes the Doris exporter.OpenTelemetry integration
Existing Logstash pipelinesKeep your inputs and filters, then configure the Doris output plugin to use the warehouse Stream Load endpoint.Logstash integration
File-based logs with FilebeatUse a Filebeat build that includes the Doris output plugin. Standard Filebeat distributions do not include this output.Filebeat integration
Existing Fluent Bit deploymentCheck the Doris output build and its endpoint and TLS compatibility before using it.Fluent Bit integration
Logs already in KafkaUse Routine Load when the warehouse can reach the Kafka brokers and authenticate to them.Routine Load
A custom collection applicationSend batches to Stream Load and inspect the result of every load.Log storage and analysis

Plugin names such as doris and output.doris identify Apache Doris-compatible integrations. Keep those configuration names when connecting to VeloDB Cloud.

Design tables for your queries​

Start with the fields that you filter or aggregate most often:

  • Time: Partition by event time to limit scans and manage retention. Align collector timestamps, the table's partition time zone, and dashboard time ranges.
  • Service or host: Keep frequently filtered identifiers in typed columns. A sort key can improve locality for common query patterns.
  • Log text: Add an inverted index with a text parser for token and phrase searches.
  • Trace identifiers: Store trace and span identifiers so that you can find a complete request and correlate it with logs.
  • Variable attributes: Use VARIANT for attributes whose keys change between events. Keep stable fields that are important to queries in explicit columns.

Each index consumes storage and adds work during ingestion. Index fields that your workload uses, then measure load throughput, query latency, and storage growth with representative data. See Log storage and analysis and Trace storage and analysis for schema examples.

Evaluate capacity and cost​

Use your own workload to compare alternatives. Results depend on event size, retention, indexing, query concurrency, sampling, batch size, and warehouse compute. Upstream Apache Doris benchmark results do not establish performance or cost for your VeloDB Cloud warehouse.

MeasureHow to use it
Average and peak ingestion rateInclude bursts and collector retry traffic when sizing compute.
Compressed table and index sizeEstimate retained storage from a representative sample. Include indexes in the measurement.
Time until events become queryableMeasure the complete path, including collection, batching, transmission, and publication.
Query latency and concurrencyTest keyword searches, recent-event queries, aggregations, and trace lookups while ingestion is running.
Collector queue growth and failuresCheck whether collectors can absorb an outage and recover without exceeding the warehouse's capacity.

Retention and trace sampling determine how much data you store. Shorter batches can improve freshness but increase request overhead. Larger batches use more collector memory and can increase delay. Select settings from measured behavior rather than treating a fixed batch size as a universal recommendation.

Warehouse compute and storage are billable. Collectors, visualization tools, and network transfer can incur separate costs in your cloud account. Review Billing, set a retention policy before continuous ingestion, and remove test data and unused resources after a trial.

Query and visualize telemetry​

Use SQL to search log text, aggregate events over time, and join telemetry with other data in the warehouse. Grafana can query the warehouse through a compatible data source. Trace views require queries and field mappings that match your trace schema.

The following screenshots illustrate log and trace analysis. The exact controls depend on the visualization tool and plugin version that you deploy.

Grafana log view with log entries and a time histogram

Grafana trace view showing the spans in a request

OpenTelemetry also defines metrics. Before adding a metrics pipeline, verify the exporter's supported metric types and the target schema for the Collector version you use. The guides in this section focus on logs and traces.

See also​