5.3 OpenSearch

Data Context Hub uses OpenSearch for logs, traces and metrics which are collected via Open Telemetry. This page only includes some basic information. For more head to the official documentation.

Keycloak Integration

Access to OpenSearch Dashboards is secured by Keycloak. A user also needs one of the following roles assigned to access data

• opensearch-admin grants full access
• opensearch-discover only grants access to Discover

Discover (Logs)

Discover is a tool for exploring your data in OpenSearch Dashboards. You can use Discover to visually represent your data on a dashboard and provide a high-level view of key metrics. In Data Context Hub it is mostly used to display and analyze logs. To use Discover, either opensearch-admin or opensearch-discover roles are required.

Index Patterns

Before Discover can be used for log messages, an index pattern has to be created. This step requires the opensearch-admin role.

• Navigate to "Dashboards Management" > "Index patterns"
• Create an index pattern and use dch-logs-* as "Index pattern name" to include all logs produces by Data Context Hub.
• In "Time field" select time. This field is used when filtering documents based on time. If the drop down is empty there are no messages available yet.

Once the index pattern is created, it automatically identifies all available fields from the referenced log indexes. If messages with new fields are generated after the index pattern was created, select the index pattern and click on "Refresh field list" to update the pattern.

Discover Overview

In order to see log messages in Discover, select the created index pattern in the top left corner.

All available messages, matching the default filter for "Last 15 minutes", are displayed in the "Main Area". The graph at the top is based on the time field selected in the index pattern. Beneath is the list of messages ordered by time. Per default, only "Time" and "_source", which includes the whole message, are displayed.

Displayed fields can be configured in the "Fields" panel on the left hand side. Beneath the "Selected fields", that are displayed in the "Main Area", is a list of all "Available fields". Selecting an available field automatically updates the "Main Area" which displays the corresponding column.

info

Missing fields can be due to messages added to indexes after the index pattern was created. Return to the index pattern and click "Refresh field list" to update fields in Discover.

The "Search and Filter" section allows to define search string using DQL or Lucene. Additionally it is possible to filter based on fields and time.

For more information consult the official documentation.

Traces

Traces provide comprehensive insights into distributed systems. It allows to track request throughout the system and analyze potential bottlenecks. To see Traces, opensearch-admin role is required.

Metrics

The Metrics tool allows to analyze, visualize, and correlate data across logs, traces, and metrics. To see Metrics, opensearch-admin role is required.

Index State Management (ISM)

OpenSearch stores everything in indexes be it logs, metrics or traces. Index State Management allows to automate periodic, administrative operations, such as reducing replica count, or deleting indexes, by triggering them based on changes in the index age, index size, or number of documents.

Data Context Hub defines policies to delete data produced by the system after a certain amount of time or when exceeding a certain size. For initial deployment these values can be changed in the Helm chart's values.yaml. Updating an already existing policy can either be done on OpenSearch Dashboards' Index Management page or via ISM API. Changing an existing policy creates a new version of it that is not automatically applied to existing indexes but only to newly created ones. Applying a new version of a policy to an existing index can only be done via ISM API.

caution

Changing policies may increase required storage.