Skip to main content
Version: Next

Grafana

Certified

Important Capabilities​

CapabilityStatusNotes
Column-level Lineage✅Enabled by default
Detect Deleted Entities✅Enabled by default
Extract Ownership✅Enabled by default
Extract Tags✅Enabled by default
Platform Instance✅Enabled by default
Table-Level Lineage✅Enabled by default

This plugin extracts metadata from Grafana and ingests it into DataHub. It connects to Grafana's API to extract metadata about dashboards, charts, and data sources. The following types of metadata are extracted:

  • Container Entities:

    • Folders: Top-level organizational units in Grafana
    • Dashboards: Collections of panels and charts
    • The full container hierarchy is preserved (Folders -> Dashboards -> Charts/Datasets)

  • Charts and Visualizations:

    • All panel types (graphs, tables, stat panels, etc.)
    • Chart configuration and properties
    • Links to the original Grafana UI
    • Custom properties including panel types and data source information
    • Input fields and schema information when available

  • Data Sources and Datasets:

    • Physical datasets representing Grafana's data sources
    • Dataset schema information extracted from queries and panel configurations
    • Support for various data source types (SQL, Prometheus, etc.)
    • Custom properties including data source type and configuration

  • Lineage Information:

    • Dataset-level lineage showing relationships between:
      • Source data systems and Grafana datasets
      • Grafana datasets and charts
    • Column-level lineage for SQL-based data sources
    • Support for external source systems through configurable platform mappings

  • Tags and Ownership:

    • Dashboard and chart tags
    • Ownership information derived from:
      • Dashboard creators
      • Technical owners based on dashboard UIDs
      • Custom ownership assignments

The source supports the following capabilities:

  • Platform instance support for multi-Grafana deployments
  • Stateful ingestion with support for soft-deletes
  • Fine-grained lineage at both dataset and column levels
  • Automated tag extraction
  • Support for both HTTP and HTTPS connections with optional SSL verification

Prerequisites:

  1. A running Grafana instance
  2. A service account token with permissions to:
    • Read dashboards and folders
    • Access data source configurations
    • View user information

A sample configuration file:

source:
    type: grafana
    config:
        # Coordinates
        platform_instance: production # optional
        env: PROD # optional
        url: https://grafana.company.com
        service_account_token: ${GRAFANA_SERVICE_ACCOUNT_TOKEN}

        # SSL verification for HTTPS connections
        verify_ssl: true # optional, default is true

        # Source type mapping for lineage
        connection_to_platform_map:
            postgres:
                platform: postgres
                database: grafana  # optional
                database_schema: grafana  # optional
                platform_instance: database_2  # optional
                env: PROD  # optional
            mysql_uid_1:  # Grafana datasource UID
                platform: mysql
                platform_instance: database_1  # optional
                database: my_database  # optional

CLI based Ingestion​

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
service_account_token ✅
string(password)
Grafana API token
url ✅
string
URL of Grafana instance (e.g. https://grafana.company.com)
ingest_owners
boolean
Whether to ingest owners from Grafana dashboards and charts
Default: True
ingest_tags
boolean
Whether to ingest tags from Grafana dashboards and charts
Default: True
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
platform_instance_map
map(str,string)
verify_ssl
boolean
Verify SSL certificate for secure connections (https)
Default: True
env
string
The environment that all assets produced by this connector belong to
Default: PROD
connection_to_platform_map
map(str,PlatformConnectionConfig)
Any source that produces dataset urns in a single environment should inherit this class
connection_to_platform_map.key.platform_instance
string
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://datahubproject.io/docs/platform-instances/ for more details.
connection_to_platform_map.key.platform ❓
string
Upstream platform code (e.g. postgres, ms-sql)
connection_to_platform_map.key.database
string
Database name
connection_to_platform_map.key.database_schema
string
Schema name
connection_to_platform_map.key.env
string
The environment that all assets produced by this connector belong to
Default: PROD
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Code Coordinates​

  • Class Name: datahub.ingestion.source.grafana.grafana_source.GrafanaSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Grafana, feel free to ping us on our Slack.