Skip to main content

Overview

In OctoMesh, data pipelines are integral to the Extract, Transform, Load (ETL) processes that ensure efficient data handling across distributed environments. These pipelines are categorized into Edge Pipelines and Mesh Pipelines, each executed by specific components within the system: Edge Adapters and Mesh Adapters respectively.

Edge Pipelines

Edge Pipelines operate within the Edge Adapters, which are deployed close to the data sources in the edge environments. These pipelines manage the initial stages of the data lifecycle, including data capture, preprocessing, and preliminary transformations. This localized processing ensures minimal latency and prepares data for more complex processing.

Key Features of Edge Pipelines:

Executed by Edge Adapters: Ensures close proximity processing to data sources.

  • Low latency: Fast data handling for immediate processing needs.
  • Local preprocessing: Initial data cleaning and transformation to ease the load on central systems.

Mesh Pipelines

Mesh Pipelines are executed by Mesh Adapters in centralized environments, such as cloud-based systems or data centers. They receive preprocessed data from Edge Pipelines and undertake extensive data transformation, integration, and aggregation tasks. These pipelines are robust, designed to handle large-scale data operations and provide a comprehensive view of enterprise data.

Key Features of Mesh Pipelines:

  • Executed by Mesh Adapters: Centralized processing for complex transformations.
  • Scalable architecture: Capable of managing increased data flows and complex processing requirements.
  • Advanced data management: Ensures data consistency and facilitates comprehensive analytics.

The diagram below incorporates the execution roles of Edge and Mesh Adapters in the ETL process:

Construction Kit Model

The construction kit of the model database is managing the data of adapter and pipelines that includes descriptions, state and relationships.