Skip to main content

Introduction

In the realm of OctoMesh, adapters and pipelines play a crucial role as the connective tissue between the OctoMesh platform and external data sources and services. These small, but powerful pieces of software are designed to facilitate communication and data exchange across a diverse set of endpoints, including APIs, file systems, databases, message brokers, and other custom or standard protocols. To cater to different architectural needs and deployment scenarios, OctoMesh distinguishes between two main types of adapters: Edge Adapters and Mesh Adapters.

We use the term "Plugs" when data is being pushed into the OctoMesh platform and "Sockets" when data is being pulled from the OctoMesh platform. From the technical perspective, Plugs and Sockets are the same, but the terminology helps to differentiate between the two directions of data flow.

Communication Controller Service​

The Communication Controller Service is a service that is responsible for managing the communication between the OctoMesh platform and Communication Operators and Adapters. It provides a set of APIs that allow the Adapters to register, send and receive messages, and manage their configuration. The Communication Controller Service is a critical component of the OctoMesh platform, enabling seamless communication and data exchange between the platform and the Adapters.

Communication Operator​

Communication Operators are responsible for managing the lifecycle of a Pool's workloads, including creating, updating, and deleting them. A workload is either an Adapter (a Plug or Socket that connects OctoMesh to an external system) or an Application (a containerized business application that runs alongside the platform and can be exposed publicly through an ingress). Every Communication Operator is tenant-independent and may manage one or more tenant-specific Pools. A Pool is a collection of workloads that are managed together. Pools usually have a separate namespace in Kubernetes.

A Communication Operator can run at the edge (on a remote K3s cluster close to the data sources) or centrally (alongside the Communication Controller in the central cluster, where it can also deploy Applications). Regardless of mode, the operator does not build container deployments directly: it deploys every workload from a Helm chart that it pulls from a Helm chart repository referenced by the workload (repository URL, chart name and chart version) and installs via helm upgrade --install. Chart values are layered — cluster-wide defaults from the operator, the workload's own base values, and per-workload overrides from Refinery Studio.

Edge Adapters​

Edge Adapters are designed to operate locally, close to the data sources or within the same local network. This proximity allows for rapid data exchange and reduced latency, making Edge Adapters ideal for scenarios where immediate data processing and response are critical. They are typically deployed on-premises, in edge computing environments, or directly on devices that collect or generate data. By processing data locally, Edge Adapters also help in maintaining data privacy and security by minimizing the exposure of sensitive information to external networks.

Key Features of Edge Adapters:

  • Local execution: Runs close to data sources for quick access and processing.
  • Reduced latency: Ideal for time-sensitive applications where speed is critical.
  • Enhanced security: Limits data transmission across networks, protecting sensitive information.

Mesh Adapters​

Mesh Adapters, in contrast, are designed to run in centralized locations such as cloud environments or centralized data centers. These adapters handle tasks that are suitable for centralization, such as aggregating data from multiple Edge Adapters, performing large-scale data processing, and managing data storage and distribution at a higher level. Mesh Adapters allow for the scalability and robustness needed in large-scale deployments, supporting extensive data operations and providing a global view of the data mesh.

Key Features of Mesh Adapters:

  • Centralized management: Simplifies the oversight of data flows and adapter performance across the mesh.
  • Scalability: Facilitates the expansion of data operations without compromising performance.
  • Global data integration: Integrates data from multiple edge locations, enabling comprehensive analytics and decision-making.
  • Deployment and Integration

Deploying and configuring these adapters involves understanding the specific requirements of your data architecture and the operational context of your data sources. Mesh and Edge Adapters are typically deployed through our automated provisioning tools and can be managed through central management consoles like Refinery Studio or octo-cli, that allow administrators to oversee and control multiple adapters with ease, providing tools for monitoring, troubleshooting, and analytics.

Data Pipelines​

Adapters are executing data pipelines that are responsible for moving data between different data sources and destinations. These pipelines are orchestrated by the OctoMesh platform, which ensures that data is transferred securely, efficiently, and reliably across the mesh. Data pipelines can be configured to handle various data formats, transformations, and routing rules, enabling complex data workflows to be automated and optimized.

A typical Data Pipeline consists of the following components:

  • Edge Pipeline: The part of the pipeline that processes data at the edge, close to the data source, and publishes it to the mesh.
  • Mesh Pipeline: The part of the pipeline that aggregates, processes, and distributes data across the mesh.

Mesh Adapters are executing Mesh Pipelines, while Edge Adapters are executing Edge Pipelines. The OctoMesh platform orchestrates the execution of these pipelines, ensuring that data is moved efficiently and securely across the mesh.