Skip to main content

Introduction

In the realm of OctoMesh, adapters and pipelines play a crucial role as the connective tissue between the OctoMesh platform and external data sources and services. These small, but powerful pieces of software are designed to facilitate communication and data exchange across a diverse set of endpoints, including APIs, file systems, databases, message brokers, and other custom or standard protocols. To cater to different architectural needs and deployment scenarios, OctoMesh distinguishes between two main types of adapters: Edge Adapters and Mesh Adapters.

We use the term "Plugs" when data is being pushed into the OctoMesh platform and "Sockets" when data is being pulled from the OctoMesh platform. From the technical perspective, Plugs and Sockets are the same, but the terminology helps to differentiate between the two directions of data flow.

Communication Controller Service​

The Communication Controller Service is a service that is responsible for managing the communication between the OctoMesh platform and Communication Operators and Adapters. It provides a set of APIs that allow the Adapters to register, send and receive messages, and manage their configuration. The Communication Controller Service is a critical component of the OctoMesh platform, enabling seamless communication and data exchange between the platform and the Adapters.

Communication Operator​

Communication Operators are responsible for managing the lifecycle of the Adapters, including creating, updating, and deleting Adapters. Every Communication Operator is tenant-independent and may manage one or more tenant-specific Pools. A Pool is a collection of Adapters that are managed together. The Communication Operator is responsible for deploying the Adapters to the desired environment and managing their lifecycle. Pools usually have a separate namespace in Kubernetes.

Edge Adapters​

Edge Adapters are designed to operate locally, close to the data sources or within the same local network. This proximity allows for rapid data exchange and reduced latency, making Edge Adapters ideal for scenarios where immediate data processing and response are critical. They are typically deployed on-premises, in edge computing environments, or directly on devices that collect or generate data. By processing data locally, Edge Adapters also help in maintaining data privacy and security by minimizing the exposure of sensitive information to external networks.

Key Features of Edge Adapters:

  • Local execution: Runs close to data sources for quick access and processing.
  • Reduced latency: Ideal for time-sensitive applications where speed is critical.
  • Enhanced security: Limits data transmission across networks, protecting sensitive information.

Mesh Adapters​

Mesh Adapters, in contrast, are designed to run in centralized locations such as cloud environments or centralized data centers. These adapters handle tasks that are suitable for centralization, such as aggregating data from multiple Edge Adapters, performing large-scale data processing, and managing data storage and distribution at a higher level. Mesh Adapters allow for the scalability and robustness needed in large-scale deployments, supporting extensive data operations and providing a global view of the data mesh.

Key Features of Mesh Adapters:

  • Centralized management: Simplifies the oversight of data flows and adapter performance across the mesh.
  • Scalability: Facilitates the expansion of data operations without compromising performance.
  • Global data integration: Integrates data from multiple edge locations, enabling comprehensive analytics and decision-making.
  • Deployment and Integration

Deploying and configuring these adapters involves understanding the specific requirements of your data architecture and the operational context of your data sources. Mesh and Edge Adapters are typically deployed through our automated provisioning tools and can be managed through central management consoles like Admin Panel or octo-cli, that allow administrators to oversee and control multiple adapters with ease, providing tools for monitoring, troubleshooting, and analytics.

Data Pipelines​

Adapters are executing data pipelines that are responsible for moving data between different data sources and destinations. These pipelines are orchestrated by the OctoMesh platform, which ensures that data is transferred securely, efficiently, and reliably across the mesh. Data pipelines can be configured to handle various data formats, transformations, and routing rules, enabling complex data workflows to be automated and optimized.

A typical Data Pipeline consists of the following components:

  • Edge Pipeline: The part of the pipeline that processes data at the edge, close to the data source, and publishes it to the mesh.
  • Mesh Pipeline: The part of the pipeline that aggregates, processes, and distributes data across the mesh.

Mesh Adapters are executing Mesh Pipelines, while Edge Adapters are executing Edge Pipelines. The OctoMesh platform orchestrates the execution of these pipelines, ensuring that data is moved efficiently and securely across the mesh.