Skip to main content

OctoMesh - State-of-the-Art Data Infrastructure Platform

OctoMesh is a cutting-edge software platform designed to build robust data infrastructures and enable the creation of Data Meshes. Built on modern cloud-native technologies, OctoMesh runs on Kubernetes and consists of various specialized services. The platform combines open-source Core Services with licensed Pro Services, providing flexibility for different organizational needs. Data source and target connectivity is achieved through Adapters that can be deployed either in the cloud or at the edge.

Platform Overview​

Click to enlarge

Core Principles​

Data Mesh Architecture​

OctoMesh implements the Data Mesh paradigm, treating data as a product with:

  • Domain-oriented ownership: Each domain owns and operates its data products
  • Self-serve data platform: Teams can autonomously create and manage data products
  • Federated governance: Consistent standards across decentralized data products
  • Product thinking: Data treated as valuable products with clear ownership

Service Categories​

Open Source Core Services​

  • Asset Repository Service: Central management of data products and resources
  • Identity Service: Enterprise-grade authentication and authorization
  • Communication Controller: Secure message routing and device management
  • Time Series Service: Efficient handling of temporal data and metrics

Pro Services (Licensed)​

  • Bot Service: Advanced automation and workflow orchestration
  • Advanced Analytics: Machine learning and AI integration capabilities
  • Governance Service: Compliance, quality, and policy management
  • Monitoring Service: Comprehensive performance and health tracking

Open Source Edge Services​

  • Edge Adapter Management: Local deployment and management of adapters
  • Local Data Processing: Edge computing capabilities for real-time data handling
  • Offline Operation: Functionality in disconnected or low-connectivity environments
  • Lightweight Data Storage: Local caching and storage solutions

Architecture​

Click to enlarge

OctoMesh supports multi-tenancy, allowing different tenants to operate within a single instance of the application without sharing data. Each tenant's data is stored in a separate MongoDB database, ensuring data isolation and security. Additionally, tenants can be cascaded, which means that tenants can build upon each other, inheriting properties and configurations if required.

Key Architecture Components​

Construction Kit​

The Construction Kit is the core of OctoMesh's Data Mesh technology, enabling object-oriented data modeling with:

  • Attributes: Simple data types (string, number, boolean) or complex Records
  • Associations: Relationships between classes and data entities
  • Type: Define entity types with their attributes, associations and inheritance
  • Record: Structures for embedded documents within types, containing multiple attributes
  • Enum: Enumerations for controlled vocabularies and semantic values
  • Versioning: Manage schema evolution with version-controlled models

Runtime and Stream data​

Runtime Entities are instances of Construction Kit types that represent actual data objects in the system. They are created, read, updated, and deleted through the OctoMesh API and are stored in the underlying databases. Stream Entities represent time-series data, events, and metrics that are continuously generated by devices, applications, or processes. They are optimized for high-throughput ingestion and real-time querying.

Data Persistence and Distribution​

Two specialized databases handle different data types:

  • MongoDB: Master data and tenant isolation
  • CrateDB: Time-series data and real-time analytics

RabbitMQ facilitates asynchronous communication between services, ensuring scalability and reliability.

Deployment Flexibility​

OctoMesh supports multiple deployment models:

  • Cloud Deployment: Full platform on AWS, Azure, GCP or Rancher
  • Edge Deployment: Lightweight K3s for IoT and industrial environments
  • Hybrid Model: Combining cloud control with edge data collection

Core OctoMesh Services​

Core Services describes OctoMesh components that are installed centrally and provide essential platform functionalities.

Identity Service​

OctoMesh delivers a state-of-the-art identity service tailored for the Industrial Internet of Things (IIoT) sector. It offers a centralized authentication and authorization mechanism that interfaces with multiple directory services. Through OctoMesh's identity service, enterprises can establish secure and efficient connectivity across an array of devices, applications, and systems within the IIoT ecosystem.

Supported identity provider:

  • Open LDAP
  • Microsoft Active Directory
  • Microsoft Entra ID
  • Google Authentication
  • Meta Authentication
  • Microsoft Authentication

Asset Repository Service​

The Repository Service is a vital component of the data mesh architecture, designed to enhance data management and operability. This service focuses on the description, organization, and discovery of data products and resources, streamlining the workflow for data teams and stakeholders.

Data Products and Data Resources​

Data Products​

A data product in the context of the Repository Service is a logical grouping of related data assets, which may include data sets, code, or any other data-related artifacts. Data products are curated and maintained with a specific use case or business domain in mind, providing a packaged and consumable data utility.

Data Resources​

Data resources refer to the tangible elements stored within a system, such as databases, files, or any structured data container. These resources are the physical manifestation of data that can be accessed and utilized by data products.

There are two types of data resources:

  • Data Sets: Collections of data that are structured and stored in a way that allows
  • Data Files: Individual files that contain data, which can be in various formats such as CSV, JSON, XML, etc.
  • Data Streams: Continuous flows of data that are generated in real-time, often used for monitoring or event-driven applications.

Features of the Asset Repository Service​

  • Interface for Access: The service presents a user-friendly interface for users to access both data products and data resources. This interface is designed to support a wide range of data operations, from simple queries to complex data processes.
  • Support for Data Governance: The Repository Service is built with data governance at its core. It provides the necessary tools and protocols to manage data effectively, ensuring compliance with internal and external regulations and policies.
  • Centralized Organization and Description: The central Asset Repository Service allows for the systematic organization and detailed description of data products and resources. This centralized approach ensures that assets are easily identifiable and accessible within the data mesh.
  • Enhanced Discoverability: Through the use of metadata, indexing, and search functionalities, data assets are made discoverable within the organization. This enhances cross-team collaboration and reduces the time to locate and utilize data assets.
  • Metadata, Guidelines, and Quality Metrics: The service not only stores data but also provides rich metadata, usage guidelines, and quality metrics for each data product. This ensures that users are well-informed about the data they access, facilitating better decision-making and maintaining high data quality standards.
  • Consistent Use and Reuse: With a focus on enabling consistent use, the Repository Service ensures that data products are designed for reuse across different scenarios, maximizing the value extracted from each data asset.
  • Real-Time Analysis: Enables the examination of data as it is captured, providing insights instantly and allowing for prompt decision-making.
  • Trend Detection: Utilizes advanced algorithms to identify significant trends within data, aiding in understanding past behaviors and adjusting operational strategies accordingly.
  • Forecasting: Employs predictive analytics to project future patterns and outcomes, such as energy production rates, based on historical data. This feature is invaluable for planning and optimization in various industrial scenarios.
  • Processing Capabilities: Utilizes standard SQL syntax for data processing, making it accessible for those familiar with traditional database query languages. Maintains fast query performance even during high-frequency data writes, ensuring that the system's responsiveness remains consistent.

Bot Service​

Octo Bots are dynamic, automated assistants integrated within the OctoMesh platform. They are designed to streamline a variety of routine and complex tasks, fostering an environment of efficiency and automation. By leveraging Octo Bots, users can significantly reduce manual workload and enhance process execution within the OctoMesh ecosystem.

Capabilities and Automation​

Octo Bots are equipped to handle an extensive range of activities, including but not limited to:

  • Data Preparation: Streamlining the process of getting data ready for analysis.
  • Data Cleaning: Ensuring data quality by removing errors or correcting values in datasets.
  • Anonymization/Pseudonymization: Protecting personal data by obscuring identifying details.
  • Data Aggregation: Combining data from different sources to construct a consolidated summary.
  • Mass Notifications/Newsletters: Sending out large-scale communications efficiently.
  • Batch Imports: Integrating data from various formats such as XML, CSV, and JSON.
  • Archive Creation: Packaging files into compressed formats for storage or transfer.
  • Web Hooks Triggering: Automating the execution of scripts or events via external web services.
  • User Management: Deleting users and managing user-related data systematically.
  • Charting: Generating various forms of visual data representations.
  • Temporary Files Management: Automating the cleanup of non-essential files.
  • Automatic Reporting: Producing reports on a regular basis without manual intervention.
  • Database Maintenance: Ensuring the databases are optimized and functioning correctly.

Communication Service​

The Octo Communication Controller is a pivotal component of the OctoMesh ecosystem, functioning as a central hub for managing and securing communications. It integrates seamlessly with Octo Plugs and Sockets, ensuring a robust exchange of messages between the central OctoMesh platform and a wide array of connected devices and applications.

Core Functionality​

  • Regulation of Communications: The Com Controller serves as a gatekeeper, regulating the flow of messages to maintain order and prevent congestion in network communication channels.

  • Security Measures: It employs stringent encryption protocols to ensure that all communications are protected from unauthorized access, thus maintaining the highest possible transmission security standards.

  • High Connectivity: Capable of handling connections with millions of devices and backend solutions, including databases, dashboards, and Octo Apps, the Com Controller is exceptionally scalable.

  • Flexible Integration: The adaptability of Octo Plugs and Sockets makes it possible to connect almost any device or system to the Com Controller, thereby extending the capabilities of the OctoMesh platform.

Key Features​

  • Secure Channels: It establishes secure communication channels for both sending and receiving data from IoT devices.
  • Device Authentication: Manages device identities to authenticate and control access within the network.
  • Automated Provisioning: Automates the registration and provisioning process for devices, reducing the need for manual setup.
  • Edge Component Management: Monitors and maintains the integrity of local edge components, ensuring their proper functioning within the network.
  • Network Expansion: Facilitates the extension of the data network by allowing the transfer of Octo Plugs and services to edge devices with ease.

Edge OctoMesh Services​

To facilitate communication with external systems, OctoMesh uses specialized components known as adapters. These adapters are divided into two types:

  • Plugs: Adapters that handle incoming data from external systems.
  • Sockets: Adapters that manage outgoing data to external systems.

These adapters ensure that data flows seamlessly between OctoMesh and external applications or services, maintaining system integrity and performance.

Click to enlarge

Edge OctoMesh Services are deployed on edge devices using a lightweight Kubernetes distribution called K3s. This allows for efficient data collection and processing close to the source, reducing latency and bandwidth usage. When communicating with cloud-based systems, Edge OctoMesh Services may also operate on central Kubernetes clusters, depending on the deployment architecture.

Communication Operator​

The management of adapters is handled by the Communication Operator. This tool allows administrators to configure and manage adapters specific to each tenant, enabling customized integration and data handling that aligns with the unique requirements of each tenant.