Distinct@1

Node Distinct@1 removes duplicate objects from an array based on a unique property value. It keeps only the first occurrence of each unique value, filtering out subsequent duplicates.

Adapter Prerequisites

• General availability: All adapters support this node type.

Node Configuration

For fields path, targetPath, targetValueWriteMode, and targetValueKind, see Overview.

transformations:
  - type: Distinct@1
    path: $.items[*]              # Path to the array containing objects to deduplicate
    distinctValuePath: $.id       # Relative path to the property that defines uniqueness
    targetPath: $.uniqueItems     # Path where the deduplicated array should be stored

Parameters

Required Parameters

Parameter	Type	Description
path	string	JSON path to the source array containing objects to deduplicate
distinctValuePath	string	Relative path to the property within each object that defines uniqueness
targetPath	string	JSON path where the deduplicated array will be stored

Optional Parameters

Parameter	Type	Default	Description
targetValueWriteMode	string	Overwrite	How to write the result (Overwrite, Append, Prepend)
targetValueKind	string	Simple	The kind of value to write (Simple, Array)
documentMode	string	Extend	Document mode (Extend, Replace)

Supported Data Types

The distinctValuePath property must point to a simple type value. The following data types are supported for uniqueness comparison:

• Integer (long)
• Float (double)
• String
• Boolean
• DateTime

Deduplication Behavior

• The node processes only objects (JObject) within the source array
• The first occurrence of each unique value is kept; subsequent duplicates are removed
• Objects without a value at the distinctValuePath are skipped
• Values are compared by their typed representation (not string conversion)

Examples

Example 1: Remove Duplicate Sensors by ID

Remove duplicate sensor readings keeping only the first occurrence:

transformations:
  - type: Distinct@1
    path: $.readings[*]
    distinctValuePath: $.sensorId
    targetPath: $.uniqueReadings

Input:

{
  "readings": [
    { "sensorId": "S001", "value": 25.5, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S002", "value": 30.0, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S001", "value": 26.0, "timestamp": "2024-01-01T10:05:00Z" },
    { "sensorId": "S003", "value": 22.5, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S002", "value": 31.0, "timestamp": "2024-01-01T10:05:00Z" }
  ]
}

Output:

{
  "readings": [
    { "sensorId": "S001", "value": 25.5, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S002", "value": 30.0, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S001", "value": 26.0, "timestamp": "2024-01-01T10:05:00Z" },
    { "sensorId": "S003", "value": 22.5, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S002", "value": 31.0, "timestamp": "2024-01-01T10:05:00Z" }
  ],
  "uniqueReadings": [
    { "sensorId": "S001", "value": 25.5, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S002", "value": 30.0, "timestamp": "2024-01-01T10:00:00Z" },
    { "sensorId": "S003", "value": 22.5, "timestamp": "2024-01-01T10:00:00Z" }
  ]
}

Example 2: Deduplicate by Nested Property

Use a nested path to identify uniqueness:

transformations:
  - type: Distinct@1
    path: $.orders[*]
    distinctValuePath: $.customer.email
    targetPath: $.uniqueCustomerOrders

Input:

{
  "orders": [
    { "orderId": "O001", "customer": { "email": "john@example.com", "name": "John" } },
    { "orderId": "O002", "customer": { "email": "jane@example.com", "name": "Jane" } },
    { "orderId": "O003", "customer": { "email": "john@example.com", "name": "John" } },
    { "orderId": "O004", "customer": { "email": "bob@example.com", "name": "Bob" } }
  ]
}

Output:

{
  "orders": [
    { "orderId": "O001", "customer": { "email": "john@example.com", "name": "John" } },
    { "orderId": "O002", "customer": { "email": "jane@example.com", "name": "Jane" } },
    { "orderId": "O003", "customer": { "email": "john@example.com", "name": "John" } },
    { "orderId": "O004", "customer": { "email": "bob@example.com", "name": "Bob" } }
  ],
  "uniqueCustomerOrders": [
    { "orderId": "O001", "customer": { "email": "john@example.com", "name": "John" } },
    { "orderId": "O002", "customer": { "email": "jane@example.com", "name": "Jane" } },
    { "orderId": "O004", "customer": { "email": "bob@example.com", "name": "Bob" } }
  ]
}

Example 3: Deduplicate by Numeric Value

Remove duplicates based on an integer property:

transformations:
  - type: Distinct@1
    path: $.products[*]
    distinctValuePath: $.categoryId
    targetPath: $.uniqueCategories

Input:

{
  "products": [
    { "name": "Laptop", "categoryId": 1 },
    { "name": "Mouse", "categoryId": 1 },
    { "name": "Book", "categoryId": 2 },
    { "name": "Pen", "categoryId": 3 },
    { "name": "Notebook", "categoryId": 2 }
  ]
}

Output:

{
  "products": [
    { "name": "Laptop", "categoryId": 1 },
    { "name": "Mouse", "categoryId": 1 },
    { "name": "Book", "categoryId": 2 },
    { "name": "Pen", "categoryId": 3 },
    { "name": "Notebook", "categoryId": 2 }
  ],
  "uniqueCategories": [
    { "name": "Laptop", "categoryId": 1 },
    { "name": "Book", "categoryId": 2 },
    { "name": "Pen", "categoryId": 3 }
  ]
}

Example 4: In-Place Deduplication

Store the result back in the same location:

transformations:
  - type: Distinct@1
    path: $.events[*]
    distinctValuePath: $.eventType
    targetPath: $.events
    targetValueWriteMode: Overwrite

Input:

{
  "events": [
    { "eventType": "click", "timestamp": "2024-01-01T10:00:00Z" },
    { "eventType": "scroll", "timestamp": "2024-01-01T10:01:00Z" },
    { "eventType": "click", "timestamp": "2024-01-01T10:02:00Z" },
    { "eventType": "hover", "timestamp": "2024-01-01T10:03:00Z" }
  ]
}

Output:

{
  "events": [
    { "eventType": "click", "timestamp": "2024-01-01T10:00:00Z" },
    { "eventType": "scroll", "timestamp": "2024-01-01T10:01:00Z" },
    { "eventType": "hover", "timestamp": "2024-01-01T10:03:00Z" }
  ]
}

Use Cases

• Data Cleansing: Remove duplicate records from imported data
• Sensor Data: Keep only unique sensor readings when duplicates occur
• Event Processing: Filter duplicate events based on event type or ID
• User Data: Get unique users from a list of activities
• Category Extraction: Extract one representative item per category

Notes

• Order Preservation: The first occurrence of each unique value is retained; the order of unique items is preserved
• Null Handling: Objects without a value at distinctValuePath are excluded from the result
• Type Sensitivity: Values are compared using their native types (e.g., integer 1 is different from string "1")
• Empty Arrays: If the source array is empty or null, no output is written
• Original Data: The original source array remains unchanged unless targetPath points to the same location

Related Nodes

• Flatten@1: Flatten nested array structures
• Map@1: Transform array data by mapping multiple arrays into objects
• Join@1: Combine arrays based on matching keys