Skip to main content

GetStreamData@1

Node GetStreamData@1 reads rows directly from a stream data archive. It is the ad-hoc counterpart of GetQueryById@1: archive, columns, time range, filters and sorting are configured on the node itself, so no persisted query entity is required.

The node writes a query result (columns and rows) to targetPath, which QueryResultToMarkdownTable@1 can consume directly. Optionally the node also reports whether the requested time range is completely covered by data (gap detection).

Adapter Prerequisites​

Node Configuration​

For fields targetPath, targetValueWriteMode, and targetValueKind, see Overview. Field path is not used in this node.

transformations:
  - type: GetStreamData@1
    archiveRtId: 68a1f0c5de73e7b175575401 # Runtime ID of the archive to read from (must be activated)
    columns: # Attribute paths to project, empty reads the whole archive
      - Energy
      - DataQuality
    wellKnownNames: # Restrict to source entities with these well-known names
      - METER-4711
    wellKnownNamesPath: $.meters # Alternative: read the well-known names from the payload
    rtIds: # Restrict to these source entities
      - 68a2b1c4de73e7b175575402
    rtIdsPath: $.rtIds # Alternative: read the runtime IDs from the payload
    fieldFilters: # Additional filters, combined with AND
      - attributePath: DataQuality
        operator: GreaterEqualsThan
        comparisonValue: 90
    sortOrders: # Sort order, using the column names as they appear in the result
      - attributeName: Timestamp
        sortOrder: Ascending
    skip: 0 # Number of rows to skip
    take: 100 # Number of rows to take
    from: 2026-07-01T00:00:00 # Start of the time range (UTC)
    fromPath: $.from # Alternative: read the start of the time range from the payload
    to: 2026-08-01T00:00:00 # End of the time range (UTC)
    toPath: $.to # Alternative: read the end of the time range from the payload
    limit: 10000 # Row cap, independent of skip/take
    limitPath: $.limit # Alternative: read the row cap from the payload
    targetPath: $.data # Path where the rows are stored in the payload

Parameters​

ParameterTypeRequiredDescription
archiveRtIdstringYesRuntime ID of the archive to read from. The archive must be activated
columnsarray of stringNoAttribute paths to project (e.g. Energy, Amount.Value) or the name of a formula column. Empty reads the whole archive
wellKnownNamesarray of stringNoRestricts the result to source entities with these well-known names
wellKnownNamesPathstringNoJSONPath alternative to wellKnownNames; accepts a single value, an array, or a multi-match path
rtIdsarray of stringNoRestricts the result to these source entities
rtIdsPathstringNoJSONPath alternative to rtIds
fieldFiltersarray of FieldFilterWithPathDtoNoAdditional filters on projected or standard columns, combined with AND
sortOrdersarray of SortOrderDtoNoSort order of the returned rows
skipintegerNoNumber of rows to skip (pagination)
takeintegerNoNumber of rows to take (pagination)
fromdate/timeNoStart of the time range (UTC). Unset leaves the range open at the start
fromPathstringNoJSONPath to the start of the time range in the payload
todate/timeNoEnd of the time range (UTC). Unset leaves the range open at the end
toPathstringNoJSONPath to the end of the time range in the payload
limitintegerNoRow cap, must be greater than zero. Independent of skip / take
limitPathstringNoJSONPath to the row cap in the payload
gapsTargetPathstringNoPath where the coverage report is written. Setting it turns gap detection on
expectedIntervaldurationNoInterval the gap counts are expressed in, e.g. PT15M. Defaults to the archive's declared period
gapsOnlybooleanNoReport the gaps only and skip reading the data. Requires gapsTargetPath
maxGapScanRowsintegerNoRow cap for the coverage scan (default 200000), must be greater than zero

For the available filter operators and sort directions, see GetRtEntitiesByType@1.

Precedence and time zone​

Per value the literal wins over its JSONPath variant: from takes precedence over fromPath, to over toPath, limit over limitPath, wellKnownNames over wellKnownNamesPath and rtIds over rtIdsPath.

note

Timestamps are read as UTC. A value written without a time-zone offset (2026-07-01T00:00:00) is interpreted as UTC, not as the local time of the adapter host, so the queried window never shifts with the server's time zone.

A path that resolves to nothing leaves the value unset and logs a warning. A value that is present but not a date/time (or not an integer for limitPath) fails the node rather than silently widening the range.

Result Shape​

The result carries a leading Timestamp column, followed by the projected columns. On a windowed archive (time-range or rollup) WindowStart and WindowEnd are inserted after Timestamp — those archives have no timestamp column of their own, the time axis is the end of the window.

Leaving columns unset projects every data column the archive declares and adds a WellKnownName column ahead of them, so the minimal configuration — just an archiveRtId — already returns usable data:

Timestamp | WindowStart | WindowEnd | WellKnownName | Energy | DataQuality

Formula (computed) columns are included and are addressed by their name, not by an attribute path. A column whose backfill has not committed yet is absent and appears once the backfill completes.

When columns is set, the list is honoured exactly as given — no WellKnownName is added, and it can be requested there like any other column (rtWellKnownName). The time axis and the row window are the exception: Timestamp, WindowStart and WindowEnd are emitted for every row anyway, so naming one of them in columns does not add a second, identical column.

Column names in sortOrders and fieldFilters​

Use the names as they appear in the result: Timestamp, WindowStart / WindowEnd (windowed archives only), WellKnownName, or any column the archive declares. The node translates these onto the physical storage columns.

A name that is neither a result header nor a column of the archive fails the node, and the error lists the names that would have worked. Sorting or filtering by WindowStart on a raw archive is refused for the same reason — such an archive has no row window.

Gap Detection​

Setting gapsTargetPath makes the node additionally check whether the queried range is actually covered by data and write a report there:

transformations:
  - type: GetStreamData@1
    archiveRtId: 68a1f0c5de73e7b175575401
    from: 2026-07-01T00:00:00
    to:   2026-07-02T00:00:00
    expectedInterval: PT15M
    targetPath:     $.data
    gapsTargetPath: $.gaps

Gaps are reported per source entity — a missing quarter-hour on one meter must not be hidden by another meter delivering data. The report looks like this:

from: 2026-07-01T00:00:00Z
to:   2026-07-02T00:00:00Z
interval: PT15M
seriesCount: 2
seriesWithGapsCount: 1
isComplete: false
series:
  - rtId: 68a2b1c4de73e7b175575402
    wellKnownName: METER-4711
    expectedIntervals: 96
    presentIntervals: 94
    missingIntervals: 2
    hasOverlaps: false
    isComplete: false
    gaps:
      - from: 2026-07-01T12:30:00Z
        to:   2026-07-01T13:00:00Z
        duration: PT30M
        durationSeconds: 1800
        missingIntervals: 2
  - rtId: 68a2b1c4de73e7b175575403
    wellKnownName: METER-4712
    expectedIntervals: 96
    presentIntervals: 96
    missingIntervals: 0
    hasOverlaps: false
    isComplete: true
    gaps: []

Durations are written both as an ISO-8601 string (duration) and as seconds (durationSeconds), so they are readable as well as computable in downstream nodes. Because the report as a whole carries isComplete, a following If@1 can branch on a single field.

How it works: every stored window in the range is clamped to that range, overlapping and adjacent windows are merged, and whatever the merge does not cover is a gap. This needs no declared period and copes with windows of differing length. A known interval (expectedInterval, otherwise the archive's period) only adds the interval counts; without one the gaps are still reported as time ranges and the node warns once.

caution

Three limitations are worth knowing:

  • Gap detection requires a windowed archive (time-range or rollup) and both time boundaries. A raw archive stores single timestamps and has no interval coverage to judge — the node refuses rather than inventing an answer.
  • The scan runs its own query, deliberately separate from the data query, whose limit / skip / take would hide rows and make the scan report gaps that are not there. maxGapScanRows (default 200000) bounds it; exceeding the cap fails the node instead of reporting from a truncated scan. For scale: a year of quarter-hourly data is roughly 35,000 rows per entity.
  • An entity that delivered nothing at all is invisible to a coverage scan — it simply has no rows. Where rtIds or wellKnownNames names the expected entities, each one without rows is reported as a full-range gap instead; otherwise the limitation stands and the node logs it.

Overlapping windows are not gaps and do not fail anything — the storage concept allows them — but they are flagged per series as hasOverlaps and warned about once, because a sum over them counts the overlapping section twice.

Usage Example​

The following pipeline reads a day of measurements for one meter, checks the day for completeness, and only renders a table when the data is complete:

triggers:
  - type: FromHttpRequest@1
    path: /streamdata-report
    method: POST
transformations:
  - type: GetStreamData@1
    archiveRtId: 68a1f0c5de73e7b175575401
    columns:
      - Energy
      - DataQuality
    wellKnownNames:
      - METER-4711
    from: 2026-07-01T00:00:00
    to:   2026-07-02T00:00:00
    sortOrders:
      - attributeName: Timestamp
        sortOrder: Ascending
    expectedInterval: PT15M
    targetPath:     $.data
    gapsTargetPath: $.gaps
  - type: QueryResultToMarkdownTable@1
    path: $.data
    targetPath: $.table
tip

Use gapsOnly: true for a cheap completeness check ahead of an expensive step — the node then reports the gaps without reading the data itself.

Use Cases​

  • Ad-hoc archive queries: Read stream data without creating a persisted query entity first
  • Reporting: Retrieve the measurements of a period, sorted and filtered, for a table or an export
  • Data quality checks: Verify that a billing period is covered completely before it is invoiced
  • Monitoring: Detect missing deliveries per meter or sensor and trigger a follow-up action
  • Backfill control: Use gapsOnly to determine which ranges still need to be re-imported

Limitations​

Downsampling and resolution-aware rollup selection are not part of this node. Both are covered by GetQueryById@1 with a persisted downsampling query. GetStreamData@1 always reads exactly the archive it was configured with, so the numbers never depend on an archive choice made behind the caller's back.

See Also​