Stream Data Archives

The Stream Data Archives section of the Refinery Studio lets you create and manage the time-series archives that capture data into CrateDB. Each archive defines a CK type, a curated list of attribute paths, and a status — and the studio walks you through every state transition (Activate, Disable, Enable, Retry, Delete) with confirmation dialogs.

For the underlying mechanics see the tech guide: Stream Data Archives.

Prerequisites

• StreamData must be enabled for the tenant. If you see an empty state with a single "Enable StreamData for this tenant" button, the tenant flag is off — click it to enable (requires write permissions on the tenant).
• The StreamDataAdmin role is required for every action on this page. Reader / Writer accounts will see the navigation entry only if their role implies admin (AdminPanelManagement).
• The archives feature is hidden entirely when the instance-level StreamData:Enabled flag is false — that's an operator-level decision, not something the tenant admin can change from the UI.

Accessing the Archives Page

Navigate to Repository → Archives in the tenant menu. The page shows every archive on the active tenant — raw, rollup, and time-range — with a status badge and per-row actions.

Archives List

Column	Meaning
Name	rtWellKnownName of the archive entity.
Type	Raw, Rollup, or Time-Range.
Target CK Type	The CK type the archive captures (Industry.Energy/EnergyMeter, …).
Status	Created / Activated / Disabled / Failed. Hover for the last transition timestamp; on Failed, the underlying error code is also shown.
Columns	Count of captured attribute paths.
Actions	Context-aware: Activate, Disable, Enable, Retry, Delete, Copy ID, Edit (Created only).

The action menu is status-aware — only legal transitions are enabled:

Current status	Available actions
Created	Edit · Activate · Delete
Activated	Disable · Delete (no Edit — schema is frozen)
Disabled	Enable · Delete
Failed	Retry Activation · Delete

All destructive actions (Disable, Activate, Delete, Retry) open a confirmation dialog before the mutation runs.

Creating an Archive

Three entry points sit in the page toolbar:

• New Archive — a RawArchive for instant measurements (one row per timestamp).
• New Rollup — a RollupArchive derived from an existing archive. Picks a source first (raw or rollup-of-rollup) and then the aggregations.
• New Time-Range — a TimeRangeArchive for externally pre-aggregated data with explicit [from, to) windows (EDA reports, smart meters, weather APIs).

Raw Archive

1. Click New Archive.
2. Enter a name (rtWellKnownName) — short and stable; the archive is referenced by rtId from pipelines and queries, but the name is what you'll see in dropdowns.
3. Pick the Target CK Type. The picker auto-completes from the CK model loaded in the tenant. Inheritance is supported: an archive on a base type captures every concrete derived type with its own table.
4. Use the Attribute Path Picker to select the columns to capture:
   • Tree view of the CK type's attributes; click into records to expand.
   • Per row, toggle Required (the column becomes NOT NULL) and Indexed (default on; disable only for columns that are read but never filtered).
   • Array marker [*] is shown inline for collection-typed attributes (readings[*].value).
   • At least one column must be selected.
5. Save. The archive is created in status Created. The CrateDB table is not provisioned yet.
6. Back on the list, click Activate on the new row. The studio runs activateArchive server-side; on success the badge flips to Activated and the data plane (inserts + queries) is open.

Rollup Archive

1. Click New Rollup. The source picker lists every existing archive on the tenant — raw or rollup. Chaining rollup-of-rollup is supported.
2. Pick a source. The next screen pre-fills the source archive metadata in read-only fields:
   • Target CK type (inherited from source)
   • Source columns (available paths for aggregation)
3. Configure the rollup:
   • Bucket size in milliseconds (e.g. 3600000 for hourly).
   • Bucket alignment — FixedSize for engineering data, CalendarDay / Iso8601Week / CalendarMonth / CalendarYear for energy and EDA workflows.
   • Watermark lag in milliseconds — how far behind real time the orchestrator stays before closing a bucket. Buy yourself some headroom for late-arriving data.
   • Aggregations — one or more { source path, function, target column } rows. Functions: Avg, Min, Max, Sum, Count. Avg is stored as sum + count columns so chained rollups stay numerically correct.
4. Save. The studio derives TargetCkTypeId (from the source) and Columns (from the aggregations) server-side via createRollupArchive — you never set those by hand.
5. Activate as for a raw archive. From then on, the rollup orchestrator ticks hourly (configurable) and writes one row per closed bucket.

Time-Range Archive

1. Click New Time-Range.
2. Enter name, target CK type, and the column list (same picker as for raw archives).
3. Optionally set a Period (TimeSpan) — advisory only; the studio uses it to pick a default time range when rendering query previews. Not enforced at insert time.
4. Save and activate.
5. External producers push rows via SaveTimeRangeStreamDataInArchive@1 (pipeline) or POST /api/v1/streamData/archives/{rtId}/insertTimeRange (REST).

Editing an Archive

• While Created, the archive is fully editable — click the name or Edit to reopen the form. Columns, target CK type, name, indexing, required flags — everything can change.
• Once Activated, the schema is frozen. The form opens read-only with a banner explaining the immutability rule. The only mutable bits at that point are the name and (for rollups) the freeze / rewind controls below.
• For a breaking schema change, create a new archive with a new name and migrate consumers. There is no in-place schema migration by design.

Lifecycle Actions

Activate

• Created → Activated: provisions the CrateDB table and opens the data plane.
• Disabled → Activated: re-validates column paths against the current CK model (CK migrations between disable and re-enable can break paths — failure shows up here, not silently).
• Failed → Activated (Retry): idempotent — re-runs CREATE TABLE IF NOT EXISTS. Safe to retry repeatedly.

Disable

Activated → Disabled. Inserts and queries are rejected; the CrateDB table and its data are preserved. Use Disable to quiesce an archive before a destructive operation or to investigate a downstream issue without losing history.

Delete

Drops the CrateDB table and soft-deletes the archive entity (rtState = Archived). The confirmation dialog reads:

Delete archive '<rtId>'? The CrateDB table will be dropped and historical data lost. This action cannot be undone.

Delete is rejected with a notification if the archive is a RawArchive that has at least one active RollupArchive referencing it. Drop the rollups first, or freeze them and check what depends on them.

Retry Activation

Only available on Failed. The status hover tooltip carries the underlying error code (often a DDL or path-resolution failure). After fixing the root cause — usually a CK-model gap — click Retry.

Rollup-Specific Actions

The rollup edit form exposes three operator controls:

Control	Effect
Freeze	Sets FrozenUntil. The orchestrator stops producing buckets whose bucketEnd falls in the frozen range. Monotonic — a new until must be ≥ the current value.
Unfreeze	Clears FrozenUntil. Idempotent. The Accept gaps toggle is recorded for audit but the gap-detection guard is a follow-up.
Rewind watermark	Resets LastAggregatedBucketEnd to the chosen timestamp (truncated to the bucket boundary). Subsequent ticks re-aggregate the rewound range. Destructive — previously committed rows in that range are temporarily out of sync until the orchestrator catches up.

Typical workflows:

• Backfilling raw data: Freeze every dependent rollup → run the import → rewind each rollup's watermark to the start of the back-fill range → unfreeze. The orchestrator re-aggregates the affected buckets on its next tick.
• Fixing a misconfigured aggregation: There is no in-place "edit aggregations". Delete the rollup, create a new one with the corrected spec, activate.

Reading the Status Badge

Badge color (default theme)	Status	What it means
Gray	Created	Not provisioned yet. Inserts and queries are rejected.
Green	Activated	CrateDB table exists, data plane open.
Yellow	Disabled	Table preserved, inserts and queries rejected. Re-enable to resume.
Red	Failed	Activation DDL failed. Hover for the error code. Retry once you've fixed it.

Hover the badge to see the last transition timestamp. On Failed, the tooltip also shows the underlying error code.

Failure Modes

Symptom	Likely cause
Activate returns ArchivePathInvalidException	One of the captured paths cannot be resolved against the current CK model. Edit the archive (if still Created) or fix the CK model.
Activate returns ArchiveColumnTypeUnsupportedException	An attribute's CK type cannot be mapped to a CrateDB column type. Choose a different attribute or restructure the CK type.
Delete returns RollupSourceInUseException	Active rollups reference this raw archive. Drop or freeze the rollups first.
Retry Activation keeps failing	The error is environmental (Crate unreachable, schema permissions) — check the asset-repo logs.
Archive page is empty, no "Enable" button	Instance-level StreamData:Enabled is false. This is an operator decision (appsettings).
Buttons are disabled even though the role looks right	Token doesn't carry the role claim. Re-log in; check the identity service's API resource configuration.

Where the Data Lives

Archive metadata (the Archive runtime entities, status, columns, rollup state) lives in the tenant's MongoDB database. Time-series rows live in CrateDB, in a schema named after the tenant. Deleting a tenant drops both. mongodump --db=<tenant> captures the metadata but not the CrateDB rows — for time-series backups use CrateDB's own snapshot tools.

See Also

• Stream Data Archives (Tech Guide) — full mechanics, schema, lifecycle, API surface
• Stream Data Access (Intro) — querying the captured data
• octo-cli — Stream Data — the same operations from the command line