IPipelineScratchSpace
Namespace: Meshmakers.Octo.Sdk.Common.EtlDataPipeline.Execution
Per-execution scratch space for large binary artifacts.
Nodes that produce or consume large binaries (merged PDFs, ZIP archives, …) should write them to a scratch file and pass a small handle token through the JSON data context instead of carrying the bytes as base64. Base64 held in a .NET UTF-16 string is roughly 2.66x the binary size and every large copy lands on the Large Object Heap, which fragments and triggers OutOfMemoryException long before the process limit is reached (prod-1 BMD handover export, AB#4642).
Handles are opaque tokens — the space owns the on-disk location, so nodes never
see (or can influence) a filesystem path. The whole space is deleted when the
pipeline execution ends (see EtlDataOrchestrator.ExecutePipelineAsync's
finally block); a startup sweep (PipelineScratchSpace.SweepStaleDirectories(TimeSpan))
removes directories left behind by a hard crash / OOM that skipped the finally.
public interface IPipelineScratchSpace : IAsyncDisposable
Implements IAsyncDisposable
Methods
CreateFile(String)
Creates a new, empty scratch file and returns its opaque token. The optional
extension is only cosmetic (helps debugging); it never
affects where the file lives.
string CreateFile(string extension)
Parameters
extension String
Returns
OpenWrite(String)
Opens a scratch file for writing (creates/truncates it).
Stream OpenWrite(string token)
Parameters
token String
Returns
OpenRead(String)
Opens a scratch file for reading.
Stream OpenRead(string token)
Parameters
token String
Returns
GetLength(String)
Returns the length in bytes of a scratch file.
long GetLength(string token)
Parameters
token String
Returns
Exists(String)
Returns whether a scratch file with this token exists in this space.
bool Exists(string token)
Parameters
token String