Lifecycle Management in BanyanDB

BanyanDB uses a hot-warm-cold data tiering strategy to optimize storage costs and query performance. The lifecycle management feature automatically migrates data between storage tiers based on configured policies.

Overview

A dedicated lifecycle agent (an independent Go process) is responsible for:

• Identifying data eligible for migration based on lifecycle stages.
• Extracting data from the source node.
• Migrating data to designated target nodes.
• Removing the original data after a successful migration.

This process ensures data flows correctly through hot, warm, and cold stages as defined in your group configurations.

Prerequisite: node discovery must be configured. The lifecycle agent resolves the target nodes of every stage by listing all data nodes through the cluster’s node registry and filtering them by the stage’s node_selector. In the default --node-discovery-mode=none setting the registry is empty, so the agent cannot find any destination to migrate data to and the run will be a no-op. Configure dns or file mode on the lifecycle agent using the same source of truth as the rest of the cluster; see the node discovery documentation for the full list of --node-discovery-* flags.

Data Migration Process

The lifecycle agent performs the following steps:

1. Identify Groups for Migration:

   • Connect to the metadata service and list all groups.
   • Filter groups configured with multiple lifecycle stages.
   • Skip groups already processed (using progress tracking).

2. Take Snapshots:

   • Request a snapshot from the source data node.
   • Retrieve the stream, measure and trace data directory paths.

3. Setup Query Services:

   • Create read-only services for streams, measures and traces using the snapshot paths.

4. Process Each Group:

   • Determine the current stage based on node labels.
   • Identify target nodes for the next lifecycle stage.
   • For each stream, measure or trace:
     • Query data from the current stage.
     • Transform data into write requests.
     • Send write requests to target nodes.
   • Delete the source data after a successful migration.

5. Progress Tracking:

   • Track migration progress in a persistent progress file.
   • Enable crash recovery by resuming from the last saved state.
   • Clean up the progress file once the migration is complete.

Configuration

Lifecycle stages are defined within the group configuration using the LifecycleStage structure.

Example Configuration

metadata:
  name: example-group
catalog: CATALOG_MEASURE
resource_opts:
  shard_num: 4
  segment_interval:
    unit: UNIT_DAY
    num: 1
  ttl:
    unit: UNIT_DAY
    num: 7
  stages:
    - name: warm
      shard_num: 2
      segment_interval:
        unit: UNIT_DAY
        num: 7
      ttl:
        unit: UNIT_DAY
        num: 30
      node_selector: "type=warm"
      close: true
    - name: cold
      shard_num: 1
      segment_interval:
        unit: UNIT_DAY
        num: 30
      ttl:
        unit: UNIT_DAY
        num: 365
      node_selector: "type=cold"
      close: true

In this example:

• Data starts in the default stage with 4 shards and daily segments.
• After 7 days, data moves to the “warm” stage with 2 shards and weekly segments.
• After 30 days in the warm stage, data transitions to the “cold” stage with 1 shard and monthly segments.
• Data is purged after 365 days in the cold stage.

Command-Line Usage

The lifecycle command offers options to customize data migration:

lifecycle \
  --grpc-addr 127.0.0.1:17912 \
  --stream-root-path /path/to/stream \
  --measure-root-path /path/to/measure \
  --trace-root-path /path/to/trace \
  --node-labels type=hot \
  --progress-file /path/to/progress.json \
  --schedule @daily \
  --node-discovery-mode file \
  --node-discovery-file-path /etc/banyandb/nodes.yaml

Command-Line Parameters

Handling Orphan (Deleted-Schema) Data

A source segment can hold data for a measure or stream whose schema was later deleted from the registry (for example, when the upstream metric definition is renamed or removed). Such rows can no longer be written to a target — their schema no longer exists — so the migration treats them as orphan rows.

Orphan handling applies to the measure and stream catalogs only. Trace migration is not wired for per-series orphan handling: a trace group has a single schema, so a deleted trace schema is a whole-group concern rather than a per-series orphan within a surviving group.

On the row-replay path (used when a source segment spans multiple target segments), the migration detects orphan rows, skips them so the rest of the group still migrates, and then — unlike rows skipped because of a series-index gap — lets the source segment be deleted normally. What happens to the orphan rows themselves is controlled by --migration-orphan-policy:

• archive (default): each orphan row is written as one JSON line so an operator can recover it. The archive lives in the --migration-orphan-archive-subdir subdirectory under each catalog’s own root path (so measure orphans land under <measure-root-path>/<subdir>/… and stream orphans under <stream-root-path>/<subdir>/…); the catalog is therefore not a path level (it is recorded inside every record and manifest instead). The per-part JSONL is gzip-compressed on disk (the rows are highly repetitive, so this is ~37× smaller — e.g. 269 MB of raw JSONL becomes ~7 MB). Files are laid out as:

  <catalog-root-path>/<subdir>/<group>/seg-<segment-suffix>/shard-<id>/part-<part-id>.jsonl.gz
  <catalog-root-path>/<subdir>/<group>/seg-<segment-suffix>/manifest.json
  

  Each line is self-describing (decoded from the part’s own column types, with no registry schema needed): it carries the group, catalog, the measure/stream name (always under the JSON key measure, regardless of catalog — the catalog field disambiguates), source location, series id, entity, timestamp (both RFC3339 timestamp and epoch-nanos timestamp_unix_nano), tags, and — for measures — version, indexed_tags, and fields; streams carry element_id instead of those three and omit version entirely (rather than emitting a misleading zero). The per-segment manifest.json (plain JSON) indexes which deleted measures/streams were archived and their row counts. A part with no orphan rows writes no file; re-running a part rewrites its file idempotently.

• discard: orphan rows are dropped; nothing is written to disk.

Orphan handling is not a migration error — the source segment is migrated and deleted normally. Instead of appearing in the report’s errors buckets, it is summarized under an orphans section that records, per catalog → group → deleted measure/stream, how many rows were archived or discarded:

"orphans": {
  "policy": "archive",
  "measure": { "sw_metricsHour": { "meter_..._hour": 1234 } },
  "stream":  { "<group>": { "<stream>": 56 } }
}

(The counts accumulate across resume cycles. If the process is hard-killed mid-group before the group finishes, that run’s not-yet-flushed counts are lost from the report — the archive files and their manifest.json are unaffected.)

Reading the archive

The manifest.json files are plain text — read them directly. The per-part data is gzip-compressed JSON Lines:

# inspect one part's rows (Linux: zcat; macOS: gzcat)
gunzip -c <measure-root-path>/<subdir>/<group>/seg-20260601/shard-0/part-000000000000003b.jsonl.gz | jq .

# project a few fields
gunzip -c part-*.jsonl.gz | jq -c '{measure, ts:.timestamp, v:.fields.value.value}'

# search without fully decompressing (Linux)
zgrep "meter_banyandb_instance_disk_usage" part-*.jsonl.gz

# what was archived for a segment (counts per deleted measure)
jq '{total_rows, total_series, measures:[.measures[]|{measure,rows}]}' .../seg-20260601/manifest.json

Limitations and operations notes:

• The archive lives under each catalog’s own root path (--measure-root-path / --stream-root-path), so it shares the durability of the volume that already holds the catalog data — no separate path to provision. --migration-orphan-archive-subdir only changes the subdirectory name within that root.
• Detection only happens on the row-replay path. On the chunk-sync path (a source segment that maps to a single target segment), whole part files are copied without per-row decoding, so orphan rows pass through to the target. They then occupy target-stage disk until that stage’s TTL expires (e.g. up to the warm-stage TTL), and whether the copy succeeds depends on the receiver’s part-acceptance behavior for a now-unregistered schema.
• The archive directory is not cleaned up automatically. Prune it periodically once the data is no longer needed.

Automatic Behavior on Warm and Cold Nodes

When a data node matches a lifecycle stage (i.e., its labels match a stage’s node_selector), two behaviors are automatically configured:

• Rotation is disabled: The rotation task (which pre-creates segments on hot nodes) is not started on warm or cold nodes. Segments are created on demand by the lifecycle migration process, ensuring correct segment boundaries for multi-day intervals.
• Retention is disabled on non-last stages: TTL-based segment deletion is disabled on all stages except the last one. Data removal on intermediate stages is managed by the lifecycle migration process, not by the built-in retention mechanism.

If a node’s labels do not match any stage, both rotation and retention are disabled as a safety measure.

The lifecycle agent also accepts the full suite of --node-discovery-* flags so it can enumerate every candidate target node in the cluster. At a minimum you need --node-discovery-mode and either --node-discovery-dns-srv-addresses (DNS mode) or --node-discovery-file-path (file mode). The set of nodes visible to the lifecycle agent must include every warm/cold data node that any stage’s node_selector could match — use the same discovery configuration that the liaison nodes use. See the node discovery documentation for the complete reference.

Best Practices

1. Node Labeling:
   • Use clear, consistent labels that reflect node capabilities and roles.
2. Sizing Considerations:
   • Allocate more shards to hot nodes for higher write performance.
   • Optimize storage by reducing shard count in warm and cold stages.
3. TTL Planning:
   • Set appropriate TTLs based on data access patterns.
   • Consider access frequency when defining stage transitions.
4. Migration Scheduling:
   • Run migrations during off-peak periods.
   • Monitor system resource usage during migrations.
5. Progress Tracking:
   • Use a persistent location for the progress file to aid in recovery.

Example Workflow

1. Setup nodes with appropriate labels:

   Every node (liaison, hot, warm, cold, and the lifecycle agent itself) must point at the same node-discovery source so that the lifecycle agent and liaison nodes see the identical topology. The example below uses file mode with a shared /etc/banyandb/nodes.yaml; see the node discovery documentation for DNS mode.

   # Liaison node
   banyand liaison \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   
   # Hot data node
   banyand data \
     --node-labels type=hot \
     --grpc-port 17912 \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   
   # Warm data node
   banyand data \
     --node-labels type=warm \
     --grpc-port 18912 \
     --node-discovery-mode=file \
     --node-discovery-file-path=/etc/banyandb/nodes.yaml
   

2. Create a group with lifecycle stages.

bydbctl connects the liaison node to create a group with lifecycle stages:

   bydbctl group create -f - <<EOF
metadata:
   name: example-group
catalog: CATALOG_MEASURE
resource_opts:
   shard_num: 4
   segment_interval:
      unit: UNIT_DAY
      num: 1
   ttl:
      unit: UNIT_DAY
      num: 7
   stages:
      - name: warm
      shard_num: 2
      segment_interval:
         unit: UNIT_DAY
         num: 7
      ttl:
         unit: UNIT_DAY
         num: 30
      node_selector: "type=warm"
      close: true
EOF

3. Run the lifecycle agent:

lifecycle connects to the hot node (as its snapshot source via --grpc-addr) and uses node discovery to locate the warm/cold target nodes:

lifecycle \
  --node-labels type=hot \
  --grpc-addr 127.0.0.1:17912 \
  --stream-root-path /data/stream \
  --measure-root-path /data/measure \
  --trace-root-path /data/trace \
  --progress-file /data/progress.json \
  --node-discovery-mode=file \
  --node-discovery-file-path=/etc/banyandb/nodes.yaml

4. Verify migration:
   • Check logs for progress details.
   • Query both hot and warm nodes to ensure data availability.

This process automatically migrates data according to the defined lifecycle stages, optimizing your storage usage while maintaining data availability.