Parallel Execution

Understand how Runner executes packages in parallel using DAG-based dependency resolution and Data references.

Overview

Bluebricks Runner uses a DAG (Directed Acyclic Graph) to determine package execution order. Packages execute in parallel when possible, with dependencies resolved through Data references that link package outputs to inputs.

This approach enables:

• Efficient execution - Independent packages run concurrently

• Output-driven deployment - Only deploy what's needed downstream

• Clear dependency tracking - Explicit relationships between packages

• Automatic ordering - Runner determines optimal execution sequence

Data References

What Are Data References?

Data references allow packages to use outputs from other packages as inputs:

Data.<packageId>.<outputKey>

Components:

• Data. - Reserved prefix for cross-package references

• <packageId> - Unique ID of the source package

• <outputKey> - Name of the output from source package

Example:

{
  "packages": [
    {
      "id": "vpc_module",
      "props": {
        "region": {
          "value": "Props.region"
        }
      }
    },
    {
      "id": "eks_cluster",
      "props": {
        "vpc_id": {
          "value": "Data.vpc_module.vpc_id"
        },
        "subnet_ids": {
          "value": "Data.vpc_module.private_subnet_ids"
        }
      }
    }
  ]
}

In this example:

• eks_cluster depends on vpc_module

• Runner executes vpc_module first

• vpc_id and private_subnet_ids outputs flow to eks_cluster inputs

Data References in Expressions

Data references can be used in expressions with other values:

{
  "props": {
    "bucket_name": {
      "value": "'my-app-' + Data.vpc_module.vpc_id + '-logs'"
    },
    "enable_logging": {
      "value": "Props.create_bucket || Data.s3_module.bucket_exists"
    }
  }
}

Runner evaluates these expressions at runtime after dependencies complete.

Plan vs Apply Resolution

Data references behave differently during plan and apply:

During Plan:

• Shows that value will change

• Indicates value will be available after execution

• Dependencies identified but values not yet calculated

During Apply:

• Actual values resolved

• Passed to dependent packages

• Used in expressions and property resolution

DAG Structure

Dependency Graph

Runner builds a DAG (Directed Acyclic Graph) representing package relationships:

User Properties
      ↓
Analyze Packages
      ↓
Build Dependency Graph
      ↓
Execute in Order
      ↓
Collect Outputs

How it works:

• Identifies all packages in blueprint

• Analyzes Data references to find dependencies

• Creates execution order that respects dependencies

• Executes packages when their dependencies complete

• Passes outputs to dependent packages

Example DAG

Consider this blueprint structure:

{
  "packages": [
    {
      "id": "launch_template",
      "props": {
        "name": { "value": "Props.template_name" }
      }
    },
    {
      "id": "s3_bucket",
      "props": {
        "bucket_name": {
          "value": "'logs-' + Data.launch_template.launch_template_id"
        }
      }
    },
    {
      "id": "conditional_resources",
      "props": {
        "template_id": {
          "value": "Data.launch_template.launch_template_id"
        }
      }
    }
  ]
}

Resulting DAG:

┌─────────────────────────────────────────────────────────┐
│                     ProvidedProps                       │
│  { template_name: "my-template", region: "us-east-1" }│
└────────────────────┬────────────────────────────────────┘
                     │
                     ↓
┌────────────────────────────────────────────────────────┐
│          launch_template (no dependencies)             │
│  Type: Terraform                                       │
│  Outputs: launch_template_id                           │
└────────────────────┬───────────────────────────────────┘
                     │
                     ↓
        ┌────────────┴────────────┐
        │                         │
        ↓                         ↓
┌─────────────────┐   ┌──────────────────────┐
│   s3_bucket     │   │ conditional_resources│
│  Type: Blueprint│   │  Type: Blueprint     │
│  Depends:       │   │  Depends:            │
│  - launch_temp  │   │  - launch_temp       │
│  Props:         │   │  Props:              │
│  bucket_name =  │   │  template_id =       │
│    Data.launch  │   │    Data.launch       │
│        .id      │   │        .id           │
└─────────────────┘   └──────────────────────┘

Execution order:

1. launch_template executes (no dependencies)

2. s3_bucket and conditional_resources execute in parallel (both depend only on launch_template)

Output-Driven Execution

Only Deploy What's Needed

Runner only executes packages whose outputs are required downstream:

Example:

{
  "outs": {
    "vpc_id": {
      "value": "Data.vpc_module.vpc_id"
    }
  },
  "packages": [
    {
      "id": "vpc_module",
      "...": "..."
    },
    {
      "id": "monitoring_module",
      "...": "..."
    }
  ]
}

Execution:

• vpc_module EXECUTES (output referenced in outs)

• monitoring_module SKIPS (output not referenced anywhere)

This ensures:

• Efficient execution - No unused resources deployed

• Clear dependency tracking - DAG only includes what's needed

• Proper resource management - Resources created only if used

Conditional Execution

Use expressions to conditionally execute packages:

{
  "packages": [
    {
      "id": "s3_bucket",
      "props": {
        "create_bucket": {
          "value": "Props.enable_logging"
        }
      }
    }
  ],
  "outs": {
    "bucket_name": {
      "value": "Props.enable_logging ? Data.s3_bucket.bucket_name : 'none'"
    }
  }
}

If enable_logging is false:

• Package may still execute (depends on internal logic)

• But output can conditionally return different values

Data Flow Pattern

Props Flow Down, Data Flows Up

Bluebricks uses a hierarchical data flow model:

Props (Properties):

• Flow DOWN from parent to child packages

• Set at deployment time

• Referenced with Props.<key>

Data (Outputs):

• Flow UP from child packages to parent

• Calculated at runtime

• Referenced with Data.<packageId>.<key>

Example Flow

┌─────────────────────────────────────────────┐
│           Root Blueprint                     │
│                                              │
│  ProvidedProps:                              │
│    region: "us-east-1"                       │
│    environment: "production"                 │
│                                              │
│  Props flow DOWN ↓                          │
│                                              │
│  ┌────────────────────────────────────────┐ │
│  │  VPC Module (Terraform)                │ │
│  │  Props:                                │ │
│  │    region = Props.region               │ │
│  │    env = Props.environment             │ │
│  │                                        │ │
│  │  Outputs:                              │ │
│  │    vpc_id: "vpc-123"                   │ │
│  │    subnet_ids: ["subnet-1", ...]       │ │
│  └────────────────────────────────────────┘ │
│                ↑ Data flows UP              │
│                                              │
│  ┌────────────────────────────────────────┐ │
│  │  EKS Module (Terraform)                │ │
│  │  Props:                                │ │
│  │    vpc_id = Data.vpc_module.vpc_id     │ │
│  │    subnets = Data.vpc_module.subnet_ids│ │
│  │    region = Props.region               │ │
│  │                                        │ │
│  │  Outputs:                              │ │
│  │    cluster_name: "my-cluster"          │ │
│  │    cluster_endpoint: "https://..."     │ │
│  └────────────────────────────────────────┘ │
│                ↑ Data flows UP              │
│                                              │
│  Root Outputs:                               │
│    vpc_id = Data.vpc_module.vpc_id          │
│    cluster_name = Data.eks_module.cluster...│
└─────────────────────────────────────────────┘

Data Reference Chains

Data references can form chains:

{
  "packages": [
    {
      "id": "vpc",
      "...": "..."
    },
    {
      "id": "subnet_blueprint",
      "packages": [
        {
          "id": "subnet_module",
          "props": {
            "vpc_id": { "value": "Data.vpc.vpc_id" }
          }
        }
      ],
      "outs": {
        "subnet_id": { "value": "Data.subnet_module.subnet_id" }
      }
    },
    {
      "id": "instance",
      "props": {
        "subnet_id": {
          "value": "Data.subnet_blueprint.subnet_id"
        }
      }
    }
  ]
}

Data flow:

1. vpc outputs vpc_id

2. subnet_module receives vpc_id via Data.vpc.vpc_id

3. subnet_blueprint exposes subnet_id via Data.subnet_module.subnet_id

4. instance receives subnet_id via Data.subnet_blueprint.subnet_id

Parallel Execution Optimization

Maximizing Parallelism

Design blueprints to maximize parallel execution:

Avoid:

{
  "packages": [
    { "id": "A" },
    { "id": "B", "depends_on_A_output": "Data.A.value" },
    { "id": "C", "depends_on_B_output": "Data.B.value" }
  ]
}

Execution: A → B → C (sequential)

Prefer:

{
  "packages": [
    { "id": "A" },
    { "id": "B", "uses_props": "Props.value" },
    { "id": "C", "uses_props": "Props.value" }
  ]
}

Execution: A, B, C (all parallel if A has no dependents)

Dependency Minimization

Reduce cross-package dependencies:

Less efficient:

{
  "packages": [
    { "id": "vpc" },
    { "id": "s3_logs", "props": { "vpc_id": "Data.vpc.vpc_id" } },
    { "id": "eks", "props": { "vpc_id": "Data.vpc.vpc_id" } }
  ]
}

Both s3_logs and eks must wait for vpc.

More efficient:

{
  "packages": [
    { "id": "s3_logs" },
    { "id": "eks", "props": { "log_bucket": "Data.s3_logs.bucket_name" } }
  ]
}

s3_logs and eks can run in parallel if eks doesn't actually need log_bucket immediately.

Group Related Resources

Package related resources together to reduce dependencies:

{
  "packages": [
    {
      "id": "network_stack",
      "packages": [
        { "id": "vpc" },
        { "id": "subnets" },
        { "id": "route_tables" }
      ]
    },
    {
      "id": "compute_stack",
      "packages": [
        { "id": "launch_templates" },
        { "id": "auto_scaling_groups" }
      ]
    }
  ]
}

Internal package dependencies don't block other stacks from executing in parallel.

Best Practices

Data Reference Design

DO:

• Use Data references for runtime values only

• Reference outputs that will definitely be available

• Design packages with clear output contracts

• Use descriptive output keys

DON'T:

• Reference outputs from packages that may not execute

• Create circular dependencies (A depends on B, B depends on A)

• Use Data references for static configuration (use Props)

Dependency Management

DO:

• Minimize cross-package dependencies

• Design for parallel execution where possible

• Use Props for configuration, Data for runtime values

• Group related resources in packages

DON'T:

• Create unnecessary dependencies

• Design sequential execution when parallel would work

• Mix Props and Data references unnecessarily

Expression Usage

DO:

• Use expressions for dynamic value calculation

• Combine Data references with strings and Props

• Use conditional logic when appropriate

• Keep expressions readable

DON'T:

• Create complex nested expressions

• Use expressions for simple value passing

• Rely on side effects in expressions

Testing Parallel Execution

Test your blueprint's execution behavior:

# Plan-only to see execution plan
bricks install ./my-blueprint --collection dev --plan-only

# Check dependency graph
# Look for packages that can execute in parallel

# Monitor actual execution
bricks install ./my-blueprint --collection dev
# Watch for parallel job creation in logs

See also

• Creating Blueprints -- blueprint structure, inputs, outputs, and data references

• expr-lang reference -- expression language used in blueprint values

• Blueprint Composition Patterns -- common composition strategies