Orchestrations

Job orchestrations are the rollout layer above normal jobs.

Use a job when one reusable task tree should run against one resolved endpoint set. Use a job orchestration when several jobs must run in a controlled sequence across different scopes.

Typical examples:

• update Hyper-V hosts first, then WSUS hosts, then application VMs
• run a prep job on a server group, wait for reconnects to settle, then run a validation job on a second group
• fan out the same job to many endpoints in batches instead of launching every target at once

In the frontend, orchestrations live under Operations > Orchestrations. They have their own folder tree, independent from normal job folders and template folders.

Relationship to jobs

A job orchestration does not replace job definitions.

Instead, each orchestration stage points at an existing job_definition and adds rollout rules around it:

• which scope to resolve targets from
• which exact targets inside that scope to include
• how many targets to run at once
• whether a failed stage stops the orchestration or allows it to continue

At runtime Ordyn creates normal child job runs under the hood. The orchestration run tracks the stage order, batching, reconnect waiting, and overall stage status.

Execution model

An orchestration is a list of ordered stages.

Ordyn always evaluates stages from top to bottom. Only one stage is active at a time. The next stage starts only after the current stage reaches a terminal state or a configured continue-on-failure decision allows advancement.

For each stage Ordyn:

1. resolves endpoints from the configured scope
2. unions targets across all selected scope ids for that stage
3. deduplicates endpoints so the same endpoint is not launched twice in one stage
4. filters out endpoints where the referenced job is currently disabled
5. launches child job runs in batches according to batch_size
6. waits for each batch to finish before launching the next batch
7. optionally runs a post-stage readiness check before the next stage starts

If a child job succeeds but the endpoint disconnected during the run, the target moves into waiting_reconnect until the endpoint is seen again. This is what makes staged patch rollouts and reboot-heavy jobs behave predictably.

A readiness check is separate from reconnect waiting. It lets one stage wait for a different endpoint set, such as application VMs that should only be touched after a hypervisor stage has finished and the guests are connected again.

Stage fields

Each stage contains:

• name: operator-facing label for the stage
• job_definition_id: the reusable job definition to execute for each resolved endpoint
• scope_type: one of tenant, folder, group, or endpoint
• scope_ids: one or more ids of the selected scope type
• batch_size: how many endpoints from this stage may run concurrently
• failure_policy: stop or continue
• readiness_check: optional post-stage gate that waits for selected endpoints to be connected before the next stage starts

Scope resolution

scope_type and scope_ids always work together. A stage can target multiple folders, groups, tenants, or endpoints of the same type. Ordyn resolves all of them, merges the endpoints, and removes duplicates.

Examples:

• scope_type: "endpoint" with three endpoint ids means "run this stage only on these three endpoints"
• scope_type: "group" with two group ids means "run this stage on the union of both groups"
• scope_type: "folder" with several folder ids means "run this stage on all eligible endpoints inside those folders"

In the GUI editor this is the Scope target combobox with multi-select chips.

Batch size

batch_size is per stage.

• 1 means fully sequential endpoint execution inside the stage
• 2 means two endpoints may run in parallel, then the next two, and so on
• a larger value fans out more aggressively

This is the main control for staged or cautious rollouts.

Failure policy

failure_policy controls stage-to-stage advancement.

• stop: if any target in the stage fails, times out, or is aborted, the orchestration stops and the run fails
• continue: the stage is marked failed, but Ordyn still advances to the next stage after all batches in the current stage are finished

This setting does not change the child jobs themselves. It only controls orchestration flow after their results are known.

GUI editor

The orchestration editor is available under Operations > Orchestrations.

The GUI editor supports:

• creating and reordering stages
• selecting the referenced job definition per stage
• choosing a scope type
• selecting one or more scope targets of that type
• setting the batch size
• setting the stage failure policy
• switching between GUI and JSON editing

The form is full width because stage rows are usually wider than a normal job form.

JSON editor

Job orchestrations use the same versioned builder-document pattern as jobs, packages, and OS images.

The JSON editor works on the full orchestration document directly. Ordyn only enables switching back to GUI mode when the current JSON can round-trip into the structured editor without losing fields.

That means:

• known orchestration fields stay editable in both modes
• unsupported extra fields keep you in JSON mode until the document is simplified back to the supported shape

Current schema:

• schema_version must be 1
• kind must be job_orchestration

JSON shape

{
  "schema_version": 1,
  "kind": "job_orchestration",
  "name": "Apply Windows updates in waves",
  "description": "Update infrastructure first, then tenant workloads.",
  "stages": [
    {
      "id": "3c2be4d7-86ad-43d3-9f0a-8d1b6a4e4bb0",
      "name": "Hyper-V hosts",
      "job_definition_id": "019da5ce-c456-72c8-b418-9dce513cba01",
      "scope_type": "group",
      "scope_ids": [
        "019da5d4-8b18-73da-8bcb-c2647d8da101"
      ],
      "batch_size": 1,
      "failure_policy": "stop",
      "readiness_check": {
        "type": "endpoint_connected",
        "scope_type": "group",
        "scope_ids": [
          "019da5de-01b9-70a1-b1c6-0402812a1010"
        ],
        "timeout_seconds": 1800
      }
    },
    {
      "id": "33a73bde-7248-46be-8d1a-fbb1c8d6336a",
      "name": "WSUS hosts",
      "job_definition_id": "019da5ce-c456-72c8-b418-9dce513cba01",
      "scope_type": "group",
      "scope_ids": [
        "019da5d7-5d4c-72f2-b2cc-f6d255783777"
      ],
      "batch_size": 1,
      "failure_policy": "stop",
      "readiness_check": {
        "type": "endpoint_connected",
        "scope_type": "group",
        "scope_ids": [
          "019da5de-01b9-70a1-b1c6-0402812a1010"
        ],
        "timeout_seconds": 1800
      }
    },
    {
      "id": "5df1d1f7-9af6-4be9-97b1-e7374e38e7f6",
      "name": "Application VMs",
      "job_definition_id": "019da5ce-c456-72c8-b418-9dce513cba01",
      "scope_type": "group",
      "scope_ids": [
        "019da5df-3a67-7190-b777-499ad971b5f1",
        "019da5e1-59f4-70e2-9157-341ce6fb4f03"
      ],
      "batch_size": 3,
      "failure_policy": "continue"
    }
  ]
}

Example rollout pattern

A common safe rollout looks like this:

1. Stage 1 targets a small infrastructure group with batch_size: 1
2. Stage 2 targets a shared service group with batch_size: 1
3. Stage 3 targets workload machines with batch_size: 2 or 3
4. every stage uses the same update job definition, but on a different target set

This lets you author the update logic once and control only the rollout order in the orchestration.

Runtime behavior and statuses

Orchestration run statuses

An orchestration run can currently be:

• pending
• running
• succeeded
• failed

Stage statuses

A stage run can currently be:

• pending
• running
• waiting
• succeeded
• failed

waiting means the stage is currently blocked on one or more targets that are waiting for endpoint reconnect after a child job already reported a disconnect-sensitive success path.

Target statuses

A stage target can currently be:

• pending
• running
• waiting_reconnect
• succeeded
• failed
• timed_out
• aborted

The orchestration stores the linked child job run for each launched target so operators can drill into the exact job run that was created.

JSON editor behavior

The orchestration editor supports a GUI view and a JSON view. The JSON document contains the same orchestration stages, scopes, batch settings, and failure policy shown in the editor.

Validation rules

Current validation highlights:

• at least one stage is required
• every stage must reference an existing job_definition
• scope_type must be one of the supported scope enums
• scope_ids must contain at least one id
• every selected scope id must exist for the chosen scope type
• batch_size must be between 1 and 100
• failure_policy must be stop or continue

Use scope_ids for orchestration stage targets. Each stage must contain at least one selected scope ID.

Operational notes

• orchestration stages resolve targets at runtime, not when the document is saved
• only approved and enabled endpoints that the requester can resolve through the selected scope are launched
• if all resolved endpoints are filtered out, the stage fails with No enabled approved endpoints are currently available for this stage.
• job-level endpoint disables are respected inside orchestrations too
• the same endpoint can appear in different stages if that is how the rollout is authored
• within a single stage, duplicate endpoints from overlapping scope selections are launched only once

When to use jobs vs orchestrations

Use a plain job when:

• one task tree should run against one selected target set
• rollout order between groups does not matter
• you do not need per-stage batching or stop/continue rules

Use a job orchestration when:

• the same or different jobs must run in a deliberate stage order
• one target cohort must finish before the next cohort starts
• you want a cautious multi-wave rollout with controlled parallelism
• you need the operator-visible stage timeline and per-stage reconnect waiting model