Workflows

Workflows¶

ocid implements several key workflows for distributing OCI images in a peer-to-peer manner. This document describes each workflow in detail, including the steps involved and the interactions between components.

📤 Publish Flow¶

When a user pushes an image to ocid (e.g., podman push localhost:5050/app:1.0), the following steps occur:

sequenceDiagram
    participant P as podman
    participant R as OCI Registry (ocid)
    participant S as Blob Store
    participant G as Gossip Topic

    P->>R: POST /v2/app/blobs/uploads/ (start upload)
    R->>S: Create upload session
    R-->>P: 202 Accepted (upload UUID)

    P->>R: PATCH /v2/app/blobs/uploads/<uuid>?digest=sha256:... (upload layer)
    R->>S: Write chunk to temp file
    R-->>P: 202 Accepted

    P->>R: PUT /v2/app/blobs/uploads/<uuid>?digest=sha256:abc123 (complete upload)
    R->>S: Verify SHA256, compute BLAKE3
    R->>S: Move to blobs/ (iroh-blobs FsStore)
    R->>S: Index blob (SHA256 → BLAKE3)
    R-->>P: 201 Created

    P->>R: PUT /v2/app/manifests/1.0 (upload manifest)
    R->>R: Resolve "app" → (me, app)
    R->>S: Check all referenced blobs present
    R->>S: Store manifest blob
    R->>R: Build ReleasePayload (+ known referrers)
    R->>R: Sign release with publisher key
    R->>S: Put release (index/releases/<me>/app/1.0.json)
    R->>G: Broadcast Announcement on topic(me)
    R-->>P: 201 Created

Steps Explained¶

1. Upload Blobs:
2. podman starts an upload session for each layer/config/blob.
3. ocid writes chunks to a temporary file.
4. On completion, ocid verifies the SHA256 digest, computes the BLAKE3 hash, and moves the blob to the iroh-blobs store.

5. The blob is indexed in index/digests/<sha256>.json (maps SHA256 → BLAKE3).

6. Upload Manifest:

7. podman pushes the manifest (e.g., app:1.0).
8. ocid resolves the repository name (app) to (publisher=me, name=app).
9. ocid checks that all blobs referenced by the manifest are present.

10. ocid stores the manifest blob (same process as layers).

11. Sign and Publish Release:

12. ocid builds a ReleasePayload with the manifest and all referenced blobs.
13. ocid signs the payload with the publisher's Ed25519 key.
14. The signed release is stored in index/releases/<me>/app/1.0.json.

15. ocid broadcasts an Announcement on the publisher's gossip topic (topic(me)).

16. Referrers (OCI 1.1):

17. If the manifest has a subject field (e.g., for signatures), ocid:
    1. Stores the referrer manifest.
    2. Records it under index/referrers/<subject>/<hash>.json.
    3. Re-signs and re-publishes all local releases whose manifest is the subject.

🔄 Replication Flow¶

When a node receives a gossip announcement for a release it doesn’t have (or doesn’t have completely), it fetches the release from a peer:

sequenceDiagram
    participant A as Node A (Publisher)
    participant B as Node B (Follower)
    participant G as Gossip Topic(A)

    A->>G: Broadcast Announcement(publisher=A, name=app, tag=1.0, digest=sha256:abc123)
    G->>B: Receive Announcement
    B->>B: Check policy (follow A, last:2)
    B->>B: select_wanted(known ∪ announced) → wanted?
    B->>A: sync GetRelease(publisher=A, name=app, tag=1.0)
    A-->>B: Release (signed JSON bytes)
    B->>B: Verify signature (Ed25519)

    loop Each BlobRef not present
        B->>A: iroh-blobs download(BLAKE3:def456)
        A-->>B: Verified stream (BAO format)
        B->>B: Verify SHA256 + size
        B->>B: Index blob (SHA256 → BLAKE3)
        B->>B: Pin blob in iroh-blobs
    end

    B->>B: Put release (index/releases/A/app/1.0.json)
    B->>B: enforce_window → Prune oldest, unpin blobs

Steps Explained¶

1. Receive Announcement:
2. Node B receives a gossip announcement from Node A’s topic (topic(A)).

3. The announcement contains a ReleaseSummary (publisher, name, tag, digest, timestamp).

4. Check Policy:

5. Node B checks if the release matches its policy:

   • follow A: Subscribe to topic(A) and fetch all new releases from A.
   • seed app: Subscribe to topic(A) and fetch all releases for app.
   • pin app:1.0: Always keep app:1.0 (never prune).
   • last:2: Keep only the 2 newest releases for app.

6. Fetch Release:

7. If the release is wanted, Node B sends a sync.GetRelease request to Node A.
8. Node A returns the signed JSON bytes of the release.

9. Node B verifies the signature using Node A’s public key.

10. Download Blobs:

11. For each BlobRef in the release that Node B doesn’t have:

    • Node B downloads the blob from Node A using iroh-blobs (BLAKE3 hash).
    • The blob is streamed with incremental verification (BAO format).
    • Node B verifies the SHA256 digest and size against the signed release.
    • The blob is indexed and pinned in iroh-blobs.

12. Store and Enforce:

13. The release is stored in index/releases/A/app/1.0.json.
14. Node B runs enforce_window to prune releases outside the policy window.

📥 Pull Flow (On-Demand)¶

When a user pulls an image that isn’t available locally (e.g., podman pull localhost:5050/A/app:1.0), ocid fetches it from a peer on demand:

sequenceDiagram
    participant P as podman
    participant C as OCI Registry (Node C)
    participant X as Peers (Neighbors, Known, Publisher)

    P->>C: GET /v2/A/app/manifests/1.0
    C->>C: Resolve "A/app:1.0" → (publisher=A, name=app, tag=1.0)
    C->>C: Check local store → Not found

    loop Candidate peers until found
        C->>X: sync GetRelease(publisher=A, name=app, tag=1.0)
        X-->>C: Release | None
    end

    C->>X: Download blobs (shuffled providers)
    C->>C: Verify, index, store release (cache)
    C-->>P: Manifest (+ Docker-Content-Digest)

    P->>C: GET /v2/A/app/blobs/sha256:abc123 (Range supported)
    C-->>P: Stream blob from store

Steps Explained¶

1. Resolve Repository:
2. Node C resolves A/app:1.0 to (publisher=A, name=app, tag=1.0).

3. If A is a DNS publisher name, it resolves to the pinned key.

4. Check Local Store:

5. Node C checks if it has the release locally.

6. If not, it proceeds to fetch from peers.

7. Fetch from Peers:

8. Node C iterates through candidate peers (neighbors, known peers, publisher).
9. For each peer, it sends a sync.GetRelease request.

10. The first peer to return the release wins.

11. Download Blobs:

12. Node C downloads all blobs referenced by the release from shuffled providers (any peer that has them).

13. Blobs are verified (SHA256 + size) and stored in the cache.

14. Return to Client:

15. Node C returns the manifest to podman with the Docker-Content-Digest header.
16. podman then requests individual blobs (with Range support).

📜 Policy and Retention¶

ocid uses a policy-based retention system to manage which releases are kept locally. The policy is defined in policy.toml:

 1[[follow]]
 2publisher = "z6MkqRYqQ..."
 3mode = "last:2"
 4
 5[[seed]]
 6ref = "app"
 7mode = "full"
 8
 9[[pin]]
10ref = "app:1.0"

Policy Workflow¶

flowchart LR
    subgraph rules["policy.toml"]
        f["[[follow]] publisher, mode"]
        s["[[seed]] ref, mode"]
        st["[[seed]] ref:tag"]
        p["pin = [ref:tag]"]
    end

    f --> w
    s --> w
    st --> w
    p --> w

    w["Window(publisher, name)<br/>full | last:N | tags"] --> sel["select(known releases)<br/>newest first, tag tie-break"]
    sel --> keep["wanted set"]

    keep --> enforce["enforce_window"]
    enforce --> prune["Prune releases<br/>outside window"]
    prune --> unpin["Unpin blobs"]

Modes¶

Rules¶

1. Multiple rules for one image combine to the widest window.

2. Example: follow A with last:2 + seed app with full → keep all releases for app.

3. Pins are never pruned.

4. Example: pin app:1.0 → app:1.0 is always retained, even if outside the window.

5. Own releases are never pruned.

6. Releases published by the local node are always kept.

7. Cache is temporary.

8. Releases fetched on-demand (not via policy) are kept for gc_grace_secs (default: 30 days).

Enforcement¶

flowchart TD
    win["enforce_window"] --> rr["Remove release records<br/>(local only, not propagated)"]
    gc["GC (periodic / ocictl gc)"] --> ret["Retained = own ∪ pinned ∪ in-window ∪ cache < grace"]
    ret --> rr
    rr --> sweep["Sweep: Unpin blob:<sha256> if no retained release references"]
    sweep --> ib["iroh-blobs GC deletes unpinned data"]

🌐 Gossip and Sync¶

ocid uses iroh-gossip for peer discovery and announcement propagation. Each publisher has its own gossip topic, and nodes subscribe to topics for publishers they follow/seed/pin.

Gossip Topics¶

flowchart TB
    subgraph swarm["Swarm Topic<br/>blake3('ocid/swarm/v1')"]
        A1[A]
        B1[B]
        C1[C]
    end

    subgraph ta["Topic(A)<br/>blake3('ocid/publisher/v1/' || A)"]
        A2[A]
        B2["B (follows A)"]
    end

    subgraph tc["Topic(C)<br/>blake3('ocid/publisher/v1/' || C)"]
        C2[C]
        B3["B (pins C/x:1)"]
    end

    %% Connections
    A1 <--> B1
    A1 <--> C1
    B1 <--> C1

    A2 <--> B2
    C2 <--> B3

Topic Rules¶

1. Swarm Topic:
2. Every node joins the swarm topic.

3. Carries no announcements; only supplies neighbors (NeighborUp → inventory sync) and bootstrap peers.

4. Publisher Topics:

5. A node subscribes to its own topic and to the topic of every publisher its policy follows, seeds, or pins.
6. Announcements are broadcast only on the publisher's topic.

7. Nodes receive exactly what they can act on (based on their policy).

8. Peer Introduction:

9. A newly reachable peer (swarm neighbor, mDNS, ocictl connect) is introduced to every subscribed topic.

10. Peers not on a topic ignore the join.

11. Announcement Validation:

12. Announcements from a different publisher than the topic’s owner are dropped.

🗑️ Garbage Collection¶

ocid performs garbage collection (GC) to clean up unpinned blobs and releases outside the policy window.

GC Flow¶

flowchart TD
    gc["GC (periodic / ocictl gc)"] --> ret["Retained = own ∪ pinned ∪ in-window ∪ cache < grace"]
    ret --> sweep["Sweep: Unpin blob:<sha256> if no retained release references"]
    sweep --> ib["iroh-blobs GC deletes unpinned data"]

    %% Annotations
    click gc "https://ocid.dev/architecture/workflows#garbage-collection" _blank

GC Rules¶

1. Retained Blobs:
2. Blobs referenced by own releases.
3. Blobs referenced by pinned releases.
4. Blobs referenced by in-window releases.

5. Blobs referenced by cache releases (fetched within gc_grace_secs).

6. Sweep:

7. Unpin blobs not referenced by any retained release.

8. iroh-blobs GC deletes unpinned blobs.

9. In-Flight Protection:

10. Downloads hold a temp tag in iroh-blobs to prevent race conditions.

11. Fetch and GC are serialized by an RwLock (fetch = read, GC/prune = write).

12. Manual GC:

13. Run ocictl gc to trigger GC manually.
14. Use ocictl gc --force to ignore the grace period.

🔄 Inventory Sync¶

When a new peer is discovered (via swarm topic or mDNS), ocid performs an inventory sync to catch up on missed releases:

sequenceDiagram
    participant A as Node A
    participant B as Node B
    participant G as Swarm Topic

    G->>B: NeighborUp(Node A)
    B->>A: sync Inventory
    A-->>B: Inventory(releases: [A/app:1.0, A/app:2.0])
    B->>B: select_wanted(known ∪ inventory) → [A/app:2.0]
    B->>A: sync GetRelease(A, app, 2.0)
    A-->>B: Release (signed JSON)
    B->>B: Verify signature
    B->>A: iroh-blobs download(BLAKE3:...)
    A-->>B: Verified stream
    B->>B: Verify SHA256 + size, index, pin blob
    B->>B: Put release
    B->>B: enforce_window

Steps Explained¶

1. Neighbor Discovery:

2. Node B receives a NeighborUp event from the swarm topic (Node A joined).

3. Inventory Request:

4. Node B sends a sync.Inventory request to Node A.

5. Node A returns a list of all releases it knows about.

6. Select Wanted:

7. Node B runs select_wanted on the union of its known releases and Node A’s inventory.

8. This determines which releases Node B needs to fetch.

9. Fetch Releases:

10. Node B fetches each wanted release from Node A (same as the replication flow).

🔍 On-Demand Pull with Seeding¶

Any node that holds a complete release can serve it, even if it wasn’t the original publisher. This is called seeding:

sequenceDiagram
    participant P as podman
    participant C as Node C (Registry)
    participant B as Node B (Seed)
    participant A as Node A (Publisher, offline)

    P->>C: GET /v2/A/app/manifests/1.0
    C->>C: Local release complete? No
    C->>B: sync GetRelease(A, app, 1.0)
    B-->>C: Release (signed JSON)
    C->>B: iroh-blobs download(BLAKE3:...)
    B-->>C: Verified stream
    C->>C: Verify, index, store release (cache)
    C-->>P: Manifest (+ Docker-Content-Digest)
    P->>C: GET /v2/A/app/blobs/sha256:abc123
    C-->>P: Stream blob from store

Key Points¶

• Node C can fetch A/app:1.0 from Node B, even if Node A (the publisher) is offline.
• Node B must have replicated the release from Node A earlier (via gossip or sync).
• The release is stored in cache on Node C (subject to GC after gc_grace_secs).