Federation Architecture

Introduction

Workload identity infrastructure seeks to provision every workload in a contemporary datacenter compute environment — for example, a Kubernetes pod — with a unique cryptographic identity. This forms a trusted identity substrate which can be used to enable workloads to communicate securely with other workloads, authenticate to them and be authenticated by those workloads in turn.

A workload identity infrastructure is generally deployed at cluster granularity, against a set of nodes which are physically co-located and which possess high-bandwidth, high-reliability inter-node connectivity. However, the modern mission necessarily entails the management of multiple compute clusters in disparate locations to meet the redundancy, latency or availability levels demanded.

One solution to this problem is federation. In federation, different clusters exchange their cryptographic trust roots representing the cryptographic identity of each cluster’s workload identity infrastructure responsible for issuing cryptographic workload identities. Thus, workloads in one cluster can cross-authenticate communications to or from workloads in another cluster which has been federated.

An example of a contemporary, common, off the shelf workload identity infrastructure technology used today in cluster compute environments is SPIRE. SPIRE supports federation, and can therefore enable workloads in different clusters to communicate with one another in a secure, mutually authenticated way.

However, the approach pioneered by SPIRE Federation relies upon several assumptions which can become problematic. Namely, this approach assumes:

1. There is continuous connectivity between federated clusters.

2. Operators of each cluster have a way of exchanging information to securely bootstrap the trust anchors for federation, possess the knowledge and ability to do so securely, and are willing and able to do so manually and by hand when establishing a new federation relationship.

Continuous connectivity. The requirement for continuous connectivity emerges from the periodic rollover of the root cryptographic public keys used to issue workload identity artifacts such as X.509 certificates. Because it is desirable for the root CA used by a workload identity infrastructure to be periodically rotated, a given workload identity infrastructure will at some point in time t_2 eventually cease to possess any cryptographic key in common with itself at some previous point in time t_1 (t_1 < t_2).

For this reason, SPIRE-style federation conventionally requires connectivity to allow the trust anchors for each cluster to be periodically synchronized between clusters as they are updated. If a cluster becomes disconnected from its peer cluster for too long, the federation relationship becomes broken, as even once connectivity is restored, it is unable to authenticate its peer cluster under any key for the peer cluster previously known to it. Simply trusting the new keys of a peer cluster blindly would undermine the security of the entire identity infrastructure and facilitate man-in-the-middle attacks on any workload.

Therefore, secure resolution of this circumstance — in which a cluster was disconnected from its peer cluster for too long — requires manual intervention by an administrator by re-bootstrapping the federation relationship. If security is to be maintained, the administrator must re-establish the current trusted root key via an out-of-band channel, and then perform the manual bootstrapping process again.

Manual bootstrapping. In addition to the requirement for continuous connectivity, conventional SPIRE federation requires a manual bootstrapping process in which an administrator determines the cryptographic trust anchors, verifies them via an out-of-band channel, and then provisions them into each cluster’s cryptographic infrastructure.

However, the manual bootstrapping requirement is cumbersome and burdensome on administrators operating clusters under normal conditions, let alone those seeking to deploy federated clusters in DDIL environments — for example, a cluster might be hosted on a vehicle which has only intermittent connectivity to a central datacenter.

It also creates a significant risk of administrators who are not experts in the management of cryptographic infrastructures trying to avoid the pain of manual bootstrapping by mis-implementing their own, hand-rolled automations for provisioning federation which do not authenticate cluster identity, thus potentially introducing reduced-strength TOFU-style verification or even a complete lack of cryptographic verification into their cross-cluster cryptographic infrastructure.

Multiple infrastructures. An additional pain point arises if it is necessary to maintain multiple workload identity infrastructures within a single cluster. Conventional SPIRE supports only a single asymmetric key type (e.g. RSA, ECDSA) per instance; therefore, supporting different algorithms or other configuration variances for different workloads operating under different policies forces the creation and maintenance of multiple SPIRE instances. These instances must then be manually federated with one another using the above described manual process.

In other words, when operating multiple workload identity infrastructures within a single cluster, effective usage requires federation relationships to be established both with the other instances in the same cluster as well as with any and all instances established within any remote cluster with which federation is desired.

For example, consider the scenario where a a cluster “ground” maintains a post-quantum-secure ML-DSA-65 infrastructure (mldsa65.ground.example) and a legacy-compatibility ECDSA P256 infrastructure (ecp256.ground.example). Before this cluster even establishes any desire to federate with another cluster, both instances must at least federate with one another.

Subsequently, suppose this cluster desires to federate with a cluster “ship” which also maintains a ML-DSA-65 infrastructure and an ECDSA P256 infrastructure. The number of federation relationships which must be securely established, bootstrapped, managed, and kept online and in continuous connectivity lest they break is:

• 2 intra-cluster federation relationships for “ground” (mldsa65 to ecp256 and vice versa);
• 2 intra-cluster federation relationships for “ship” (mldsa65 to ecp256 and vice versa);
• mldsa65.ground to mldsa65.ship;
• mldsa65.ground to ecp256.ship;
• ecp256.ground to mldsa65.ship;
• ecp256.ground to ecp256.ship;
• mldsa65.ship to mldsa65.ground;
• mldsa65.ship to ecp256.ground;
• ecp256.ship to mldsa65.ground;
• ecp256.ship to ecp256.ground

for a total of 10 federation relationships. Note that each federation relationship is unidirectional, so a bidirectional federation relationship requires two relationships to be established and managed.

Managing this number of federation relationships manually, even for a trivial two-cluster, two-algorithm deployment is already untenable. As the number of clusters and algorithms increases, the number of federation relationships which must be managed increases exponentially.

As such, conventional SPIRE federation does not scale to modern cluster compute environments which must manage multiple identity infrastructures across multiple clusters, let alone under DDIL constraints which render inter-cluster connectivity occasional rather than continuous.

Purpose

This document describes an architecture for the automatic deployment and orchestration of inter-cluster federation relationships between multiple workload identity infrastructures maintained between multiple clusters.

The objective of this architecture is to enable administrators to federate clusters in a way which

• Ease of use. — is easy to use, and which has streamlined UX;

• Secure-by-default. — results in a secure, cryptographically authenticated, bidirectional federation relationship even when used by a novice administrator;

• Misuse-resistant. — is resistant to accidential misuse (no TOFU/unverified federation);

• Scalability; O(1) administrator burden. — which scales to clusters with multiple instances without increasing the operational burden upon an administrator;

• DDIL deployable. — which can be used in DDIL conditions when suitably configured.

Usage Introduction

QHx provides, as one of its core foundations, the ability to manage PKI instances seamlessly to provide a unified workload identity substrate spanning multiple clusters. It operates in a Kubernetes environment, managing the cluster’s workload identity infrastructure and providing ease of administration.

QHx is managed by QHx Manager, a controller-manager process deployed into the cluster on which QHx is installed. QHx facilitates the creation of multiple identity infrastructures, which are seamlessly created and managed autonomously by QHx Manager at administrator request.

Management of workload identity infrastructures is seamless and fully Kubernetes-driven. A singleton Kubernetes QHxCluster resource (of which there can be only one) provides cluster-scope configuration information for QHx:

apiVersion: qhx.dev/v1
kind: QHxCluster
metadata:
  name: qhx-cluster
spec:
  # Domain name for the cluster.
  identity: qhx.dev

Beneath this QHxCluster instance, which configures QHx Manager itself, as many QHxAuthority instances as needed can be created.

Creating a QHxAuthority is simple. For example, the above topology can be created using the above Kubernetes resource YAML:

---
apiVersion: qhx.dev/v1
kind: QHxAuthority
metadata:
  name: mldsa65
spec:
  identity: mldsa65
  config:
    signatureAlgorithm: mldsa65

---
apiVersion: qhx.dev/v1
kind: QHxAuthority
metadata:
  name: ec-p384
spec:
  identity: ec-p384
  config:
    signatureAlgorithm: ec-p384

Upon creation of a QHxAuthority resource, QHx Manager immediately provisions and stands up an independent public-key identity infrastructure with the specified parameters. Further, these infrastructures are automatically federated with one another on an intra-cluster basis; thus, any workload which is configured to receive an identity from the mldsa65 authority is able to interoperate and trust workloads which are issued an identity from the ec-p384 authority, and vice versa.

This intra-cluster federation is established and managed automatically by QHx Manager without any need for explicit configuration by an administrator, ensuring all workloads in a cluster have an interoperable foundation.

Multi-cluster federation. QHx Manager also facilitates the seamless management of federation between different clusters. The QHx Manager instance in each cluster manages the authorities configured for that cluster. When inter-cluster federation is configured, the QHx Manager in each cluster coordinates with its peer via the QHx Manager-to-Manager (M2M) protocol. This enables the trust root information necessary for federation to be synchronized between clusters in a coordinated way.

In particular, the configuration of multi-cluster federation between two clusters results in automatic federation on a pairwise basis between every authority in both clusters, as shown below:

Configuring federation between clusters is extremely simple and does not require cryptographic expertise. In the simple case where both clusters are simultaneously online and reachable from an administrator’s machine, a single command suffices:

# Pass a KUBECONFIG to both clusters.
$ qhx federation connect -K west.kubeconfig -K east.kubeconfig

This single command automatically provisions bi-directional inter-cluster federation between all authorities in both clusters on a pairwise basis. Moreover, establishment is cryptographically authenticated and secured, without the administrator needing to retrieve or validate any kind of key or cryptographic anchor; the TLS-secured connection to each Kubernetes cluster is used to securely obtain the necessary cryptographic key material to secure the establishment of bi-directional inter-cluster federation. There is no regression to a TOFU (“Trust On First Use”) security model with associated vulnerability to man-in-the-middle attacks during the establishment process.

Federation lifecycle. Once federation is established, the M2M protocol link manages the lifecycle of the federation relationship between the two clusters without any requirement for further administrator intervention, even in the even that an identity infrastructure rolls over its root CA key upon expiration.

Moreover, QHx-managed inter-cluster federation responds automatically to the creation of new QHx-managed public-key authorities. For example, if a new QHxAuthority resource is created in a cluster with an rsa-4096 key type, any cluster federated with that cluster will automatically pick up the new authority, even if that authority did not exist at the time federation was established. Ultimately, QHx-managed inter-cluster federation responds adaptively to any sequence of authority creation or authority deletion events.

Alternative federation processes. The process of establishing inter-cluster federation shown above is the easiest way to configure federation between two online clusters where a single operator holds administrative privileges to both clusters. However, this approach is less suitable to circumstances where direct connectivity to both clusters from the operator’s vantage point is not easily obtained, where clusters cannot communicate between themselves due to a DDIL environment, or where, for operational reasons, administrative access to both clusters by a single operator must be avoided.

If communication between clusters is possible but an operator has privilege only on their own cluster, they have the option of bootstrapping federation using a bootstrapping reference. Suppose that Alice maintains cluster west and Bob maintains cluster east. Bob wants to federate his cluster east so that his workloads can trust identities issued by Alice’s cluster to workloads stationed in west.

Alice begins by running a command to output cluster west’s bootstrap reference:

$ qhx federation info
Bootstrap Reference:
  qhx-bootstrap:k6at7la4x7vw6j2xmgfmedpj3oaofntxgorhpbtwqldkrfzlh6ca@west.qhx.dev,192.168.91.1:31833

Alice then provides this bootstrap reference to Bob. Alice is responsible for communicating the bootstrap reference to bob out-of-band via a secure means.

Bob takes the bootstrapping reference and runs the following command against his own cluster east:

$ qhx federation connect \
    qhx-bootstrap:k6at7la4x7vw6j2xmgfmedpj3oaofntxgorhpbtwqldkrfzlh6ca@west.qhx.dev,192.168.91.1:31833

Uni-directional federation is now established — all authorities in Bob’s cluster trust all identity infrastructures provisioned in Alice’s cluster. The lifecycle of federation is managed automatically by QHx Manager and survives root CA key rollover, as well as the creation and deletion of individual authorities in either cluster. If bi-directional federation is desired, the above process is simply repeated in reverse, with Bob providing his own cluster’s bootstrap reference to Alice.

Structure of a bootstrap reference. A bootstrap reference has the form

qhx-bootstrap:<hash>@<expected-trust-domain>[,<m2m-endpoint-hint>...]

and embeds several pieces of information:

• The bootstrap fingerprint, which is a cryptographic hash of the root public keys of each authority in the cluster at a particular point in time. Specifically, it is a digest of the canonical encoding of a bootstrap package’s payload.

• The expected trust domain (that is, the domain name of the cluster).

• Zero or more known addresses at which the cluster’s manager-to-manager (M2M) protocol endpoint can be reached.

The cryptographic hash ensures that bootstrapping is secure. The approach is similar to that used by Tor hidden services: the peer cluster to federate with is identified by a hash of its cryptographic public key identity. Any mismatch of this value will result in federation failing to establish, as keys will not be verified. It is not possible, or supported, to provision federation without this value; there is no insecure or trust-on-first-use mode. Therefore, the design resists misuse and insecure configuration by a careless operator.

The domain name is used to ensure that the domain name of the cluster being federated with matches operator expectations. This ensures, for example, that Carol, who owns cluster north, cannot pass off a bootstrap reference to her cluster north.qhx.dev as a reference to cluster east.qhx.dev. It is clearly visible to an operator to which cluster they are federating when they make use of a bootstrap reference.

The known addresses of the peer cluster’s M2M endpoint are used to establish M2M protocol connectivity. If none are specified, DNS resolution against the cluster’s domain name is available as a fallback; however, these M2M endpoint hints facilitate coordination in circumstances where inter-cluster DNS resolution is not available or desired.

DDIL federation establishment. The bootstrap reference-based federation process enables federation to occur when no one operator has administrative privileges to two clusters simultaneously. However, it requires that two clusters be able to communicate at the time federation is established in order to exchange bootstrapping information such as cryptographic keys. A DDIL federation establishment process enables two clusters to federate even before they have the opportunity to communicate with one another over a network. This may be of use, for example, when future connectivity is envisaged but not yet available.

In this scenario, Alice operates cluster west and wants to enable Bob, who operates cluster east, to federate. There is currently no connectivity between west and east, but it is anticipated that connectivity will be possible in the future.

Alice runs the following command to export a bootstrapping package from west:

$ qhx federation export bootstrap-package --file west.qhxfederationpackage

This creates a file containing the QHx federation bootstrapping information for west. This file must be transferred to Bob via a secure out-of-band means, who then imports it:

$ qhx federation import bootstrap-package --file west.qhxfederationpackage

As with the bootstrap reference method, this creates uni-directional federation, and the process can be repeated with parties reversed if bidirectional federation is desired.

Even though no connectivity between west and east clusters has ever occurred, Bob is now able to authenticate any workload identity issued by an authority extant under west at the time the bootstrap package was exported. This can be useful, for example, if a workload identity is used not just for mTLS but to sign data objects which are transported via an out-of-band channel.

This DDIL capability comes with time limits, as eventually west’s root public keys will be rotated. M2M federation coordination remains healthy so long as connectivity (or an import of an updated bootstrap package) occurs no less frequently than an interval computed from the root CA rotation interval in use and other QHx cluster configuration parameters. Administrators can configure these parameters to balance security and DDIL stability according to their needs.

Status Monitoring

The qhx federation connect command provides an easy-to-use frontend for establishing federation. Behind the scenes, this command establishes federation by configuring one or two QHxForeignCluster resources, one in each cluster to be federated. Each QHxForeignCluster resource represents a uni-directional federation relationship; thus, if bi-directional federation is desired, one such resource is created in each cluster.

Determining federation status and health can be done via a simple describe command:

$ kubectl describe qhxforeigncluster
Name:         east
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  qhx.dev/v1
Kind:         QHxForeignCluster
Metadata:
  Creation Timestamp:  2026-06-06T17:07:06Z
  Generation:          1
  Resource Version:    887216
  UID:                 8208105a-08a2-4e3a-b4ae-3aa67efe576b
Spec:
  Bootstrap Hash:         cf7hl25fm7knptgnbbg4g24yxirptaxu4x2yfets5noqbh6wdsma
  Expected Trust Domain:  east.qhx.dev
  m2mEndpoints:
    192.168.92.1:30910
Status:
  Conditions:
    Last Transition Time:  2026-06-06T17:07:06Z
    Message:               Spec is valid and admitted.
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted

    Last Transition Time:  2026-06-06T17:07:06Z
    Message:               Foreign cluster federation is ready.
    Observed Generation:   1
    Reason:                Synchronized
    Status:                True
    Type:                  Ready

    Last Transition Time:  2026-06-06T17:07:06Z
    Message:               Accepted M2M trust state is held and continuity is intact.
    Observed Generation:   1
    Reason:                Bootstrapped
    Status:                True
    Type:                  Bootstrapped

    Last Transition Time:  2026-06-06T17:07:06Z
    Message:               Accepted M2M trust state is up to date with the peer.
    Observed Generation:   1
    Reason:                Synchronized
    Status:                True
    Type:                  Synchronized

    Last Transition Time:  2026-06-06T17:07:06Z
    Message:               All verifier roots remain replayable within the recovery grace.
    Observed Generation:   1
    Reason:                RetentionHealthy
    Status:                False
    Type:                  RetentionDegraded

    Last Transition Time:  2026-06-06T17:07:06Z
    Message:               Accepted M2M state is consistent and ready for derived federation.
    Observed Generation:   1
    Reason:                NormativeStateReady
    Status:                True
    Type:                  FederationReconciled

  Foreign Authorities:
    Authority Id:              c8da4d70-6182-417c-a6f2-aa670454235f
    Bundle Hash:               sha256:2257e7a7cdeaf0b845be94e7f7fd3f5e82e96a9032f82a39a7b5443cc8febef0
    Last Bundle Refresh Time:  2026-06-06T17:07:06Z
    Trust Domain:              mldsa65.east.qhx.dev
  m2m:
    Active Set Hash:                sha256:388b6386cfd51957093549068265ab234b1bbaf6ff1766c1ed92d83103574100
    Last Accepted Sequence Number:  1
    Last Accepted Signed At:        2026-06-06T17:06:32Z
    Last Fetch Time:                2026-06-06T17:07:06Z
    Last Success Time:              2026-06-06T17:07:06Z
    Transition Log Id:              0197d7c8-4a1e-7f3c-9b02-2f6c0e5a9d41
    Verifier Set:
      Authority Id:  c8da4d70-6182-417c-a6f2-aa670454235f
      Bundle:        ...base64 value omitted...=
      Bundle Hash:   sha256:2257e7a7cdeaf0b845be94e7f7fd3f5e82e96a9032f82a39a7b5443cc8febef0
      Trust Domain:  mldsa65.east.qhx.dev
  m2mEndpoints:
    192.168.92.1:30910
Events:  <none>

Several status values are reported here:

• Ready
  • Accepted
  • Bootstrapped
  • FederationReconciled
• Synchronized
• RetentionDegraded
• PeerLatent

The Ready condition is reported true when Accepted, Bootstrapped and FederationReconciled are all true; it is by definition the logical AND of these sub-conditions, which are defined as follows:

• Accepted means that the QHxForeignCluster resource’s spec field has been validated for basic errors (such as trying to federate a cluster with itself or omitting a bootstrap hash used to authenticate initial trust anchors; insecure or TOFU usage by an operator is never allowed). If Accepted becomes false, the Reason field reports why, with further details in the Message field:

  • InvalidSpec: The spec lacks basic validity; for example, it does not specify a valid expected trust domain.

  • SelfFederation: The resource represents an attempt to federate a cluster with itself (in other words, the expected trust domain is the same as the local cluster).

• Bootstrapped means that a validated M2M state is held for the peer cluster. Initial trust has been acquired and verified, and continuity is intact.

  If Bootstrapped becomes false, the possible Reason values are:

  • AwaitingBootstrap: There is insufficient data to bootstrap (for example, the spec section of the resource lacks a valid bootstrap hash).

  • BootstrapUnreachable: None of the peer cluster’s resolved M2M endpoints was able to be used to fetch a bootstrap package matching spec.bootstrapHash. This might indicate that the peer cluster’s M2M endpoints are all unreachable, or that the bootstrap hash is incorrect.

  • BootstrapFingerprintMismatch: A bootstrap package was retrieved from the peer cluster’s M2M endpoint, but its computed fingerprint did not equal spec.bootstrapHash.

  • VerificationFailed: A bootstrap package was successfully retrieved from the peer cluster’s M2M endpoint, but cryptographic verification of it failed.

  • RebootstrapRequired: Communication between the two clusters via the M2M protocol has been interrupted for too long, so that trust continuity has been broken. Re-bootstrapping must be performed for federation to return to a healthy state.

• FederationReconciled means that a consistent state has been obtained and verified and the QHx Manager is able to reconcile this state into the configuration of the local authorities within a cluster.

In general, status conditions may be True, False, or Unknown. The Unknown status value propagates; for example, if a sub-condition of the Ready condition becomes Unknown, Ready also becomes Unknown.

The following conditions do not affect Ready. Transient interruptions to connectivity may cause these conditions to transition to false without disrupting overall federation; therefore, these conditions reflect the current health of the M2M protocol link between two clusters, rather than the present viability of the federation relationship, which can survive such disruption for a configurable period of time.

• Synchronized describes whether the most recent M2M protocol synchronization attempt succeeded and the replicated state of the peer cluster’s trust roots are fully caught up. This condition may become false for transient reasons such as connectivity interruptions.

  If Synchronized becomes false, the possible Reason values are:

  • FetchFailed: It was not possible to fetch the current trust replication state from the peer cluster’s M2M endpoint. This generally reflects a connectivity break.

  • VerificationFailed: The current trust replication state was fetched from the peer cluster’s M2M endpoint, but cryptographic verification of it failed.

  • M2MStatusTooLarge: The accepted M2M state is too large.

• RetentionDegraded is a negative-polarity condition (that is, it is desirable for it to report False). It is True only when a held verifier root has already expired even beyond the recovery grace period (that is, rootExpiry + recoveryGrace < now), so that the peer’s transition log entries can no longer be replayed even with recoveryGrace, and the federation relationship must therefore be re-bootstrapped manually by an administrator.

• PeerLatent is an advisory condition with neative polarity (that is, it is desirable for it to report False). It is True if the peer cluster’s M2M endpoint indicates that the peer cluster is in the latent state, meaning it does not currently operate any PKI authorities. It does not gate Ready nor does it being asserted cause any change or teardown of accepted trust state, as the signal is unauthenticated and purely informational. The condition is cleared if a subsequent poll is successful. If this condition reports True, it might indicate that the peer cluster has lost all PKI state (i.e. root keys) or that the cluster has been torn down comprehensively. The peer cluster may return to operation in the future, most likely using new key material and therefore requiring re-bootstrap.

Temporal Settings

The amount of time connectivity between two clusters may be interrupted without requiring manual re-bootstrapping upon restoration of connectivity is determined by two configuration settings, which can be tuned as desired:

• caTTL, which is the lifetime of a root CA certificate maintained by a PKI authority;

• recoveryGrace, which is the amount of time a root CA’s certificate can be used to endorse transition to a successor key after its expiry. The recoveryGrace period permits use of an expired root CA certificate solely to endorse its successor root CA certificate, but not to issue workload identities. It can be set to 0 for strict enforcement with no grace period. This setting allows for flexibility by allowing federation recovery even when connectivity is interrupted for a period in excess of caTTL at the cost of strict enforcement of expiry times.

The guaranteed (worst-case) survivability period for interruption of connectivity is approximately caTTL/2 + recoveryGrace. This is a worst case, so recovery may successfully occur even when this value is exceeded depending on the phase offset of rotations. Administrators can set these values to achieve their objectives based on the guaranteed amount of time they require federation to be recoverable after interrupted connectivity.

QHx Federation Internals & Architecture

Roles

How does QHx federation work? Broadly, QHx federation is facilitated by the QHx Manager instance in each cluster interacting with its peer over the M2M protocol.

A QHx federation relationship is naturally unidirectional; if bidirectional federation is desired, two federation relationships are configured in opposite directions. As such, when discussing the operation of QHx federation in regards to a specific federation instance, only unidirectional federation is considered. The following roles are assumed with regards to that federation relationship:

• The Publisher Cluster is the cluster which publishes information about its cryptographic roots (the trust information).

• The Acceptor Cluster is the cluster which fetches the trust information from the Publisher Cluster.

• The Publisher is the subsystem of QHx Manager which performs the publisher function, which includes serving the M2M protocol endpoint.

• The Acceptor is the subsystem of QHx Manager which performs the acceptor function, which includes querying another cluster’s M2M protocol endpoint periodically.

Federation is pairwise and non-transitive. Federation of a cluster A with a cluster B and cluster B with C does not imply that cluster A federates with cluster C.

Data Model

Transition Log

Fundamentally, a QHx federation relationship is defined by a replicated, append-only signed transition log. The trust state a cluster holds with regards to a peer cluster at any given point in time is the result of the deterministic application of a contiguous set of cryptographically-verified transition log entries with sequence numbers [1..n] published by that peer cluster.

The trust state (with regards to a given unidirectional federation relationship) is the state maintained by the Publisher Cluster or Acceptor Cluster which characters the current known state of that relationship. The trust state of a Publisher Cluster evolves over time; whenever it changes, the Publisher Cluster generates one or more Transition Log Entries, which are durably persisted. These log entries are replicated to the Acceptor Cluster, which applies them in order after cryptographically verifying each entry in turn, thereby causing its own understanding of the trust state to match that of the Publisher Cluster.

The trust state of the Publisher and Acceptor clusters do not necessarily match exactly at a given point in time; in particular, when a publisher cluster experiences a change to its trust state and generates a transition log entry, it may be some time before the acceptor cluster polls the publisher’s M2M protocol endpoint and retrieves the new transition log entries, thereby bringing its state up to date. If a connectivity outage occurs between the Publisher Cluster and the Acceptor Cluster, the Acceptor’s trust state may become significantly outdated. Once connectivity is restored, transition log entries are replayed and the Acceptor Cluster’s trust state becomes up to date once more.

The publisher cluster’s trust state is called the normative trust state. The trust state of an acceptor cluster is called the shadow trust state.

Transition log entries are cryptographically validated by an acceptor cluster before being applied to its previous shadow trust state to produce the new shadow trust state. Thus, QHx Federation is the managed replication of a sequence of cryptographically verified ordered log entries which represent authenticated changes to a cluster’s current state:

  ShadowTrustState' = Apply(ShadowTrustState, TransitionLogEntry)

A cluster maintains one transition log for the entire cluster, not per federation relationship. Therefore, as federation relationships are created and destroyed, each federation relationship exists only for a subset of the lifespan of the transition log. For example:

• When a cluster is first created, it begins with a transition log entry with a sequence number of 1 denoting the initial authority created within it.

• Authorities are added and removed over time, each reflected in a transition log entry, bringing the sequence number up to 5.

• Another cluster (the acceptor) federates with the cluster. It begins with a snapshot of the publisher cluster’s normative trust state, which becomes its initial shadow trust state; therefore, it does not have to replay all transition log entries, but is capable of replaying any future transition log entries that arise with a sequence number greater than 5.

In other words, the initial creation of a federation relationship is facilitated by a snapshot of a cluster’s current normative trust state; future changes are then applied by replication of the transition log. Every transition log entry carries an integer sequence number. When an acceptor cluster contacts a publisher cluster, it asks for any transition log entries with a sequence number greater than that of the most recent transition log entry it holds.

Trust State

The trust state conceptually comprises:

• the sequence number of the most recent transition log entry published (in the case of a Publisher’s Normative Trust State), or the last accepted transition log entry sequence number (in the case of an Acceptor’s Shadow Trust State); and

• a set of authority-specific trust states, one for each authority published by the Publisher Cluster. Each authority-specific trust state comprises:

  • the trust domain of the authority (a domain name);

  • the Authority ID, a UUID uniquely identifying the authority (this ensures that a temporal succession of unrelated authorities with the same trust domain are not confused for one another);

  • the trust bundle for the authority, comprising its X.509 root public keys (for validating X.509-SVIDs) and its JWT root public keys (for validating JWT-SVIDs).

Trust domain naming. An acceptor pins the trust domain it expects a peer to serve (spec.expectedTrustDomain, e.g. east.qhx.dev). Every authority in an accepted active set must have a trust domain beneath it (e.g. mldsa65.east.qhx.dev); an active set containing any authority outside that subtree is rejected. This ensures a peer cannot cause the local cluster to trust a trust domain the local cluster operator did not expect.

A transition log entry logically describes a mutation to this trust state in a canonical, deterministic way. There are three types of transition log entries (the wire entryType values are given in parentheses):

• Add Authority (AddAuthority)
• Remove Authority (DeleteAuthority)
• Bundle Rotation (BundleRotation)

The Add Authority event reflects addition of a new authority to the set of authorities provisioned within a cluster. It defines initial authority-specific trust state for that authority (trust domain, UUID, root public keys). Applying a transition log entry of this type adds the described authority to the logical set of authorities described in the trust state.

The Remove Authority event reflects the deletion of an existing authority from the set of authorities published by a cluster.

The Bundle Rotation event occurs when an existing authority rotates its root public key, and describes the new root public key for that authority. When an acceptor cluster applies a transition log entry of this type, it replaces the known trust bundle for the authority in question.

Signing Model

Transition log entries are signed. In particular, they are signed in a multi-signature architecture by every authority extant in a publisher cluster at the time the transition log entry is generated. In order for a transition log entry to be validated by an acceptor, all authorities described in the prior shadow trust state must have generated a valid signature over the entry. The multi-signature threshold for acceptance is n-of-n.

This model is chosen to avoid having the federation mechanism relying on the security of any one public key algorithm in a circumstance where multiple public key algorithms are deployed. Thus, if an operator deploys both ML-DSA-65 and ECDSA-P256 authorities, the federation protocol’s signature mechanism is guaranteeably at least as secure as either of them.

An alternative design would have been to introduce a new “super-CA” above each individual authority. This approach was deliberately avoided, as it introduces significant complexity and a whole new PKI infrastructure which must be managed; moreover, it would require that super-CA to commit to a specific public key algorithm, which could not obtain the level of assurance of the above design.

Coordinated removal. A particular complication of the n-of-n requirement occurs when an authority is removed by logging a Remove Authority transition log entry type. The authority must sign its own removal, therefore QHx Manager coordinates this process automatically and does not destroy the authority or its key material until the removal log entry has been correctly signed. This means that if any authority in a federation relationship loses its key material, or experiences uncoordinated “surprise removal”, the federation relationship becomes broken and must be re-bootstrapped. This design is chosen in favor of security over convenience.

Persistence

In Kubernetes, a publisher cluster durably stores transition log entries as QHxM2MTransitionLogEntry resources. Each log entry has a sequence number and appears in a contiguous sequence. The first transition log entry has a sequence number of 1.

An acceptor cluster records its current shadow trust state for a federation relationship in the QHxForeignCluster resource’s status field, including the current root public key and other authority-specific trust state for each authority, and the most recently applied transition log entry sequence number.

M2M Wire Protocol

The M2M protocol used to synchronize trust state between clusters revolves around two types of data structure:

• the bootstrap package, which provides a point in time snapshot of a cluster’s trust state. It is signed by all authorities in the cluster and contains the root trust anchor information for each authority;

• transition log entries, each of which contains a single change to a cluster’s trust state. It is also signed by all authorities in the cluster at the time the log entry was created.

A bootstrap fingerprint is a cryptographic hash over the contents of a bootstrap package.

The bootstrap packages and transition log entries are encoded using deterministic CBOR. These artifacts are transferred over an HTTPS-based API exposed as the M2M protocol endpoint. The following endpoints are exposed:

• GET /.well-known/qhx/federationM2M/bootstrap retrieves the bootstrap package reflecting the current cluster trust state. It changes over time whenever the trust state changes.

• GET /.well-known/qhx/federationM2M/bootstrap/<fingerprint> can be used to retrieve a (historical) bootstrap package by its bootstrap fingerprint. This is used when establishing federation using a non-current (but still valid) bootstrap reference, and allows the accepting cluster to fetch the older bootstrap package by its hash, validate that it matches that hash, and then apply transition log entries to bring the accepted trust state up to date. An unknown fingerprint simply returns 404.

• GET /.well-known/qhx/federationM2M/updates?after=<seqNo> retrieves a list of transition log entries with a sequence number greater than <seqNo>, allowing polling and synchronization of any new entries as they may arise. The after parameter may be omitted or set to 0 to retrieve the log from the very beginning.

Out-of-band status codes. The endpoints may return several HTTP status codes to indicate out of band error conditions:

• HTTP 503 (Service Unavailable). This may indicate that the transition log has not been created yet, for example because the cluster is still being created, or that some other backend failure occurred. The acceptor simply retries later. Note that this is an unauthenticated signal, so it does not cause an acceptor to make any normative change to its view of the federation relationship’s state; it is assumed to be a transient condition.

• HTTP 410 (Gone). The peer cluster is in the latent state and does not currently have a transition log at all because it has not been configured to begin operating or has been commanded to cease operating. This state is entered when (for example) a cluster has its QHxCluster Kubernetes resource deleted, or before such a resource is created. Like HTTP 503, this is an unauthenticated signal which is assumed to be transient. The Acceptor exposes it through the PeerLatent condition.

Example exchanges. An acceptor first fetches the current bootstrap checkpoint:

GET /.well-known/qhx/federationM2M/bootstrap HTTP/1.1
Host: west.qhx.dev:7500
Accept: application/qhx-m2m-bootstrap-package+cbor

HTTP/1.1 200 OK
Content-Type: application/qhx-m2m-bootstrap-package+cbor

<CBOR-encoded QHxBootstrapPackage>

The following is an approximate YAML respresentation of the binary CBOR bootstrap package shown here for exposition purposes:

# Signed data.
signed:
  kind: QHxBootstrapPackage
  clusterTrustDomain: east.qhx.dev

  # This is a UUID identifying a transition log (not a transition log entry).
  # It essentially identifies a temporal continuity of transition log entries;
  # in other words, it allows different clusters with the same trust domain
  # (but existing at different points in time) to be disambiguated so that they
  # are not accidentially confused.
  transitionLogID: <UUID>

  # The sequence number which was current at the time this bootstrap package
  # was created.
  sequenceNumber: 6

  signedAt:  2026-06-06T17:06:32Z
  notBefore: 2026-06-06T17:06:32Z
  notAfter:  2026-06-20T17:06:32Z

  # The active set of authorities.
  authorities:
    - authorityID: <UUID>
      trustDomain: mldsa65.east.qhx.dev
      bundle:
        # DER-encoded X.509 root certificates for validating X.509-SVIDs.
        x509Roots:
          - <DER root certificate>
        # COSE_Key public keys for validating JWT-SVIDs.
        jwtRoots:
          - <COSE_Key structure>

      # This is checksum over the above data which must match. It is redundant
      # but included to easily detect errors in active set hash computation.
      activeSetHash: <cryptographic hash>

# Signatures by each authority (e.g. ML-DSA-65, ECDSA-P384, RSA-4096) over the
# same signed data.
signatures:
  - signingAuthorityID: <UUID>
    signature: <COSE Signature>

# Unsigned advisory hints on layer 4 (TCP) endpoints at which the M2M protocol
# endpoint can (potentially) be reached.
unsignedHints:
  m2mEndpoints:
    - 192.0.2.1:7500

Having accepted the checkpoint at sequence number 5, it then polls for any newer transition log entries:

GET /.well-known/qhx/federationM2M/updates?after=5 HTTP/1.1
Host: west.qhx.dev:7500
Accept: application/qhx-m2m-transition-log-entry+cbor

HTTP/1.1 200 OK
Content-Type: application/qhx-m2m-transition-log-entry+cbor

<CBOR-encoded list of QHxTransitionLogEntry>

The following is an approximate YAML respresentation of a binary CBOR transition log entry showing a BundleRotation, shown here for exposition purposes:

signed:
  kind: QHxTransitionLogEntry
  clusterTrustDomain: east.qhx.dev

  # Uniquely identifies a contiguous transition log, as above.
  transitionLogID: <UUID>

  # Sequence number of this log entry.
  sequenceNumber: 6

  # One of 'BundleRotation', 'AddAuthority' or 'DeleteAuthority'.
  entryType: BundleRotation

  # These cryptographic hashes are technically redundant but detect errors in
  # the expected prior state and computation of the computed subsequent state
  # once this transition log entry is applied. They are the hashes of the
  # prior trust state and the resulting trust state which this log entry should
  # produce.
  previousActiveSetHash: <cryptographic hash>
  resultingActiveSetHash: <cryptographic hash>

  # Operation-specific arguments. BundleRotation has none.
  operation: {}

  # The full resulting active set of authorities after applying this entry.
  resultingAuthorities:
    - authorityID: <UUID>
      trustDomain: mldsa65.east.qhx.dev
      bundle:
        x509Roots:
          - <DER bytes of the new root certificate>
        jwtRoots:
          - <COSE_Key>

  # Used only for DeleteAuthority entries; indicates deleted authorities.
  tombstones: []

  signedAt:  2026-06-13T09:15:00Z
  notBefore: 2026-06-13T09:15:00Z
  notAfter:  2026-06-27T09:15:00Z

  # Advisory poll interval which the publisher uses to recommend to acceptors
  # how frequently to fetch.
  refreshHint: 5m0s

# Signatures by each authority (e.g. ML-DSA-65, ECDSA-P384, RSA-4096) over the
# same signed data. Note that these are n-of-n signatures from every authority
# in the *previous* active set.
signatures:
  - signingAuthorityID: <UUID>
    signature: <COSE Signature>

The publisher’s M2M protocol endpoint is exposed by every QHx Manager replica via the m2m Service, often as a NodePort Service.

When M2M endpoint hints are used, the default port number if unspecified is 7500. If no endpoint hints are provided when establishing federation, the endpoint m2m.qhx-system.svc.<expectedTrustDomain>:7500 is used by default.