Signed Resources

Overview

Signed resources enable external authorities to control Kubernetes resources through cryptographic signatures. This allows policy updates and application deployments in environments where field operators should not have the ability to make arbitrary changes.

Use cases:

• Air-gapped environments receiving updates via USB drive
• Central authority distributing policy to remote sites
• Trusted application deployment where only signed apps can run
• Immutable security policies that can only be updated by specific keys

How It Works

1. Create Resources: Define Kubernetes resources (policies, deployments, services)
2. Sign Bundle: Sign the resource set with a trusted private key
3. Create QHxSignedResource: Encapsulate signed resources in a QHxSignedResource CRD
4. Apply to Cluster: Load QHxSignedResource via kubectl (or USB drive)
5. Automatic Verification: QHx Manager validates signature and applies resources
6. Status Reporting: Success or failure is reported in the resource status

QHxSignedResource Format

A signed resource encapsulates one or more Kubernetes resources:

apiVersion: qhx.dev/v1
kind: QHxSignedResource
metadata:
  name: signed-policy-update
  namespace: qhx-system
spec:
  resources:
    - apiVersion: qhx.dev/v1
      kind: QHxClusterPolicy
      metadata:
        name: qhx-cluster-policy
      spec:
        autolabelling: true
        groupMapping:
          classification:
            - group: "mls:classification:secret"
              level: "us:s"
  resourceVersion: "2026-02-02T10:00:00Z"
  resourceSignature: "eyJhbGciOiJFUzI1NiIs..."

Fields:

• resources: List of Kubernetes resources to create or update
• resourceVersion: Version string for rollback protection
• resourceSignature: JWS signature over canonicalized resource bundle

Resource Versioning

Signed resources use monotonic versioning to prevent rollback attacks.

Version Tracking

Each signed resource has a resourceVersion (arbitrary string, compared lexicographically):

resourceVersion: "2026-02-02T10:00:00Z"

When a signed resource is applied, QHx adds an annotation to every created resource:

metadata:
  annotations:
    qhx.dev/signedResourceVersion: "2026-02-02T10:00:00Z"

Rollback Protection

Attempting to apply an older signed version fails:

# Applied version: "2026-02-02T10:00:00Z"
# Attempt to apply: "2026-02-01T10:00:00Z"
# Result: Rejected (earlier version)

Lexicographic comparison determines ordering:

• "2026-02-02" > "2026-02-01" ✓
• "v2" > "v1" ✓
• "2.0.0" > "1.9.9" ✓ (string comparison, not semantic versioning)

Recommendation: Use ISO 8601 timestamps for unambiguous ordering.

Sealed Resources

Resources can be sealed to prevent modification except via signed updates.

Sealing a Resource

Add the qhx.dev/sealed annotation:

apiVersion: qhx.dev/v1
kind: QHxClusterPolicy
metadata:
  name: qhx-cluster-policy
  annotations:
    qhx.dev/sealed: "true"
spec:
  # ... policy configuration

Once sealed:

• Resource cannot be modified via normal kubectl operations
• Resource cannot be deleted via normal kubectl operations
• Status updates and some system annotations are still allowed
• Can only be updated via QHxSignedResource

Sealing at Creation

Include the sealed annotation in a QHxSignedResource:

spec:
  resources:
    - apiVersion: qhx.dev/v1
      kind: QHxClusterPolicy
      metadata:
        name: qhx-cluster-policy
        annotations:
          qhx.dev/sealed: "true"

Restricted Sealing

Prevent ad-hoc sealing by setting restrictSealing in QHxClusterPolicy:

apiVersion: qhx.dev/v1
kind: QHxClusterPolicy
metadata:
  name: qhx-cluster-policy
spec:
  restrictSealing: true

With restrictSealing: true:

• Only signed resources can seal resources
• Prevents accidental sealing that could wedge the cluster
• Operators cannot seal arbitrary resources

Trust Anchors

Trusted signing keys are configured in QHxClusterPolicy:

apiVersion: qhx.dev/v1
kind: QHxClusterPolicy
metadata:
  name: qhx-cluster-policy
spec:
  trustedSigners:
    - kid: "authority-key-2026"
      kty: "EC"
      crv: "P-256"
      x: "MKBCTNIcKUSDii11ySs3526iDZ8AiTo7Tu6KPAqv7D4"
      y: "4Etl6SRW2YiLUrN5vfvVHuhp7x8PxltmWWlbbM4IFyM"
    - kid: "backup-key-2026"
      kty: "EC"
      crv: "P-256"
      x: "..."
      y: "..."

Trust anchor format: JWK (JSON Web Key)

Multiple keys enable:

• Key rotation
• Multiple authorities (e.g., security team and ops team)
• Backup keys

Creating Signed Resources

1. Prepare Resources

Create the resources you want to sign:

resources.yaml

apiVersion: qhx.dev/v1
kind: QHxClusterPolicy
metadata:
  name: qhx-cluster-policy
  annotations:
    qhx.dev/sealed: "true"
spec:
  autolabelling: true
  groupMapping:
    classification:
      - group: "mls:classification:secret"
        level: "us:s"
---
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    qhx.dev/sealedForKinds: "*"

2. Generate Signing Key

# Generate EC P-256 key pair
openssl ecparam -name prime256v1 -genkey -noout -out private-key.pem

# Extract public key in JWK format
qhx sign export-jwk --private-key private-key.pem > public-key.jwk

3. Sign Resources

qhx sign create \
  --resources resources.yaml \
  --private-key private-key.pem \
  --version "2026-02-02T10:00:00Z" \
  --output signed-resources.yaml

The output signed-resources.yaml contains a QHxSignedResource with signature.

4. Distribute

Copy signed-resources.yaml to target environment:

• USB drive for air-gapped sites
• Secure file transfer
• Configuration management system

5. Apply

kubectl apply -f signed-resources.yaml

6. Verify

Check the QHxSignedResource status:

kubectl get qhxsignedresource signed-policy-update -o yaml

Status indicates success or failure:

status:
  applied: true
  appliedAt: "2026-02-02T10:05:00Z"
  message: "Successfully applied 2 resources"

Or on failure:

status:
  applied: false
  message: "Signature verification failed: untrusted key"

Restricted Resource Creation

Control what can be created without signatures.

Namespace-Level Restrictions

Seal a namespace to require signed creation of specific resource types:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    qhx.dev/sealedForKinds: "Pod,Deployment,StatefulSet"

Now pods, deployments, and statefulsets in this namespace can only be created via signed resources.

Use * to require signatures for all resource types:

annotations:
  qhx.dev/sealedForKinds: "*"

Cluster-Level Restrictions

Require signed namespace creation:

apiVersion: qhx.dev/v1
kind: QHxClusterPolicy
metadata:
  name: qhx-cluster-policy
spec:
  requireSignedNamespaces: true

With this setting, new namespaces can only be created via signed resources.

Combined effect:

# In QHxClusterPolicy
requireSignedNamespaces: true

# In every namespace
annotations:
  qhx.dev/sealedForKinds: "*"

Result: All resource creation requires signatures.

Signed Deletion

Delete resources using signed deletion directives:

apiVersion: qhx.dev/v1
kind: QHxSignedResource
metadata:
  name: delete-old-policy
spec:
  deletedResources:
    - apiVersion: qhx.dev/v1
      kind: QHxClusterPolicy
      metadata:
        name: old-policy
        namespace: qhx-system
  resourceVersion: "2026-02-03T10:00:00Z"
  resourceSignature: "..."

This deletes the specified resource after signature verification.

Worked Example: Locked-Down Cluster

Initial Setup

Deploy cluster with sealed policy and trusted signing key:

apiVersion: qhx.dev/v1
kind: QHxClusterPolicy
metadata:
  name: qhx-cluster-policy
  annotations:
    qhx.dev/sealed: "true"
    qhx.dev/signedResourceVersion: "2026-01-01T00:00:00Z"
spec:
  requireSignedNamespaces: true
  restrictSealing: true
  trustedSigners:
    - kid: "authority-2026"
      kty: "EC"
      crv: "P-256"
      x: "MKBCTNIcKUSDii11ySs3526iDZ8AiTo7Tu6KPAqv7D4"
      y: "4Etl6SRW2YiLUrN5vfvVHuhp7x8PxltmWWlbbM4IFyM"

Seal existing namespaces:

for ns in default kube-system; do
  kubectl annotate namespace $ns qhx.dev/sealedForKinds="*" --overwrite
done

Deploy Signed Application

Authority creates signed application bundle:

app-bundle.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: trusted-app
  annotations:
    qhx.dev/sealedForKinds: "*"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: trusted-app
  labels:
    mls.qhx.dev/level: "us:s"
spec:
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
        mls.qhx.dev/level: "us:s"
    spec:
      containers:
      - name: api
        image: registry.example.mil/api-server@sha256:abc123

Sign bundle:

qhx sign create \
  --resources app-bundle.yaml \
  --private-key authority-key.pem \
  --version "2026-02-02T12:00:00Z" \
  --output signed-app.yaml

Distribute signed-app.yaml via USB drive to remote site.

Field operator applies:

kubectl apply -f signed-app.yaml

Result: Application deploys successfully because it’s signed by trusted key.

Attempted Unauthorized Deployment

Operator tries to deploy unsigned application:

kubectl create namespace unauthorized-app
# Error: requireSignedNamespaces is true

Operator tries to modify sealed resource:

kubectl annotate namespace default foo=bar
# Error: namespace is sealed

Operator cannot circumvent restrictions without the signing key.

Self-Protection Mode

When QHxClusterPolicy is sealed or has signing restrictions, QHx Manager automatically enables self-protection.

Self-protection prevents:

• Deletion of QHx CRDs
• Modification of webhook configurations
• Deletion of qhx-manager deployment
• Modification of sealed QHxClusterPolicy

Protected resources:

• qhx.dev CRDs
• MutatingWebhookConfiguration (qhx-related)
• ValidatingWebhookConfiguration (qhx-related)
• Deployment/qhx-manager in qhx-system namespace
• Service/qhx-manager in qhx-system namespace

Attempts to modify protected resources are rejected:

kubectl delete crd qhxclusterpolicies.qhx.dev
# Error: resource is protected by QHx self-protection

Updating QHx Itself

QHx updates must be delivered as signed resources:

# Authority signs QHx upgrade
qhx sign create \
  --resources qhx-upgrade-v0.7.0.yaml \
  --private-key authority-key.pem \
  --version "2026-02-05T10:00:00Z" \
  --output signed-upgrade.yaml

# Apply upgrade
kubectl apply -f signed-upgrade.yaml

Disabling Self-Protection

Self-protection can only be disabled by:

1. Applying a signed resource that unseals the QHxClusterPolicy
2. Gaining root access to control plane nodes and manually editing etcd
3. Modifying Kubernetes API server configuration to disable webhooks

Option 1 is the intended path. Options 2-3 require physical/root access and should be restricted.

Exemptions

System processes are automatically exempted from restrictions:

• Kubernetes controllers creating pods for deployments
• System namespaces (kube-system) are not subject to requireSignedNamespaces
• Status updates by controllers are allowed even on sealed resources

This ensures normal cluster operation continues while enforcing restrictions on user actions.

Troubleshooting

Signature Verification Failed

Check trust anchors in QHxClusterPolicy:

kubectl get qhxclusterpolicy qhx-cluster-policy -o jsonpath='{.spec.trustedSigners}'

Verify the signing key matches one of the trusted keys:

# Extract kid from signed resource
kubectl get qhxsignedresource signed-policy-update -o jsonpath='{.spec.resourceSignature}' | \
  base64 -d | jq -r '.kid'

# Check if kid is in trusted signers
kubectl get qhxclusterpolicy qhx-cluster-policy -o json | \
  jq '.spec.trustedSigners[] | select(.kid=="authority-key-2026")'

Version Rollback Detected

Current version is newer than attempted version:

# Check current version on resource
kubectl get qhxclusterpolicy qhx-cluster-policy \
  -o jsonpath='{.metadata.annotations.qhx\.dev/signedResourceVersion}'

# Attempting to apply earlier version will be rejected

Solution: Create a new signed resource with a later version.

Cannot Modify Sealed Resource

Verify resource is sealed:

kubectl get <resource-type> <resource-name> \
  -o jsonpath='{.metadata.annotations.qhx\.dev/sealed}'

If output is "true", resource can only be updated via signed resource.

Create signed update:

qhx sign create \
  --resources updated-resource.yaml \
  --private-key authority-key.pem \
  --version "2026-02-06T10:00:00Z" \
  --output signed-update.yaml

kubectl apply -f signed-update.yaml

QHxSignedResource Status Pending

Check qhx-manager logs:

kubectl logs -n qhx-system -l app=qhx-manager | grep "signed-resource"

Common issues:

• Signature verification in progress
• Resource conflicts (e.g., namespace already exists with different config)
• Invalid resource definitions

Accidentally Sealed Critical Resource

If a resource is accidentally sealed and you cannot unseal it with a signed resource:

Recovery requires root access to control plane nodes:

1. SSH to control plane node
2. Edit etcd directly or modify API server configuration to disable webhooks
3. Remove sealed annotation
4. Re-enable webhooks

Prevention: Use restrictSealing: true to prevent ad-hoc sealing.

Security Considerations

Private Key Protection

The signing key is the root of trust:

• Store offline in Hardware Security Module (HSM) or air-gapped system
• Use key ceremony procedures for key generation
• Maintain backup keys in separate physical locations
• Rotate keys periodically (update trustedSigners with new key, sign rotation with old key)

Signature Algorithm

QHx supports EC keys (NIST P-256, P-384) and RSA keys (2048-bit minimum):

# EC P-256 (recommended)
openssl ecparam -name prime256v1 -genkey -noout -out key.pem

# EC P-384 (higher security)
openssl ecparam -name secp384r1 -genkey -noout -out key.pem

# RSA 4096 (compatibility)
openssl genrsa -out key.pem 4096

EC P-256 provides strong security with smaller signatures.

Trust Anchor Distribution

Initial trust anchor must be established securely:

• Include in base cluster deployment
• Physically verify during cluster installation
• Use out-of-band verification (phone call, in-person verification)

Rotation requires signing the update with an existing trusted key.

Audit Trail

All signed resource operations are logged:

kubectl logs -n qhx-system -l app=qhx-manager | grep "signed-resource"

Export logs for compliance:

kubectl logs -n qhx-system -l app=qhx-manager --since=24h | \
  grep "signed-resource" > signed-resource-audit.log

Related Documentation

• Admission Controller
• Network Isolation
• Cluster Policy Reference