故障现象
使用Rancher2.0添加了k3s集群进行Kubernetes的相关web管理。
因删除k3s集群监控服务:删除Rancher依赖的namespace(cattle-system),状态一直是Terminating。
root@k3s:~# kubectl get namespace
NAME STATUS AGE
kube-system Active 76d
kube-public Active 76d
default Active 76d
kube-node-lease Active 76d
hollysys-edge Active 49d
hollysys-project Active 71d
monitoring Active 51d
public Active 71d
emojivoto Active 72d
cattle-system Terminating 34d
导致新的监控服务无法添加(Error from server (Forbidden): error when creating “STDIN”:)
root@k3s:~# curl --insecure -sfL https://***.***.***.**:9443/v3/import/kqxdpmckkj5xsgg696hhpx5xhchzh8nnq74kvczbvhbp5b5b5cl2s5.yaml | kubectl apply -f -
clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver created
clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master created
namespace/cattle-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding created
clusterrole.rbac.authorization.k8s.io/cattle-admin created
Error from server (Forbidden): error when creating "STDIN": serviceaccounts "cattle" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
Error from server (Forbidden): error when creating "STDIN": secrets "cattle-credentials-855d8e8" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
Error from server (Forbidden): error when creating "STDIN": deployments.apps "cattle-cluster-agent" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
Error from server (Forbidden): error when creating "STDIN": daemonsets.apps "cattle-node-agent" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
解决办法
kubectl patch namespace cattle-system -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
kubectl delete namespace cattle-system --grace-period=0 --forcekubectl patch namespace cattle-global-data -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
kubectl delete namespace cattle-global-data --grace-period=0 --forcekubectl patch namespace local -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-systemfor resource in `kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -o name -n local`; do kubectl patch $resource -p '{"metadata": {"finalizers": []}}' --type='merge' -n local; donekubectl delete namespace local --grace-period=0 --force
执行完成后查看namespace
root@k3s:~# kubectl get namespaces
NAME STATUS AGE
kube-system Active 76d
kube-public Active 76d
default Active 76d
kube-node-lease Active 76d
hollysys-edge Active 49d
hollysys-project Active 71d
monitoring Active 51d
public Active 71d
emojivoto Active 72d
之后再rancher界面添加新的集群监控即可重新部署该k3s集群到新的rancher中进行web管理