K3s is a lightweight Kubernetes that supports all features of Kubernetes, including driver plugins and OIDC login. It's well suited for deployment of a Kubernetes cluster in a homelab with Raspberry Pis or other low-end hardware, and can be extended with Kubernetes plugins to add additional features.
K3s supports Kube API Server configuration with the --kube-apiserver-arg
flag in the server
command. Modify the K3s service file at /etc/systemd/system/k3s.service
to add the following arguments when running the server:
oidc-issuer-url
: <issuer-url>
oidc-client-id
: <client-id>
oidc-username-claim
: email
oidc-groups-claim
: groups
You can also modify the /etc/rancher/k3s/config.yaml
and add the args:
kube-apiserver-arg:
- oidc-issuer-url=https://auth.example.com
- oidc-client-id=kube-apiserver
- oidc-username-claim=preferred_username
- oidc-groups-claim=groups
Finally, restart the K3s service with systemctl
:
sudo systemctl restart k3s
Kubernetes RBAC can be extended to groups or users by the OIDC provider by applying ClusterRoleBinding
or RoleBinding
:
Example for ClusterRoleBinding
granting admin rights to a specific user:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: User
name: https://auth.ravianand.me#<username>
Example for RoleBinding
granting restricted rights to a OIDC group:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-cluster-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: restricted-user
subjects:
- kind: Group
name: k8s-restricted-users
Resources
If you plan on using spec.securityContext.sysctls
some may be marked by the Kubelet as unsafe. This will result in the sysctlForbidden
scheduling error for pods.
To allow certain sysctls
values, the kubelet-arg
value can be appended in /etc/rancher/k3s/config.yaml
:
kubelet-arg:
- allowed-unsafe-sysctls=net.ipv4.conf.all.src_valid_mark
- allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6
failed to create fsnotify watcher: too many files
This issue can be caused by the default limits set by fsnotify. You can fix it by setting the following values in your /etc/sysctl.conf
:
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288