K3s is a lightweight Kubernetes that supports all features of Kubernetes, including driver plugins and OIDC login. It's well suited for deployment of a Kubernetes cluster in a homelab with Raspberry Pis or other low-end hardware, and can be extended with Kubernetes plugins to add additional features.
K3s supports Kube API Server configuration with the --kube-apiserver-arg flag in the server command. Modify the K3s service file at /etc/systemd/system/k3s.service to add the following arguments when running the server:
oidc-issuer-url: <issuer-url>oidc-client-id: <client-id>oidc-username-claim: emailoidc-groups-claim: groupsYou can also modify the /etc/rancher/k3s/config.yaml and add the args:
kube-apiserver-arg:
- oidc-issuer-url=https://auth.example.com
- oidc-client-id=kube-apiserver
- oidc-username-claim=preferred_username
- oidc-groups-claim=groups
Finally, restart the K3s service with systemctl:
sudo systemctl restart k3s
Kubernetes RBAC can be extended to groups or users by the OIDC provider by applying ClusterRoleBinding or RoleBinding:
Example for ClusterRoleBinding granting admin rights to a specific user:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: User
name: https://auth.ravianand.me#<username>
Example for RoleBinding granting restricted rights to a OIDC group:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-cluster-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: restricted-user
subjects:
- kind: Group
name: k8s-restricted-users
Resources
If you plan on using spec.securityContext.sysctls some may be marked by the Kubelet as unsafe. This will result in the sysctlForbidden scheduling error for pods.
To allow certain sysctls values, the kubelet-arg value can be appended in /etc/rancher/k3s/config.yaml:
kubelet-arg:
- allowed-unsafe-sysctls=net.ipv4.conf.all.src_valid_mark
- allowed-unsafe-sysctls=net.ipv6.conf.all.disable_ipv6
failed to create fsnotify watcher: too many files
This issue can be caused by the default limits set by fsnotify. You can fix it by setting the following values in your /etc/sysctl.conf:
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288