Kubernetes/K8S Deployment Help

Ask the Community Password Manager
1

Hello,

I’ve spent a good amount of my weekend attempting to get this work but I’m stuck now. This may be a large chunk of text to read but any assistance would be appreciated.

Goal: Bitwarden functioning with K8S Cluster
Steps to migrate to K8S:

  1. Use auto installer from Install and Deploy - Linux | Bitwarden Help Center
  2. Confirm everything is working and move all necessary folders over to K8S PVCs
  3. Change K8S envs to match the perviously auto install instance
  4. Create Bitwarden Deployment and ingresses for K8S

Problem I’m running into:

  • When attempting to login via web I receive Failed to load resource: the server responded with a status of 404 (Not Found) /identity/connect/token:1

  • AND Failed to load resource: the server responded with a status of 404 (Not Found)" /api/accounts/prelogin:1

  • From the identity container: info: Microsoft.AspNetCore.Server.Kestrel[32] Connection id "", Request id "": the application completed without reading the entire request body. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1] Request starting HTTP/1.1 POST http://bitwarden.domain.com/identity/connect/token application/x-www-form-urlencoded; charset=utf-8 121 info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2] Request finished in 0.5162ms 404 info: Microsoft.AspNetCore.Server.Kestrel[32] Connection id "", Request id "": the application completed without reading the entire request body.

  • From the API container info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1] Request starting HTTP/1.1 POST http://bitwarden.domain.com/api/accounts/prelogin application/json; charset=utf-8 34 info: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[4] CORS policy execution successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2] Request finished in 0.6784ms 404 info: Microsoft.AspNetCore.Server.Kestrel[32] Connection id "", Request id "": the application completed without reading the entire request body.

What works at this time:

  • Login page loads

  • Admin page loads and I’m able to send an email to access the admin functions perfectly fine.

K8S Config:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitwarden-mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitwarden-web-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitwarden-attachment-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitwarden-id-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitwarden-core-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bitwarden-ssl-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bitwarden-config
  namespace: default
data:
  ACCEPT_EULA: "Y"
  MSSQL_PID: "Express"
  SA_PASSWORD: "<SA PASSWORD>"
  ASPNETCORE_ENVIRONMENT: "Production"
  globalSettings__selfHosted: "true"
  globalSettings__baseServiceUri__vault: "https://bitwarden.domain.com"
  globalSettings__baseServiceUri__api: "https://bitwarden.domain.com/api"
  globalSettings__baseServiceUri__identity: "https://bitwarden.domain.com/identity"
  globalSettings__baseServiceUri__admin: "https://bitwarden.domain.com/admin"
  globalSettings__baseServiceUri__notifications: "https://bitwarden.domain.com/notifications"
  globalSettings__baseServiceUri__internalNotifications: "http://localhost:5006"
  globalSettings__baseServiceUri__internalAdmin: "http://localhost:5004"
  globalSettings__baseServiceUri__internalIdentity: "http://localhost:5003"
  globalSettings__baseServiceUri__internalApi: "http://localhost:5002"
  globalSettings__baseServiceUri__internalVault: "http://localhost:5000"
  globalSettings__pushRelayBaseUri: "https://push.bitwarden.com"
  globalSettings__installation__identityUri: "https://identity.bitwarden.com"
  globalSettings__sqlServer__connectionString: "Data Source=tcp:localhost,1433;Initial Catalog=vault;Persist Security Info=False;User ID=sa;Password=<SA PASSWORD>;MultipleActiveResultSets=False;Connect Timeout=30;Encrypt=True;TrustServerCertificate=True"
  globalSettings__identityServer__certificatePassword: "<CERTIFICATE PASSWORD>"
  globalSettings__attachment__baseDirectory: "/etc/bitwarden/core/attachments"
  globalSettings__attachment__baseUrl: "https://bitwarden.domain.com/attachments"
  globalSettings__dataProtection__directory: "/etc/bitwarden/core/aspnet-dataprotection"
  globalSettings__logDirectory: "/etc/bitwarden/logs"
  globalSettings__licenseDirectory: "/etc/bitwarden/core/licenses"
  globalSettings__internalIdentityKey: "<INTERNAL IDENTITY KEY>"
  globalSettings__duo__aKey: "<DUO KEY>"
  globalSettings__installation__id: "<INSTALLATION ID>"
  globalSettings__installation__key: "<INSTALLATION KEY>"
  globalSettings__yubico__clientId: "REPLACE"
  globalSettings__yubico__key: "REPLACE"
  globalSettings__mail__replyToEmail: "[email protected]"
  globalSettings__mail__smtp__host: "smtp.domain.com"
  globalSettings__mail__smtp__port: "25"
  globalSettings__mail__smtp__ssl: "false"
  globalSettings__mail__smtp__username: ""
  globalSettings__mail__smtp__password: ""
  globalSettings__disableUserRegistration: "false"
  adminSettings__admins: "[email protected]"
  LOCAL_UID: "0"
  LOCAL_GID: "977"    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s.app: bitwarden
  name: bitwarden
  namespace: default
spec:
  selector:
    matchLabels:
      k8s.app: bitwarden
  template:
    metadata:
      labels:
        k8s.app: bitwarden
    spec:
      containers:
      - image: bitwarden/mssql:1.32.0
        name: bitwarden-mssql
        envFrom:
        - configMapRef:
            name: bitwarden-config
        ports:
        - containerPort: 1433
          protocol: TCP
        volumeMounts:
        - mountPath: /var/opt/mssql/data
          name: mssql-data
        - mountPath: /var/opt/mssql/log
          name: mssql-logs
        - mountPath: /etc/bitwarden/mssql/backups
          name: mssql-backup
      - image: bitwarden/web
        name: bitwarden-web
        envFrom:
        - configMapRef:
            name: bitwarden-config
        ports:
        - containerPort: 5000
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/web
          name: web-data
      - image: bitwarden/attachments:1.32.0
        name: bitwarden-attachments
        envFrom:
        - configMapRef:
            name: bitwarden-config
        env:
        - name: ASPNETCORE_URLS
          value: http://+:5001
        ports:
        - containerPort: 5001
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/core/attachments
          name: attachment-core
      - image: bitwarden/api:1.32.0
        name: bitwarden-api
        envFrom:
        - configMapRef:
            name: bitwarden-config
        env:
        - name: ASPNETCORE_URLS
          value: http://+:5002
        ports:
        - containerPort: 5002
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/core
          name: bitwarden-core
        - mountPath: /etc/bitwarden/ssl
          name: ssl
        - mountPath: /etc/bitwarden/logs
          name: api-logs
      - image: bitwarden/identity:1.32.0
        name: bitwarden-identity
        envFrom:
        - configMapRef:
            name: bitwarden-config
        env:
        - name: ASPNETCORE_URLS
          value: http://+:5003
        ports:
        - containerPort: 5003
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/identity
          name: id-core
        - mountPath: /etc/bitwarden/core
          name: bitwarden-core
        - mountPath: /etc/bitwarden/ssl
          name: ssl
        - mountPath: /etc/bitwarden/logs
          name: id-logs
      - image: bitwarden/admin:1.32.0
        name: bitwarden-admin
        envFrom:
        - configMapRef:
            name: bitwarden-config
        env:
        - name: ASPNETCORE_URLS
          value: http://+:5004
        ports:
        - containerPort: 5004
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/core
          name: bitwarden-core
        - mountPath: /etc/bitwarden/ssl
          name: ssl
        - mountPath: /etc/bitwarden/logs
          name: admin-logs
      - image: bitwarden/icons:1.32.0
        name: bitwarden-icons
        envFrom:
        - configMapRef:
            name: bitwarden-config
        env:
        - name: ASPNETCORE_URLS
          value: http://+:5005
        ports:
        - containerPort: 5005
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/ssl
          name: ssl
        - mountPath: /etc/bitwarden/logs
          name: icon-logs
      - image: bitwarden/notifications:1.32.0
        name: bitwarden-notifications
        envFrom:
        - configMapRef:
            name: bitwarden-config
        env:
        - name: ASPNETCORE_URLS
          value: http://+:5006
        ports:
        - containerPort: 5006
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/bitwarden/ssl
          name: ssl
        - mountPath: /etc/bitwarden/logs
          name: notify-logs
      volumes:
      - name: mssql-data
        persistentVolumeClaim:
          claimName: bitwarden-mysql-pvc
      - name: web-data
        persistentVolumeClaim:
          claimName: bitwarden-web-pvc
      - name: attachment-core
        persistentVolumeClaim:
          claimName: bitwarden-attachment-pvc
      - name: id-core
        persistentVolumeClaim:
          claimName: bitwarden-id-pvc
      - name: bitwarden-core
        persistentVolumeClaim:
          claimName: bitwarden-core-pvc
      - name: ssl
        persistentVolumeClaim:
          claimName: bitwarden-ssl-pvc
      - hostPath:
          path: /PATH/HERE/Backups/bitwarden/mssql
        name: mssql-backup
      - hostPath:
          path: /PATH/HERE/Logs/bitwarden/mssql
        name: mssql-logs
      - hostPath:
          path: /PATH/HERE/Logs/bitwarden/api
        name: api-logs
      - hostPath:
          path: /PATH/HERE/Logs/bitwarden/identity
        name: id-logs
      - hostPath:
          path: /PATH/HERE/Logs/bitwarden/admin
        name: admin-logs
      - hostPath:
          path: /PATH/HERE/Logs/bitwarden/icons
        name: icon-logs
      - hostPath:
          path: /PATH/HERE/Logs/bitwarden/notifications
        name: notify-logs        
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s.app: bitwarden
  name: bitwarden
  namespace: default
spec:
  ports:
  - name: web
    port: 5000
    protocol: TCP
    targetPort: 5000
  - name: attachments
    port: 5001
    protocol: TCP
    targetPort: 5001
  - name: api
    port: 5002
    protocol: TCP
    targetPort: 5002
  - name: identity
    port: 5003
    protocol: TCP
    targetPort: 5003
  - name: admin
    port: 5004
    protocol: TCP
    targetPort: 5004
  - name: icons
    port: 5005
    protocol: TCP
    targetPort: 5005
  - name: notifications
    port: 5006
    protocol: TCP
    targetPort: 5006
  selector:
    k8s.app: bitwarden
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/config-backend: |
      http-response set-header Referrer-Policy same-origin
      http-response set-header X-Content-Type-Options nosniff
      http-response set-header X-XSS-Protection "1; mode=block"
      http-response set-header Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com; child-src 'self' https://*.duosecurity.com; frame-src 'self' https://*.duosecurity.com; connect-src 'self' wss://bitwarden.domain.com https://api.pwnedpasswords.com https://twofactorauth.org; object-src 'self' blob:;"
      http-response set-header X-Frame-Options SAMEORIGIN
      http-response set-header X-Robots-Tag "noindex, nofollow"
      http-response set-header Content-Type application/fido.trusted-apps+json if { var(txn.path) -m str /app-id.json }
    ingress.kubernetes.io/secure-backends: "false"
    ingress.kubernetes.io/ssl-passthrough: "false"
#    kubernetes.io/ingress.class: haproxy-ingress   
  name: bitwarden
  namespace: default
spec:
  rules:
  - host: bitwarden.domain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: bitwarden
          servicePort: 5000
  tls:
  - hosts:
    - bitwarden.domain.com
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/config-backend: |
      http-response set-header Referrer-Policy same-origin
      http-response set-header X-Content-Type-Options nosniff
      http-response set-header X-XSS-Protection "1; mode=block"
      http-response set-header X-Frame-Options SAMEORIGIN
    ingress.kubernetes.io/secure-backends: "false"
    ingress.kubernetes.io/ssl-passthrough: "false"
#    kubernetes.io/ingress.class: haproxy-ingress
  name: bitwarden-admin
  namespace: default
spec:
  rules:
  - host: bitwarden.domain.com
    http:
      paths:
      - path: /admin
        backend:
          serviceName: bitwarden
          servicePort: 5004
  tls:
  - hosts:
    - bitwarden.domain.com
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/config-backend: |
      http-response set-header Referrer-Policy same-origin
      http-response set-header X-Content-Type-Options nosniff
      http-response set-header X-XSS-Protection "1; mode=block"
    ingress.kubernetes.io/secure-backends: "false"
    ingress.kubernetes.io/ssl-passthrough: "false"
    ingress.kubernetes.io/rewrite-target: /
#    kubernetes.io/ingress.class: haproxy-ingress
  name: bitwarden-api-id-icons
  namespace: default
spec:
  rules:
  - host: bitwarden.domain.com
    http:
      paths:
      - path: /api/
        backend:
          serviceName: bitwarden
          servicePort: 5002
      - path: /identity/
        backend:
          serviceName: bitwarden
          servicePort: 5003
      - path: /icons/
        backend:
          serviceName: bitwarden
          servicePort: 5005
      - path: /notifications/
        backend:
          serviceName: bitwarden
          servicePort: 5006
      - path: /attachments/
        backend:
          serviceName: bitwarden
          servicePort: 5001
  tls:
  - hosts:
    - bitwarden.domain.com

If anyone has any idea how I should be moving forward any help would be appreciated. I’m fairly new to using Kubernetes but have had my Bitwarden self hosted instance running perfectly fine. Can attach more logs if needed or mess around with the config as necessary.

2

Hello, did you try to use your own nginx ingress rules or did you use their bitwarden/nginx docker image ? Because I had the same behaviors as you with a fresh install on kubernetes when I try to route my traffic dirrentctly with ingress rules and without the bitwarden/nginx. I switch back to use the bitwarden/nginx docker image and everything works now.

3

Did you get anywhere with this? I’m trying to deploy onto k3s and am struggling with getting it converted from docker.