Issues with recent deployment onto EC2 - Unable to login

Hello community! I’ve just recently deployed Bitwarden (manual setup) into AWS for my organization, but I’m running into two strange issues that I’m hoping some here may have some input for.

My setup is there are two EC2 instances in different private subnets within the same VPC. Outbound traffic to the internet is through a NAT Gateway. Both of these instances are fronted with a single load balancer which has traffic sent to it from Global Accelerator.

On one of my containers, when logging in, a failed request to PUSH Bitwarden is appearing in the logs for bitwarden-identity. Here is the actual error message:

Bit.Core.Services.RelayPushRegistrationService[0]
      => SpanId:<omitted from post>, TraceId:<omitted from post>, ParentId:0000000000000000 => ConnectionId:0HMOGCLSSQDJ2 => RequestPath:/identity/connect/token RequestId:0HMOGCLSSQDJ2:00000002
      Request to https://push.bitwarden.com/push/register is unsuccessful with status of BadRequest-Bad Request

I’m still able to sign in and create/modify contents of my vault. But it appears there is an issue with a push being triggered. I do have a blanket whitelist that allows any requests from api.bitwarden.com or push.bitwarden.com to come through.

The second issue I’m running into is a pretty big deal because I can’t figure out what is going on. On the second EC2 instance (already tried recreating the host twice to be sure it wasn’t infra related), login attempts are never successful and I am not met with invalid username/password messages. Instead it’s always a “there was an error” and the bitwarden-identity logs contain the following:

crit: IdentityServer4.Hosting.IdentityServerMiddleware[0]
      => SpanId:<omitted from post>, TraceId:<omitted from post>, ParentId:0000000000000000 => ConnectionId:0HMOGCLUSMCVA => RequestPath:/identity/connect/token RequestId:0HMOGCLUSMCVA:00000002
      Unhandled exception: The key {<omitted from post>} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning
      System.Security.Cryptography.CryptographicException: The key {<omitted from post>} was not found in the key ring. For more information go to aka ms / datasoftwareprotection (removed link due to 2 link max for new users rule)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
         at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
         at Bit.Infrastructure.Dapper.Repositories.UserRepository.UnprotectData(User user) in /home/runner/work/server/server/src/Infrastructure.Dapper/Repositories/UserRepository.cs:line 234
         at Bit.Infrastructure.Dapper.Repositories.UserRepository.UnprotectData(IEnumerable`1 users) in /home/runner/work/server/server/src/Infrastructure.Dapper/Repositories/UserRepository.cs:line 254
         at Bit.Infrastructure.Dapper.Repositories.UserRepository.GetByEmailAsync(String email) in /home/runner/work/server/server/src/Infrastructure.Dapper/Repositories/UserRepository.cs:line 45
         at Bit.Core.Identity.UserStore.FindByEmailAsync(String normalizedEmail, CancellationToken cancellationToken) in /home/runner/work/server/server/src/Core/Identity/UserStore.cs:line 52
         at Microsoft.AspNetCore.Identity.UserManager`1.FindByEmailAsync(String email)
         at Bit.Identity.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /home/runner/work/server/server/src/Identity/IdentityServer/ResourceOwnerPasswordValidator.cs:line 62
         at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      => SpanId:<omitted from post>, TraceId:<omitted from post>, ParentId:0000000000000000 => ConnectionId:0HMOGCLUSMCVA => RequestPath:/identity/connect/token RequestId:0HMOGCLUSMCVA:00000002
      Connection id "0HMOGCLUSMCVA", Request id "0HMOGCLUSMCVA:00000002": An unhandled exception was thrown by the application.
      System.Security.Cryptography.CryptographicException: The key {<omitted from post>} was not found in the key ring. For more information go to aka ms datasoftwareprotection. (removed link due to new user 2 link max rule)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
         at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
         at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
         at Bit.Infrastructure.Dapper.Repositories.UserRepository.UnprotectData(User user) in /home/runner/work/server/server/src/Infrastructure.Dapper/Repositories/UserRepository.cs:line 234
         at Bit.Infrastructure.Dapper.Repositories.UserRepository.UnprotectData(IEnumerable`1 users) in /home/runner/work/server/server/src/Infrastructure.Dapper/Repositories/UserRepository.cs:line 254
         at Bit.Infrastructure.Dapper.Repositories.UserRepository.GetByEmailAsync(String email) in /home/runner/work/server/server/src/Infrastructure.Dapper/Repositories/UserRepository.cs:line 45
         at Bit.Core.Identity.UserStore.FindByEmailAsync(String normalizedEmail, CancellationToken cancellationToken) in /home/runner/work/server/server/src/Core/Identity/UserStore.cs:line 52
         at Microsoft.AspNetCore.Identity.UserManager`1.FindByEmailAsync(String email)
         at Bit.Identity.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /home/runner/work/server/server/src/Identity/IdentityServer/ResourceOwnerPasswordValidator.cs:line 62
         at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
         at IdentityServer4.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
         at Bit.Core.Utilities.CurrentContextMiddleware.Invoke(HttpContext httpContext, ICurrentContext currentContext, GlobalSettings globalSettings) in /home/runner/work/server/server/src/Core/Utilities/CurrentContextMiddleware.cs:line 20
         at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
         at Bit.SharedWeb.Utilities.ServiceCollectionExtensions.<>c__DisplayClass11_0.<<UseDefaultMiddleware>b__1>d.MoveNext() in /home/runner/work/server/server/src/SharedWeb/Utilities/ServiceCollectionExtensions.cs:line 505
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.Extensions.UsePathBaseMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)
         at Bit.Identity.Startup.<>c__DisplayClass10_1.<<Configure>b__2>d.MoveNext() in /home/runner/work/server/server/src/Identity/Startup.cs:line 175
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Has anyone seen either of these issue previously and found a fix for them? I’d really like to get my team using Bitwarden.

This actually turned out to be an issue with the initial request being handled by one node and the following request being received at a second node. Session stickiness on the ELB’s resolved this issue.