Unexpected Error on web login and TOPT is gone

Hey Community and Bitwarden-Team

I am facing an Issue with my Bitwarden installation. It is self hosted. Version is 2022.05.02

This morning I noticed, that my TOTPs are simply gone. I am 100% shure that they worked yesterday. All Entries that had TOTPs, are still there, just the generated TOTP is not shown. When editing the Entry using the mobile App (Android), the URI is shown - but I am also quiet sure that It only showed the Secret in the past when editing, now it shows an otpauth:// URI. wouldn’t bet on it though…

Besides that, I am not able to log into my instance using the Vault Webclient. The login is shown as normal, but I am not able to login.
Error is: An unexpected Error has occured
The F12 console shows response: null, statusCode: 500 with no further information that seems usable to me. ( Yes, my PW and mail is correct. I checked VERY closely )
This only happens on the Webclient. Using the mobile Client or Browser extension (FireFox) I can login as usual.
I am mostly using the Browser Extension and Android Client.

Interesting part is - I am using the Paid Feature so that I can share my Instance with more People and another account I know the credentials of ( my wife ) has the same Issue while logging in on the Webclient, BUT only on the first try. On the second try, by just hitting enter again with the login form still filled, I can login AND all TOTPs are still working.

So I checked some logs from my Instance. I attached to some containers that bitwarden spins up and then logged in to see if any errors are thrown.
and the container bitwarden/identity throws 2 traces when I try to login using the webclient.
They do not seem to contain sensitive information - so here you go:

crit: IdentityServer4.Hosting.IdentityServerMiddleware[0]
      => SpanId:970bbe5043d7eb43, TraceId:4b73d398206857458808454c48f40d6a, ParentId:0000000000000000 => ConnectionId:0HMIVG6M7AFON => RequestPath:/identity/connect/token RequestId:0HMIVG6M7AFON:00000002
      Unhandled exception: 535: Authentication credentials invalid
      MailKit.Security.AuthenticationException: 535: Authentication credentials invalid
       ---> MailKit.Net.Smtp.SmtpCommandException: Authentication credentials invalid
         --- End of inner exception stack trace ---
         at MailKit.Net.Smtp.SmtpClient.AuthenticateAsync(Encoding encoding, ICredentials credentials, Boolean doAsync, CancellationToken cancellationToken)
         at Bit.Core.Services.MailKitSmtpMailDeliveryService.SendEmailAsync(MailMessage message) in /home/runner/work/server/server/src/Core/Services/Implementations/MailKitSmtpMailDeliveryService.cs:line 94
         at Bit.Core.Services.HandlebarsMailService.SendNewDeviceLoginTwoFactorEmailAsync(String email, String token) in /home/runner/work/server/server/src/Core/Services/Implementations/HandlebarsMailService.cs:line 135
         at Bit.Core.Services.UserService.SendTwoFactorEmailAsync(User user, Boolean isBecauseNewDeviceLogin) in /home/runner/work/server/server/src/Core/Services/Implementations/UserService.cs:line 372
         at Bit.Core.IdentityServer.BaseRequestValidator`1.BuildTwoFactorResultAsync(User user, Organization organization, T context, Boolean requires2FABecauseNewDevice) in /home/runner/work/server/server/src/Core/IdentityServer/BaseRequestValidator.cs:line 270
         at Bit.Core.IdentityServer.BaseRequestValidator`1.ValidateAsync(T context, ValidatedTokenRequest request, CustomValidatorRequestContext validatorContext) in /home/runner/work/server/server/src/Core/IdentityServer/BaseRequestValidator.cs:line 167
         at Bit.Core.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /home/runner/work/server/server/src/Core/IdentityServer/ResourceOwnerPasswordValidator.cs:line 99
         at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      => SpanId:970bbe5043d7eb43, TraceId:4b73d398206857458808454c48f40d6a, ParentId:0000000000000000 => ConnectionId:0HMIVG6M7AFON => RequestPath:/identity/connect/token RequestId:0HMIVG6M7AFON:00000002
      Connection id "0HMIVG6M7AFON", Request id "0HMIVG6M7AFON:00000002": An unhandled exception was thrown by the application.
      MailKit.Security.AuthenticationException: 535: Authentication credentials invalid
       ---> MailKit.Net.Smtp.SmtpCommandException: Authentication credentials invalid
         --- End of inner exception stack trace ---
         at MailKit.Net.Smtp.SmtpClient.AuthenticateAsync(Encoding encoding, ICredentials credentials, Boolean doAsync, CancellationToken cancellationToken)
         at Bit.Core.Services.MailKitSmtpMailDeliveryService.SendEmailAsync(MailMessage message) in /home/runner/work/server/server/src/Core/Services/Implementations/MailKitSmtpMailDeliveryService.cs:line 94
         at Bit.Core.Services.HandlebarsMailService.SendNewDeviceLoginTwoFactorEmailAsync(String email, String token) in /home/runner/work/server/server/src/Core/Services/Implementations/HandlebarsMailService.cs:line 135
         at Bit.Core.Services.UserService.SendTwoFactorEmailAsync(User user, Boolean isBecauseNewDeviceLogin) in /home/runner/work/server/server/src/Core/Services/Implementations/UserService.cs:line 372
         at Bit.Core.IdentityServer.BaseRequestValidator`1.BuildTwoFactorResultAsync(User user, Organization organization, T context, Boolean requires2FABecauseNewDevice) in /home/runner/work/server/server/src/Core/IdentityServer/BaseRequestValidator.cs:line 270
         at Bit.Core.IdentityServer.BaseRequestValidator`1.ValidateAsync(T context, ValidatedTokenRequest request, CustomValidatorRequestContext validatorContext) in /home/runner/work/server/server/src/Core/IdentityServer/BaseRequestValidator.cs:line 167
         at Bit.Core.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /home/runner/work/server/server/src/Core/IdentityServer/ResourceOwnerPasswordValidator.cs:line 99
         at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
         at IdentityServer4.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
         at Bit.Core.Utilities.CurrentContextMiddleware.Invoke(HttpContext httpContext, ICurrentContext currentContext, GlobalSettings globalSettings) in /home/runner/work/server/server/src/Core/Utilities/CurrentContextMiddleware.cs:line 21
         at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
         at Bit.SharedWeb.Utilities.ServiceCollectionExtensions.<>c__DisplayClass12_0.<<UseDefaultMiddleware>b__1>d.MoveNext() in /home/runner/work/server/server/src/SharedWeb/Utilities/ServiceCollectionExtensions.cs:line 510
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.Extensions.UsePathBaseMiddleware.Invoke(HttpContext context)
         at Bit.Identity.Startup.<>c__DisplayClass10_1.<<Configure>b__2>d.MoveNext() in /home/runner/work/server/server/src/Identity/Startup.cs:line 167
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

I am thankful for any tip or idea. I am happy to collect more logs if necessary.
cheers

edit: I noticed that the Traces are saying Authentication credentials invalid - but I am 100% have validated that they are correct. and as mentioned, they are working using Browser extension and Android Client.

Hey @dasrohr, can you confirm if premium is still active, or if it has expired?

hey @dwbit - thanks for your quick reply
much appreciated

since i can not login to my instance i can not veryfy the status of the subscription there. but logging into my account unter vault.bitwarden.com i can confirm that my subscription ist valid until 2023.

do you know a way i might be able to validate this on my inctance using the cli maybe?

cheers

Thanks for confirmation, let me check in with the team.

Thanks! Much appreciated.

I am happy to provide any additional information you might need. Just let me know.

I did some digging on the side.

The TOPT issue is most likely related to an expired license on my instance. I was not aware that I have to update it manually.

I found the license file on my instance under bwdata/core/licenses/organization/ stating "Expires": "2022-07-05T13:39:56Z" which lines up with the fact that I noticed TOTP is not working in the morning of 2022-07-06.

But this kinda does not explain the fact that I am not able to login to my webvault, which also means that I am not able to replace the license.

Any ideas?

Thanks for confirmation, can you create a ticket directly with the support team at Get in Touch | Bitwarden

I just did.

Will update here on how it went

The problem is solved.

The issue that the TOPTs were gone, was indeed caused by the outdated license.

The login issue was caused by a faulty smtp config. which sounds counter intuitive at first, made total sense after fixing it.
After providing my credentials, the system requested a PIN, that was send by email. so - the system tried to send a mail right after the login - which caused the error.

While it makes all sense now - the error should have been way more descriptive to point to the error.

anyway - it is fixed now an I know better next time. hope this helps someone coming across this.
Thans to @dwbit and the Bitwarden Support Team!

1 Like

Thanks for the update! I’ll be sure to highlight the feedback regarding error reporting to the team :+1:

1 Like