AH00898: Error reading from remote server returned by /notifications/hub

Ask the Bitwarden Community
#1

EDIT: I’ve tried to increase the ProxyPass timeout to 300 seconds and it seems like the errors are gone. (For now at least)

Would it help with performance if I move the docker VM to a SSD storage? The VM is currently equipped with 8 GiB of RAM and 4 CPU cores and it’s barely doing anything. So maybe a faster storage would help it perform better?

Hi,

I’ve installed a new Bitwarden instance on my docker host. The server runs with self signed certificates and is proxied by an Apache2.4 on another server.

All servers are sitting behing NAT and ports 80 and 443 are forwarded to the Apache proxy. The proxy also terminates SSL with Let’s Encrypt certificates.

Everything seems to be working fine but the Apache logs the following lines in the error log over and over again:

[Mon May 06 16:19:05.125620 2019] [proxy:error] [pid 25588] [client <redacted client ip>:16214] AH00898: Error reading from remote server returned by /notifications/hub, referer: https://<redacted bitwarden url>/
[Mon May 06 16:19:12.953719 2019] [proxy_http:error] [pid 27149] (70007)The timeout specified has expired: [client <redacted client ip>:57953] AH01102: error reading status line from remote server <redacted bitwarden url>:443

This is my proxy config:

<VirtualHost *:443>
        ServerAdmin <redacted>
        DocumentRoot /var/www/vhosts/<redacted>/htdocs
        ServerName <redacted bitwarden url>
        ServerAlias www.<redacted bitwarden url>
        ErrorLog /var/www/vhosts/<redacted>/logs/<redacted>-ssl-error.log
        CustomLog /var/www/vhosts/<redacted>/logs/<redacted>-ssl-access.log common
        DirectoryIndex index.html index.shtml index.htm index.php

        <Directory /var/www/vhosts/<redacted>/htdocs>
                Allowoverride All
                order allow,deny
                Allow from all
                Options -Indexes -FollowSymlinks
        </Directory>

        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*)           ws://<redacted>:443/$1 [P,L]

        SSLProxyEngine On
        SSLEngine on
        SSLCertificateFile /var/www/vhosts/<redacted>/conf/cert.pem
        SSLCertificateKeyFile /var/www/vhosts/<redacted>/conf/privkey.pem
        SSLCertificateChainFile /var/www/vhosts/<redacted>/conf/chain.pem
        SSLProxyCACertificateFile "/var/www/vhosts/<redacted>/conf/ca-bitwarden.crt"
        SSLProxyVerify require
        SSLCipherSuite "HIGH:!aNULL:!MD5:!3DES:!CAMELLIA:!AES128"
        SSLHonorCipherOrder on
        SSLProtocol TLSv1.2
        Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains"

        ProxyPreserveHost On
        ProxyRequests Off
        ProxyPass / https://<redacted>:443/
</Virtualhost>

I couldn’t figure our how exactly these notifications work or what I need to do to make this work through a proxy. The rewrite rule is something I’ve copied from here.

I’ve changed the port to 443 because https is the only thing listening on the bitwarden side, really.

As I said everything seems to be working but the error messages pop up every few seconds in the log so I really want to fix it.

EDIT:

I found this in the notification log:

2019-05-06 14:59:17.308 +00:00 [Error] Connection id ""0HLMI6DKK15VA"", Request id ""0HLMI6DKK15VA:00000001"": An unhandled exception was thrown by the application.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'http://identity:5000/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'http://identity:5000/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: Connection refused ---> System.Net.Sockets.SocketException: Connection refused
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
   at IdentityServer4.AccessTokenValidation.IdentityServerAuthenticationHandler.HandleAuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

but if I curl the URL from the notifications container it works

[email protected]:/app# curl -v http://identity:5000/.well-known/openid-configuration
* Trying 172.24.0.6...
* TCP_NODELAY set
* Connected to identity (172.24.0.6) port 5000 (#0)
> GET /.well-known/openid-configuration HTTP/1.1
> Host: identity:5000
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 06 May 2019 16:00:22 GMT
< Content-Type: application/json; charset=UTF-8
< Server: Kestrel
< Transfer-Encoding: chunked
< Server-Version: 1.30.1
<