Styjun

I have GRPC service that binded on localhohst and nginx proxy as ssl terminator on same host(lets name it localhost-proxy).
Also at the edge I have another nginx grpc proxy which I want to use as a balancer+failover, this proxy redirects to localhost-proxy.
Problem is in edge-proxy failover configuration - I doesn't work when service is switched of but localhost-proxy is working. When service is switched off - localhost-proxy returns http status 204 with grpc-status=14 header.
First my confiuration was:

#localhost-proxy

upstream direct 
 server localhost:50032;
 keepalive 10;
 keepalive_timeout 60s;


server 
 listen 50042 ssl http2;

 ssl_certificate /etc/service/certs/internal_api_server/server.crt;
 ssl_certificate_key /etc/service/certs/internal_api_server/server.key;
 ssl_client_certificate /etc/service/certs/internal_api_server/clientCA.crt;
 ssl_verify_client on;

 access_log /var/log/nginx/direct.json grpc_json;

 location / 
 grpc_set_header X-Real-IP $client_ip;

 grpc_pass grpc://direct;
 

 include /etc/nginx/errors.grpc.conf;
 default_type application/grpc;



edge-proxy:
upstream edge 
 server localhost-proxy1:50042;
 server localhost-proxy2:50042 backup;
 keepalive_timeout 60s;


server 
 listen 50052 ssl http2;

 ssl_certificate /etc/service/certs/internal_api_server/server.crt;
 ssl_certificate_key /etc/service/certs/internal_api_server/server.key;
 ssl_client_certificate /etc/service/certs/internal_api_server/clientCA.crt;
 ssl_verify_client on;

 access_log /var/log/nginx/edge.json grpc_json;

 grpc_ssl_certificate /etc/service/certs/internal_api_client/client.crt;
 grpc_ssl_certificate_key /etc/service/certs/internal_api_client/client.key;
 grpc_ssl_trusted_certificate /etc/service/certs/internal_api_client/serverCA.crt;
 grpc_ssl_verify on;
 grpc_ssl_name api.server;
 grpc_set_header X-Real-IP $client_ip;
 grpc_connect_timeout 5s;
 location / 
 grpc_connect_timeout 5s;



 grpc_pass grpcs://edge;

 

But with this configuration failover works only if localhost-proxy is switched off. When it is up - edge proxy just returns response from localhost-proxy. This is not what I want.
After I've add this to localhost-proxy config:

...
 grpc_pass grpc://direct;
#from here 
 error_page 502 = /error502grpc;
 

 location = /error502grpc 
 internal;
 default_type application/grpc;
 add_header grpc-status 14;
 add_header grpc-message "unavailable";
 return 502;
 

It starts return 502 to edge-proxy when service is unavailable.

After this I've tried this with edge-proxy:

1. Add grpc_next_upstream error timeout http_502 non_idempotent invalid_header; - no effect


2. Add

 ...
 grpc_intercept_errors on;
 error_page 502 = @failover;
 }
 location @failover 
 grpc_pass grpcs://edge_failover;
 

upstream edge_failover 
 server localhost-proxy2:50052;
 keepalive 10;
 keepalive_timeout 60s;

Still no effect - it returns same 502 response.

I want failover on edge-proxy, and it should change upstream when service is down(regardless of localhost-proxy). But I don't now how to do this.

UPDATE

After some investigation in failover logs I've found that with latest variant(2) request actually goes to failover backend, but - nginx resend only grpc headers, no request body(GRPC DATA packet) sent. So failover awaits full request and then edge-proxy resets connection by timeout.