Styjun

UPDATE

I have found out that it works if I restart nginx on the container with docker exec -it backend_gateway_1 nginx -s reload. I have found some information from this question. It seems that connecting to 127.0.0.1 rather than localhost would work, however I am using docker networks so I'm not sure what my situation would be.

ORIGINAL QUESTION

I am trying to use nginx as a reverse proxy with docker. I cannot quite work out what's wrong as sometimes it works when but then if I use docker-compose down && docker-compose up to restart it all, i will then usually get this 504 error

I have a feeling this is to do with the nginx or docker configuration. I am able to make a request to / fine but any request to /user is causing the intermittent issue.

When I am not receiving a 504, the node app is working well and creates/returns a user. I am also able to connect to the database fine with mongo-express so that should help narrow something down.

docker-compose.yml

version: '3.5'
services:
 gateway:
 image: nginx:latest
 restart: always
 ports:
 - 8080:80
 volumes:
 - ./nginx/nginx.conf:/etc/nginx/nginx.conf
 - ./nginx/nginx-error.log:/var/log/nginx/error.log
 networks:
 - poker_network
 poker_mongo:
 image: mongo:latest
 container_name: poker_mongo
 restart: unless-stopped
 volumes:
 - db-data:/data/db
 expose: 
 - "27017"
 environment:
 MONGO_INITDB_ROOT_USERNAME: test
 MONGO_INITDB_ROOT_PASSWORD: 123
 networks:
 - poker_network
 depends_on:
 - gateway
 poker_user:
 container_name: poker_user
 build: ./user/
 image: poker/user:latest
 volumes:
 - ./user/:/usr/src/app
 #- /usr/src/app/node_modules
 expose:
 - "8080"
 depends_on:
 - gateway
 - poker_mongo
 networks:
 - poker_network
 environment:
 WAIT_HOSTS: poker_mongo:27017
 mongo-express:
 container_name: mongo-express
 image: mongo-express
 restart: always
 ports:
 - 8085:8081
 environment:
 ME_CONFIG_MONGODB_ADMINUSERNAME: test
 ME_CONFIG_MONGODB_ADMINPASSWORD: 123
 ME_CONFIG_MONGODB_SERVER: poker_mongo
 depends_on:
 - gateway
 - poker_mongo
 networks:
 - poker_network

networks:
 poker_network:
volumes: 
 db-data:

nginx.conf

events worker_connections 1024; 

http 

 server 
 listen 80;
 location /user 
 rewrite /user/(.*) /$1 break;
 proxy_pass http://poker_user:8080/;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $remote_addr;
 proxy_set_header Host $host;
 
 location / 
 return 200 'hello';
 

 

Any help is appreciated, thanks.

I found several similar related questions and will try to compile two possible solutions out of those:

• https://serverfault.com/questions/240476/how-to-force-nginx-to-resolve-dns-of-a-dynamic-hostname-everytime-when-doing-p


• Docker Network Nginx Resolver

First solution

First solution is to reverse dependency between services: instead of poker_user depend on gateway you can may gateway depend on user. This way when gateway starts if will be able to resolve poker_user hostname.

The Dockerfile would look this way in this case:

version: '3.5'
services:
 gateway:
 image: nginx:latest
 restart: always
 ports:
 - 8080:80
 depends_on: poker_user
 volumes:
 - ./nginx/nginx.conf:/etc/nginx/nginx.conf
 - ./nginx/nginx-error.log:/var/log/nginx/error.log
 networks:
 - poker_network
 poker_mongo:
 image: mongo:latest
 container_name: poker_mongo
 restart: unless-stopped
 volumes:
 - db-data:/data/db
 expose: 
 - "27017"
 environment:
 MONGO_INITDB_ROOT_USERNAME: test
 MONGO_INITDB_ROOT_PASSWORD: 123
 networks:
 - poker_network
 poker_user:
 container_name: poker_user
 build: ./user/
 image: poker/user:latest
 volumes:
 - ./user/:/usr/src/app
 #- /usr/src/app/node_modules
 expose:
 - "8080"
 depends_on:
 - poker_mongo
 networks:
 - poker_network
 environment:
 WAIT_HOSTS: poker_mongo:27017
 mongo-express:
 container_name: mongo-express
 image: mongo-express
 restart: always
 ports:
 - 8085:8081
 environment:
 ME_CONFIG_MONGODB_ADMINUSERNAME: test
 ME_CONFIG_MONGODB_ADMINPASSWORD: 123
 ME_CONFIG_MONGODB_SERVER: poker_mongo
 depends_on:
 - gateway
 networks:
 - poker_network

networks:
 poker_network:
volumes: 
 db-data:

I also removed some other dependencies as I think they are redundant.

Second solution

This solution is based on changing nginx configuration. It looks like nginx tries to resolve hostname specified in proxy_pass upfront and since it's not available yet when nginx starts it will keep responding with 504 because it doesn't try to resolve it again.

There seems to be a workaround with using variables inside proxy_pass directive which are treated differently and force nginx to perform DNS resolution again. This requires resolver to be configured in nginx and luckily there's one built into Docker when using user-defined networks.

Here's an example nginx configuration (though I haven't tested it myself):

events worker_connections 1024; 

http 

 server 
 listen 80;
 location /user 
 # Docker DNS server in user-defined networks
 # https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/
 resolver 127.0.0.11 ipv6=off;

 rewrite /user/(.*) /$1 break;

 # Using a variable will force nginx to re-resolve DNS name
 # https://serverfault.com/a/593003
 set $backend "http://poker_user:8080/";
 proxy_pass $backend;

 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $remote_addr;
 proxy_set_header Host $host;
 
 location / 
 return 200 'hello';
 

 

Hope this help!