Styjun

I'm trying to add another Django app to my server. I already have the xmlanalyzer.maciejg.pl up&running, now I'm trying to add to the existing nginx & gunicorn setup another app to be available at fencing.maciejg.pl.

I've used the existing XMLAnalyzer gunicorn setup (working fine):

#!/bin/bash

NAME="xmlanalyzer" # Name of the application
DJANGODIR=/home/django/xmlanalyzer # Django project directory
SOCKFILE=/home/django/xmlanalyzer/run/gunicorn.sock # we will communicte using this unix socket
USER=my-user-name # the user to run as
GROUP=my-user-name # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn
DJANGO_SETTINGS_MODULE=xmlanalyzer.settings # which settings file should Django use
DJANGO_WSGI_MODULE=xmlanalyzer.wsgi # WSGI module name

echo "Starting $NAME as `whoami`"

# Activate the virtual environment
cd $DJANGODIR
source ../venv/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your Django Unicorn
# Programs meant to be run under supervisor should not daemonize themselves (do not use --daemon)
#exec gunicorn -b 127.0.0.1:8001 xmlanalyzer.wsgi:application --pid /tmp/gunicorn.pid ;
exec gunicorn -b 127.0.0.1:8001 $DJANGO_WSGI_MODULE:application 
 --name $NAME 
 --workers $NUM_WORKERS 
 --user=$USER --group=$GROUP 
## --bind=unix:$SOCKFILE 
 --bind=127.0.0.1:8001 
 --log-level=debug 
 --log-file=-

Here's gunicorn setup for Fencing app (not working):

#!/bin/bash

NAME="fencing" # Name of the application
DJANGODIR=/home/django/fencing # Django project directory
SOCKFILE=/home/django/fencing/run/gunicorn.sock # we will communicte using this unix socket
USER=my-user-name # the user to run as
GROUP=my-user-name # the group to run as
NUM_WORKERS=3 # how many worker processes should Gunicorn spawn
DJANGO_SETTINGS_MODULE=mysite.settings # which settings file should Django use
DJANGO_WSGI_MODULE=mysite.wsgi # WSGI module name

echo "Starting $NAME as `whoami`"

# Activate the virtual environment
cd $DJANGODIR
source ../venv/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your Django Unicorn
# Programs meant to be run under supervisor should not daemonize themselves (do not use --daemon)

exec gunicorn -b 127.0.0.1:8002 $DJANGO_WSGI_MODULE:application 
 --name $NAME 
 --workers $NUM_WORKERS 
 --user=$USER --group=$GROUP 
## --bind=unix:$SOCKFILE 
 --bind=127.0.0.1:8002 
 --log-level=debug 
 --log-file=-

Both seem to be up:

ps -ef | grep gunicorn
my-user-name 780 20697 0 10:20 ? 00:00:01 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8002 mysite.wsgi:application --name fencing --workers 3 --user=my-user-name --group=my-user-name
my-user-name 787 780 0 10:20 ? 00:00:00 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8002 mysite.wsgi:application --name fencing --workers 3 --user=my-user-name --group=my-user-name
my-user-name 788 780 0 10:20 ? 00:00:00 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8002 mysite.wsgi:application --name fencing --workers 3 --user=my-user-name --group=my-user-name
my-user-name 789 780 0 10:20 ? 00:00:00 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8002 mysite.wsgi:application --name fencing --workers 3 --user=my-user-name --group=my-user-name
my-user-name 1712 1656 0 12:40 pts/1 00:00:00 grep --color=auto gunicorn
root 1730 1 0 2018 ? 01:04:09 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8001 xmlanalyzer.wsgi:application --name xmlanalyzer --workers 3 --user=my-user-name --group=my-user-name
my-user-name 17483 1730 0 Mar25 ? 00:01:12 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8001 xmlanalyzer.wsgi:application --name xmlanalyzer --workers 3 --user=my-user-name --group=my-user-name
my-user-name 17554 1730 0 Mar25 ? 00:01:05 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8001 xmlanalyzer.wsgi:application --name xmlanalyzer --workers 3 --user=my-user-name --group=my-user-name
my-user-name 17953 1730 0 Mar25 ? 00:00:41 /home/django/venv/bin/python3 /home/django/venv/bin/gunicorn -b 127.0.0.1:8001 xmlanalyzer.wsgi:application --name xmlanalyzer --workers 3 --user=my-user-name --group=my-user-name

I've created nginx setup using the following:

/etc/nginx/sites-available# more xmlanalyzer
server 
 server_name xmlanalyzer.maciejg.pl;

 access_log off;

 location /static/ 
 alias /home/django/xmlanalyzer/XMLAnalyzer/static/;
 

 location / 
 proxy_set_header Host $host;
 proxy_pass http://127.0.0.1:8001;
 proxy_set_header X-Forwarded-Host $server_name;
 proxy_set_header X-Real-IP $remote_addr;
 add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
 
 # managed by Certbot

 listen 443 ssl; # managed by Certbot
 ssl_certificate /etc/letsencrypt/live/xmlanalyzer.maciejg.pl/fullchain.pem; # managed by Certbot
 ssl_certificate_key /etc/letsencrypt/live/xmlanalyzer.maciejg.pl/privkey.pem; # managed by Certbot
 include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot



server 
 if ($host = xmlanalyzer.maciejg.pl) 
 return 301 https://$host$request_uri;
 # managed by Certbot


 server_name xmlanalyzer.maciejg.pl;
 listen 80;
 return 404; # managed by Certbot
 client_max_body_size 64M;

I've changed the folders and the port from 8001 to 8002. As a result I've got:

/etc/nginx/sites-available# more fencing
server 
 server_name fencing.maciejg.pl;

 access_log off;

 location = /favicon.ico access_log off; log_not_found off; 
 location /static/ 
 alias /home/django/fencing/fencingtournament/static/;
 

 access_log /home/django/fencing/logs/nginx-access.log;
 error_log /home/django/fencing/logs/nginx-error.log;

 location / 
 proxy_set_header Host $host;
 proxy_pass http://127.0.0.1:8002;
 proxy_set_header X-Forwarded-Host $server_name;
 proxy_set_header X-Real-IP $remote_addr;
 add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
 
 # managed by Certbot

 listen 443 ssl; # managed by Certbot
 ssl_certificate /etc/letsencrypt/live/fencing.maciejg.pl/fullchain.pem; # managed by Certbot
 ssl_certificate_key /etc/letsencrypt/live/fencing.maciejg.pl/privkey.pem; # managed by Certbot
 include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


server 
 if ($host = fencing.maciejg.pl) 
 return 301 https://$host$request_uri;
 # managed by Certbot


 server_name fencing.maciejg.pl;
 listen [::]:80;
 return 404; # managed by Certbot

Now, while https://xmlanalyzer.maciejg.pl works great, I get ERR_TOO_MANY_REDIRECTS while trying to access https://fencing.maciejg.pl/

Curl proves that domain is set up correctly and nginx is available. For some reason it does not redirect to my app:

curl fencing.maciejg.pl
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
 body 
 width: 35em;
 margin: 0 auto;
 font-family: Tahoma, Verdana, Arial, sans-serif;
 
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Both sites are available.

ll ../sites-enabled/
total 12
drwxr-xr-x 2 root root 4096 Mar 25 20:39 ./
drwxr-xr-x 6 root root 4096 Mar 25 22:59 ../
lrwxrwxrwx 1 root root 26 Mar 25 20:39 fencing -> ../sites-available/fencing
lrwxrwxrwx 1 root root 30 Feb 23 2018 xmlanalyzer -> ../sites-available/xmlanalyzer

gunicorn log looks good to me:

tail gunicorn-error.log
Starting fencing as my-user-name
[2019-03-26 10:20:01 +0000] [780] [INFO] Starting gunicorn 19.7.1
[2019-03-26 10:20:01 +0000] [780] [INFO] Listening at: http://127.0.0.1:8002 (780)
[2019-03-26 10:20:01 +0000] [780] [INFO] Using worker: sync
[2019-03-26 10:20:01 +0000] [787] [INFO] Booting worker with pid: 787
[2019-03-26 10:20:01 +0000] [788] [INFO] Booting worker with pid: 788
[2019-03-26 10:20:02 +0000] [789] [INFO] Booting worker with pid: 789

nginx-access.log and nginx-error.log in my /home/django/fencing/logs foler are empty.

What did I miss? I'll appreciate any input.

EDIT Just to add - the app can be accessed if started manually:

Starting development server at http://159.65.24.62:8002/
[26/Mar/2019 14:24:37] You're accessing the development server over HTTPS, but it only supports HTTP.

The error here is expected - this is just to show that a web request to https://fencing.maciejg.pl:8002/ did hit the right spot, so the domain is set up ok.

EDIT 2 Changed nginx setup for Fencing app to access_log on; with the result as follows:

more nginx-access.log

37.30.26.37 - - [26/Mar/2019:14:14:04 +0000] "GET / HTTP/1.1" 301 5 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko
) Chrome/73.0.3683.86 Safari/537.36"
37.30.26.37 - - [26/Mar/2019:14:14:04 +0000] "GET / HTTP/1.1" 301 5 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko
) Chrome/73.0.3683.86 Safari/537.36"

So I can see that the request reaches nginx and it is refusing.

EDIT 3 After disabling redirects with certbot I now get nginx landing page when visiting http://fencing.maciejg.pl/ - so again, it is coming through, just not redirected to my app run by gunicorn. So I still believe this is a bug in my nginx setup - yet, I still do not see it...

Ok, I finally got this one resolved! There were multiple issues.

First, I had a collision in nginx setup.
The though part was, even once I got this one resolved, I still got HTTP errors. This was due to the fact, that gunicorn setup was invalid.
The though part was, even once I got this one resolved, I still got HTTP errors. This was due to the fact, that Django setup was invalid.

So I messed around with every piece round and round again and my advice and recipee for resolving would be: do a step by step analysis.

First: I've run my Django app in development mode by starting python manage.py runserver 127.0.0.1:8000 with Debug = True and SSL disabled. This way with curl I was able to get it responding and confirm it works.

Next I've killed the app and run it via gunicorn script. I found out that it was throwing an error due to the fact that the environmental variable holding the SECRET_KEY was not available. Honestly - I didn't get this one resolved, I've switched to keeping the key in a separate file. So, one issue resolved, I got my gunicorn running fine.

Nest step: kill the gunicorn process and invoke it using supervisiord. Turned out there was a privilige issue as supervisiord runs on a different account.

Once I got that one resolved, I've recreated my nginx setup from a scratch till it started forwarding the requests. No SSL.

Next, I rerun the certbot to put SSL back. Here I also found out that you need to clear browser cache as it did not show the page even though the setup underneath was fine. This took a while as well - I've accidentaly discovered this one as I tried reaching my site on a differnt laptop to discover, that it works fine.

Finally, I put my other nginx server configurations in sites-enabled back, one by one.

I hope I will never need this solution, and that it will only be useful to others :)