Welcome to the Personal and Professional page for Don Windrem.

Getting Django and Django CMS up and running on AWS EB

So I have had DWindrem.com up and running since 2018 on Amazon Web Services Elastic Beanstalk using Django but I did not like the setup.  I was using SQL Lite as my database which was working but was not great.  I was using a home built Content Management system which was OK but again not great.  I decided about 2 months ago that I would learn Django CMS and convert my site over to that and change the database to use the Amazon supplied MySQL database.  I chose Django CMS because it seems to be the most popular CMS package available for Django and it has built in support for Bootstrap 4, which I know and use on my site.  I thought that the change would be pretty straightforward, but alas, nothing is as easy as it seems.  It took me about 1 month to get competent enough with Django CMS to begin the switchover.  While Django CMS does not seem too hard to learn, I needed to get a good grasp on how it works and how navigation and menu systems worked as my current ones were all home grown. 

Lets Get Started

This guide is not really a Step by Step guide into how to get a Django CMS site working on AWS EB.  I will link to How to Guides that will step you through the process if you need it, but my real goal is to let you know what I did not know when I started this process.  It is really just the problems I faced and the guides that helped me through.   If you need a How to guide on how to setup Django on Elastic Beanstalk,  This Amazon guide  is what I would suggest.  It does a very good of explaining everything from:

  • The packages needed to install.
  • Setting up your Python virtual environment
  • Setting up Django projects and applications
  • configuring your project to work on EB
  • Deploying your application.

It is a very simple and straightforward guide but does not give you much more than a simple introduction to Django.  This guide on the other hand, is very in depth and has separate tutorials discussing:

  • Getting started with Django
  • Django Fundamentals
  • Authentication
  • Django ORM
  • Class-based views

and more.  I looked on the web for information regarding tutorials for Django CMS and really did not see anything that walked me through the CMS package in a way I could learn from so I leaned on my O'Reilly subscription to find a book.  The Book I found and used was Beginning django CMS by Nigel George. It's a little dated, but it helped me get a good enough understanding of CMS so I could use the How To guides and reference manuals on the Django CMS documentation site. The site is very good but I needed a better tutorial than I could find on the site.

First Step, Convert the database

So once I had a good grasp on Django CMS and the process needed to get pages setup in CMS, I decided before I went too far, I would convert my local site over to MySQL, which is the Database format that my AWS Elastic Beanstalk instance had created for me.  Some people did not know, but yes, your Elastic Beanstalk instance has a default database that is created for you (if you choose to set it up).  Amazon makes using this database within your instance very easy.  They setup os.environ variables that make connecting to it as simple as adding this code to your settings.py file:

 

if 'RDS_HOSTNAME' in os.environ:
    print('Using AWS database through OS ENVIRON')
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': os.environ['RDS_DB_NAME'],
            'USER': os.environ['RDS_USERNAME'],
            'PASSWORD': os.environ['RDS_PASSWORD'],
            'HOST': os.environ['RDS_HOSTNAME'],
            'PORT': os.environ['RDS_PORT'],
        }
    }
else:
    print("Using Don's database hard coded")
...

 

If you are unsure if your EB instance has a database setup or how to find out, use this guide in the AWS docs to find out. This guide also explores the database setup as well as the next topic.  The code above is setup so that if I find the RDS_HOSTNAME dictionary key, I assume I am on AWS and connect to that database, otherwise I connect to my new local MySQL database.  This was easier than I thought it would be.  For my local database, I was able to setup a MySQL database without much problem.  If you use Windows, this guide seems pretty straightforward.  I have a Debian Linux server in my home that is not being used too much so setting up MySQL on it was not to difficult.

Next, The issue with media and static files

As you may or may not know, when you deploy an application to AWS Elastic Beanstalk, AWS actually does a lot more than just deploy your new application code.  First, it spins up a brand new instance, then it deploys your app to that new instance. Finally, if all is well, AWS deletes your old instance and now links your new instance to your account.  Because of this, your media files and possibly your static files will get blown away. To handle this issue, I decided to move my static and media files over to Amazon S3 for hosting. I followed the guide here which explained everything needed to accomplish this. The references to AWS S3 pages are a little old and need refreshing, but I managed to get through the guide pretty easily.

Next, issues with the deployment process

I had several issues with the deployment process and some of them took some time to research and fix. 

Windows Development and Linux Production environment

The first issue was with Windows development environment and Linux Production environment.  My AWS EB instance uses the AWS Linux 2.9.7 build with Python 3.6 and my home setup uses Python 3.6.8 with Windows 10.  This has not been an issue with previous builds, but is an issue now because of all of the helper packages needed, some of them are windows specific.

  The fix for this is fairly simple but actually finding out that this was the issue took several deployments and reviewing the logs to get through all of them.  If you are unsure how to view the logs in your environment, log into the AWS console, select Elastic Beanstalk to get to your environments page. Then select your environment and then select the Logs link on the left hand side of the page.  The packages specifically that needed to be removed for me were:

  • pypiwin32
  • psycopg2 - PostgreSQL driver which I am not using anymore.  If you do need this, you need to add it to your packages.config file as stated in this Stackoverflow post.
  • pywin32

After seeing the errors I thought, "Oh, of course this failed", but it was annoying nonetheless. My next issue was a little harder to pick out.

App failures and namespace problems:

So once I got the Deployment errors corrected, I thought that I was ready to roll and the site was working.  But alas, it was not yet to be.  And worse yet, none of my logs were showing any errors other than "populate() isn't reentrant".  So I dug around a little bit and really did not see an answer other than changing my wsgi.py file so that it would quit masking the issue and show the actual error.  This Stackoverflow thread was very helpful.  I modified my wsgi.py file to look like this

import os
import time
import traceback
import signal
import sys
from django.core.wsgi import get_wsgi_application

try:
    application = get_wsgi_application()
    print('WSGI without exception')
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myblog.settings')
except Exception:
    print('handling WSGI exception')
    # Error loading applications
    if 'mod_wsgi' in sys.modules:
        traceback.print_exc()
        os.kill(os.getpid(), signal.SIGINT)
        time.sleep(2.5)

and the errors just started popping up in the logs.  Now to view these errors, I needed to pull the full logs and navigate to var > log > httpd  and open the error_log.  You can open it with notepad, ultraedit, vi or your favorite text editor.  I found that one of my app names collided with something within AWS EB so I updated that and there were several other little errors that popped up as well.  Once these were fixed,  Whallah ! ! the site worked.  Now all I need to do is add some content.