Reading:
Building a Django App with a Celery Backend
Share:

Building a Django App with a Celery Backend

Avatar
by Asher
August 13, 2020
Django Celery

This is the third article in the series to deploy a full stack on AWS ECS using Fargate:

  • Part 1:
    • A complete VPC with security groups, subnets, NAT gateways and more
  • Part 2::
    • Deploying an ECS cluster and IAM roles for Fargate services
    • Setting up a CloudFront distribution to serve static files
  • Part 3 (this article):
    • Creating a simple Django app with a celery backend to process asynchronous requests
  • Part 4:
    • Creating an RDS database & Redis instance
    • Registering the Django app in ECR and deploying it to ECS
  • Part 5:
    • Configuring the deployment to an use Nginx to route events to the Django app and to block malicious activity

Contents

Overview

We've now set up all of the basic infrastructure that we need to deploy our application. We deployed those resources in our first two walkthroughs because we will now be referencing some of them in the upcoming Django configurations.

So what exactly will our application do? We're just going to have a two input fields that will each take an email address as an input. Each of those inputs will send a POST request to Django. Depending on which input field a user provides the email address our backend will ether create a new user or it will serialize the email address and send to Redis as a Celery task at which point Django will then respond immediately to the client. Celery will continue to process the event asynchronously, sleeping for 15 seconds and validating that the email exists in our user base and sending an email to the address provided. We'll use a little bit of vanilla JS to make it happen on the front end.

Django App Flow
Django App Flow

To give you a visualization of what we're trying to build here is what it will look like:

Visualization of the App
Visualization of the App

At this point, if you read the previous post you may be asking why the Celery app sits within Django. The short answer is that we will be using the Django models within the Celery processes to validate the email address. This may not be the best production deployment strategy, especially if you can create a much thinner container for your Celery app but it works for convenience to demonstrate the ECS deployment and it does give us the ability to leverage both the Celery @shared_task decorator as well as the Django models within the Celery task.

Of course, the real work here is in setting up the applicaiton. Whenever I reference a snippet for a file I will also list the name and location of the file, when you follow along make sure to check out the full content for each file in GitHub . Let's get to it!

Setting up Django

First thing first, make sure you are using a virtual environment. You will be able to find all of the code in the Tree Schema GitHub repo for the Sample ECS project if you would prefer to clone the code. To make things easy it may be worth installing the requirements before beginning; from here on out I'll assume you have these requirements installed. To install the requirements you can run the following snippet from the repo base directory:

          
  pip install -r app/requirements.txt
          
          

To kick things off we will create the Django project using the django-admin command. There are other great starter projects out there such as cookiecutter-django which create a lot of this boiler plate for you and more. Start by initializing the django app (make sure that you replace ecs_example everywhere in this tutorial with the name of your app):

          
  django-admin startproject ecs_example
          
          

As depicted in the official Django tutorial, this creates the following structure:

          
  ecs_example/
    manage.py
    ecs_example/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py
          
          

We are going to make a couple of minor changes to this structure to allow us to inject environment variables separately for development and production. The project should be set up in a way that allows us to have different configuratoins for different environments; we also want to structure our code for modularity so that we can write multiple, independent, Django applications and be able to manage them separately. This is what we're going for:

          
  ecs_example/
      manage.py
      config/
          settings/
              __init__.py
              base.py
              development.py
              production.py
          __init__.py
          urls.py
          wsgi.py
      ecs_example/
          __init__.py
          
          

We've moved the urls.py and wsgi.py under the config directory and we've create a directory for our settings and split out the settings file into three distinct files, one that is the base for which all other settings will inherit, one for development and one for production. We also removed the asgi.py file since we won't be using it to serve our application. You can replicate this structure by running these commands immediately after the django-admin startproject command:

            
    cd ecs_example
    mkdir config
    mkdir config/settings
    touch config/__init__.py
    touch config/settings/__init__.py
    mv ecs_example/settings.py config/settings/base.py
    touch config/settings/development.py
    touch config/settings/production.py
    mv ecs_example/urls.py config/
    mv ecs_example/wsgi.py config/
    rm ecs_example/asgi.py
          
          

Since we moved the settings file we will also need to update the location in the manage.py file to have the default value correspond to our new development file. The location of the root urls.py file will also need to match our new location under the configs and finally the reference to the WSGI should be updated to point to the same location in the configs as well. The original values can be seen here:

Django Default Settings Location
Django Default Root URL Location
Django Default Root URL Location

All three can be changed with the following commands:

            
  sed -i -e 's/ecs_example.settings/config.settings.development/g' manage.py
  sed -i -e 's/ecs_example.urls/config.urls/g' config/settings/base.py
  sed -i -e 's/ecs_example.wsgi/config.wsgi/g' config/settings/base.py
            
            
Further Settings Changes

There are a handful of other changes in the settings files that I recommend you copy/paste from GitHub if you're following along. Make sure to fully copy all of the content in the base.py and development.py files into your local versions EXCEPT for the SECRET_KEY value in the development.py file - you don't want to accidently have the same value that is in a public repo in your code and accidently slip into produciton!

It would be too verbose to fully depict all of the changes here line by line so I will quickly summarize the changes that are made:

  • All of the common settings that apply to both the development and production environment are kept in the base settings, similarly, environment specific settings have been split out to the development and production settings respectively
  • The django-environ package is being used to simplify creating and setting environment variables
  • The configuraions that will not be used, such as Auth password, have been removed
  • Configs for static file management have been added
  • An email backend and Celery configurations have been added
Start Up the App

That should let us run the app, go ahead and run the development server to make sure everything is up and running. As part of the previous set of commands we changed the current directory to be inside the Django project that was created for us, make sure that when you run this manage.py is in your current directory.

            
  python manage.py runserver 0.0.0.0:8001
            
            

You should see some output stating that the application is now running on localhost:8001, plug localhost:8001 into your browser and you should see the defualt Django page.

Configure a Local Database

I will be using SQLite for local development. Django set up the configurations for this when we created the project so we're good to go here! There won't be any custom SQL for union operations, complex windows or anything else that would make us want to validate that our SQL runs with Postgres. If we were, I would highly suggest using Postgres (or whatever database you will end up using) locally as well. Should you decide to go that route here is a one-liner to set up Postgres with Docker, you will be able to find the configurations for how to connect to Postgres in the Django documentation .

            
  docker run -d --rm \
  --name pg-docker \
  -e POSTGRES_PASSWORD=mypgpassword \
  -p 5432:5432 \
  -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data postgres
            
            

Create the User Model

When creating a full application for production it is a good idea to use an existing framework for managing users and authentication, such as django-allauth. The Django community has a ton of these pluggable components that help you get your site off the ground quickly. but for now we're going to create our own User model that allows us to execute on our use-case. Again, since we only want to verify that an email address exists in our database before we send an email to it we will create a User model that essentially just holds the email.

To do this, we'll create a new file: ecs_example/models.py and in that file there will be one new model defined:

Django App Flow
Base User Model

The db_table parameter under the Meta class explicitly defines the name of the table in our database; in general I find that this simply works better than Django's default naming convention. Similarly, we're overriding the base __str__ method so that the value returned has "email: " prepended to it whenever we cast this object to a string.

Now that our User model is created we want Django to create the corresponding table in the database. In order for Django to know that we have created a custom model we need to register our new application with the INSTALLED_APPS in our base settings. The directory name (and also our applicaiton name) just above the models.py is called ecs_example, therefor we will add this string value to the INSTALLED_APPS

Django App Flow
Adding the App to Installed Apps

Once this has been added we can now execute the migrations to update the database. The following two commands will create the set of steps required to update the database and then to execute those changes. Make sure that the name of your application is the last argument in each command.

            
  python manage.py makemigrations ecs_example     
  python manage.py migrate ecs_example 
            
            

You will see this output for each step, respectively, showing the migration steps that Django has created.

Django Default Settings Location
Django Default Root URL Location

We've created our custom User model, but if you remember from a few steps ago in the INSTALLED_APPS there were a handful of other apps that already existed. As a first time setup we will need to run the migrations for these as well; although we will not need to specify a specific app. This will set up our database with the tables that support the session, auth, admin and content types modules that come with Django.

            
python manage.py migrate 
            
            
Django Default Root URL Location

That's it! We shouldn't need to make any more changes to the database for the rest of this tutorial.

Update the Webpage

We now need to replace the landing page that users go to when they come to our website. We will use Django's template engine to serve the HTML file that we want the user to see. The way that we have our template settings configured in the base settings file is that Django will look inside each app directory for a templates directory. We will create HTML files inside our app and serve them as views when the user navigates to a specific page.

Let's create a new file that contains our landing page, the file will be ecs_example/templates/landing_page.html. Inside this file we're going to add some basic HTML, for the sake of readability I'll defer to the file on GitHub for you to copy and read through. Take note that have the string "{ { curr_dt } }" on line 16 in the template file, we will pass in a parameter from Python and Django will update our HTML accordinly. We're not going to fully explore this feature today but it is wonderfully powerful if used properly.

In order for Django to know that we want to serve this file we need to make a few more changes to create a vew that will convert this template into a fully compatible HTML and we also need to update the URL configurations so that the base path points to our template. I'll start with creating a new view that will be used to return the contents of our template by creating the file: ecs_example/views.py and within this creating a function that populates the variable mentioned above:

Serving the landing page
Serving the Landing Page

Next, we need to create a ecs_example/urls.py file that will tell Django how to route requests for this application. Since this is going to be our landing page we will set it to be the default URL, that is, there won't be anything after the ".com" when we visit this site under our domain. This configuration also sets the "view" that will be returned to the user, in this case that resolves to the function that was created above.

Routing to the Landing Page View
Routing to the Landing Page View

We've now given instructions for how to route requests within our app, but we also need to tell Django which requests should be sent to our app. We currently only have a single application, but consider if you had an app that did something completely different such as blog, and you wanted to deploy both applications together. In this scenario you need to tell Django which requests go to which app by updating the config/urls.py. We want events to be routed to our landing page by default therefore this URL path will also have an empty string for the first parameter for path and we will tell it to "include" all of the URLs from our ecs_example app.

Routing to the Application
Routing to the Application

If we run this now we should see the output of our app in the browser, going back to localhost:8001 in the browser hopefully you see this:

Initial App Setup
Initial App in Place

Create Some Vanilla JS

Our web page is in place but now we need to get the data from the user inputs into the backend. If we just sprinkle in the right amount of JavaScript we can give a new meaing to "single page app" with this one landing page app we're developing 😉.

The base settings file contains the configurations for how static files will be managed. In particular, there are several lines shown below that tell Django to:

  • Where to place all of the static files across all apps being deployed
  • The URL to use when referring to static files, you shouldn't need to change this away from "/static/"as Django should work with it behind the scenes
  • What directories to use when looking for static files
  • What backends to use when looking for static files

In practice, the values provided here are relatively standard and should not require too much tweaking.

Initial App Setup
Static File Configs
Developing Front End Validation & AJAX Requests

I know some front end devs are going to read this next part and pull their hair out. Developing for the front end is not my forte, but hey, it works and I'm trying to learn every day.

The JS that we're going to use will be the file ecs_example/static/ecs_example/js/emailUser.js and can be found here . The code only does a few basic things:

  • Validate than a field is not empty when a button is pressed
  • Send the value in the field to the backend when the button is pressed
  • Display some basic messages to the user

Now, we could have used Django forms for this and just used HTML form actions to send the content to the backend. But that wouldn't be full stack! Also, I'm not a big fan of Django forms being used unless there is front end validation being done on the data. Having the backend provide validaiton when a form is being cleaned is a bad user experience.

It is really important that we look at some of the JavaScript code that is currently in the HTML template, you will see this function:

Setting the CSFR Header
Setting the CSFR Header

The CSRF stands for Cross Site Forgery request and it can be a major security risk if not handled. Django does have ways to handle this if you are using their built in forms and we can even make the CSFR token available to the client at any time using the built-in template tags but we would still need to add that value to the POST request manually since we're not using a form submit. The approach taken here grabs the token from the cookies and then adds that to the header for all AJAX requests. Someone on the team found this on stack exchange and it's worked wonders in the areas where Django is used in a loosely-coupled deployment. Without this token you will get a 403 forbidden error from Django when making POST requests.

Handling the Backend Requests

Now that the front end is sending data to the backend, we need to do something with that data.

We will follow the same process as before when setting up a view to serve the template, the only difference is that this time we will not respond with a rendered template but with a JSON object. The JS that was written sends all POST requests to a single endpoint, /email_action/ and it passes an argument that specifies whether to create a new user or to send the user an email. We'll add a single function to the ecs_example/views.py file to handle this request:

Handle Email Action
Handle Email Action

The URLs for the ecs_example app will also need to be updated with the new endpoint. The updated URLs should be:

Updated Application URLs
Updated Application URLs

Doing a quick check again on localhost:8001 should let us verify that the front-end validation and backend code is working. Running through some of the different happy / not happy paths should yield the following messages:

Updated Application URLs
Updated Application URLs
Updated Application URLs

Setting up Celery

You probably noticed that half of the backend function above to handle the send-email action was not yet implemented. Since the process to send an email will be done asynchronously Celery will need to be configured first; we'll walk through that now and then come back and finish the implmentation of that function and from there we should pretty much be done coding!

Before we code, I will quickly touch on why I've chosen Celery for this task. In many use-cases, such as one that our customers here at Tree Schema love the most where we automatically parse your schema, you may want to create an event to be processed later while you respond back to the client immediately. Celery provides simple abstractions for both routing events to queues where they will be processed as well as assigning functions to process from these queues. In addition, it provides monitoring, raising of exception, a nice GUI (Flower) and much more. Building the orchestration to handle this yourself on top of another messaging service can be time consuming to ensure that all of the semantics around processing, dead letter queues and serialization of events would be time consuming. The final reason is that Celery has a really nice Django integration and we get the added benefit of being able to access our Django models.

Another serverless alternative in this space that I like a lot is to write events to AWS SQS and to have it trigger lambdas. If the async process that you're creating does not need access to your Django ORM then going this route may be a better option since the immediate and near limitless scalability of SQS + Lambda is going to be much better than scaling an ECS task.

Add Celery Configs to the Settings

In the config/settings/development.py file we're going to add the celery broker, make sure this is the development file because we will have a different broker configuration for production:

Celery Broker Configuraiton
Celery Broker Configuraiton

Celery does need a persistant data store available and since we will be using Redis in produciton and Redis is super simple to set up locally, we will just run a Docker container for our local redis instance:

            
  docker run -d --rm \
  --name redis \
  -p 6379:6379 \
  redis
            
            

We will need to define a Celery "app", this should be done within the application that you created by adding a celery.py file, for us that will be ecs_example/celery.py. The contents of that file can largely come from the Celery website but you will need to make two changes: first update the default environment variable to point to development settings file and second update your application name. We will use this application name when starting up workers that read from Celery. The rest can largely stay the same for most use-cases.

Celery App Definition
Celery App Definition

One of the coolest features about Celery that I love is this autodscovery of tasks. We can now write our tasks in a way that fits in with the logical organizaiton of our code and the tasks will be picked up and registered with Celery. Our example is not overly complex so we're just going to create a new file ecs_examples/tasks.py right beside celery.py. Inside this file we will define a new function that sleeps for 15 seconds and then sends our email. The task is registered with Celery by using the @shared_task decorator.

Celery Send Email Function
Celery Send Email Funciton

The full set of imports and custom exceptions are in the code. Since we're using the native Django send_mail function the content of the email will be printed to the logs when we are running locally. If we end up using SES in our app or one of the other native email providers that comes with Django, such as Sendgrid, we get the benefit of having the same content of our email be sent by the email provier that we choose.

There is one very important bit of copy/pasta that we need to put into the __init__.py file that corresponds to where our Celery app is, for us that will be the ecs_example/__init__.py. This will ensure that the Celery app is imported when Django starts up. I cannot begin to describe how many cumulative hours I've spent debugging this issue multiple times.. I can never seem to remember this step. If you find that your application is hanging when you try to call delay(...) on your funciton it's likely because the app was not imported during initialization.

Import Celery App in __init__
Import Celery App in __init__

The last step is to just call this function from the existing manage_email_action function in our views.py file. Celery will automatically serialize this event for us and run it asynchronously! We will even deploy the Celery app in AWS later on in it's own container so that it can scale independently of our website. The code that we changed is highlighted below:

Send Emails Async
Send Emails Async

It is important that you do not call your function directly, rather you use the delay function which is made available through the @shared_task decorator. By calling delay you are invoking the process asynchronously. Since we have a relatively trivial task simply using the delay method works fine but the moment we have more than one type of asynchronous task that needs to run we will want to assign queues and priorities to the task. This is where Celery really shines, control over which queues your events are sent to as well as the priority within the queues is immensely powerful when operating on a complex set of activities at scale. On the other side, Celery also offers control over how your workers read and process from a queue. Here is a really concise article that I find provides great context on these topics.

Testing the Celery Integration

To recap, we should already have Django and Redis running locally, if you don't go ahead and start them up. We will now start celery in a separate terminal, using the arguments:

  • -A for application, we will use our ecs_example app (defined in celery.py)
  • -l for the logging level info
  • worker to tell Celery to start a worker to process (from a default queue, which is omitted)
            
  celery -A ecs_example worker -l info
            
            

You should see output that looks like the screenshot below, the redis host we're using (localhost:6379) and our task ecs_example.tasks.send_async_email can both be seen once we start Celery.

Celert Startup Info
Celert Startup Info

All we should need to do now is to go back to localhost:8001 in our browser and enter a valid email address in the second input box and hit enter, we should be able to see the results in the front end immedaitely.

App Synchronous Results
App Synchronous Results

And then after we wait 15 seconds we should see the email being sent via Celery.

Celery Email Sent
Celery Email Sent

I apologize that the text wrapping looks bad here, I couldn't find a good setting for it to display well. The top and the bottom lines of this screenshot show that Celery started and completed processing this event, which we can track from the unique ID provided by Celery. In the middle is the content of our email, since we're using the Django console email backend, as defined our the base settings (seen below), the email content prints to the logs which is really nice for testing.

Base Email Settings
Base Email Settings

Closing Thoughts

Creating a working Django protype with Celery is relatively straight forward and, after you go, through the process once or twice, shouldn't take more than half an hour or so to whip up a nice landing page and string together a few APIs to make it run locally. I cannot speak highly enough about the community libraries available for Django and as an initial reccomendation for further reading I would absolutely suggest reading about django-allauth as discussed above as this is one of the most powerful authentication engines in Django and is something you will absolutely need if you're building a production read application. Further community libraries used in this tutorial and references to their documentation can be found in the requirements file for this project.

The next article will deploy this code to AWS as pseudo production environment, including dockerizing the app, adding an Nginx reverse proxy and deploying it to ECS. Stay tuned for more!


Share this article:

Like this article? Get great articles direct to your inbox