Initial setup Django 1.5 with django-celery 3.0 and rabbitmq broker

System/Requirements – Django 1.5 with django-celery 3.0 and a rabbitmq broker on ubuntu 13.10 (Xubuntu)

• Assumption – you already have a functioning django site up and running, you can see my previous post.

The end goal is to be able to observe and track tasks (especially longer running tasks) from the django admin. For example, to use the following task.py

from celery import task

@task
def add(x, y):
    return x + y

Then observe tasks being recorded in the admin site. http://127.0.0.1:8000/admin/djcelery/taskstate/?

Using Bootstrap with Django

Part 2

This is a continuation from the previous post on using django-skel to create a new django project.

Overview

In this post we will walk through using Bootstrap (a front-end framework for developing responsive projects) github.

Assumptions: you have already completed the previous post and have an initial working django site.

Goal: The result of this tutorial / walk-through is a django page with the basic template from the http://getbootstrap.com/getting-started/ guide.

Using django-skel to create a new Django site

Part 1

This is the first step – in Part 2 I cover adding bootstrap support.

Overview

Create a django site using django-skel on xubuntu 13.10 using django 1.5.

I have not created a new django site for a while (having been consumed with opencv and TurboGears), so I wanted to start a fresh walkthrough (also to prepare for startup weekend http://redesignsea.startupweekend.org/). By the end I was able to get an admin page going, a good layout for the project, and lots of helpful developer tools in place.

Following the excellent documentation provided for django-skel:

• https://github.com/rdegges/django-skel
• http://django-skel.readthedocs.org/en/latest/

Python depth first search with graphviz

Graphviz is an awesome tool to help visualize graphs, and there is a handy interface for python http://github.com/pygraphviz. This is a basic example of how to do a depth first search in python and then visual the results. I will walk through the basic steps to get graphviz and what I did to make it work.

NOTE: graphviz provides search implementations, I have created a beginner/toy implementation for demonstration purposes.

Requirements

1. Linux – Xubuntu 13.10 – can be made to work for other systems but this is what I am using for this guide.
2. Python 2.7 – assumed that you got it as part of xubuntu.
3. virtualenv – could do without but I have used it here.

Example Results

What you can expect at the end – generate a png for each step of the depth first search

Working through MongoEngine Tutorial

Having reviewed several available options – I decided it would be fun to try MongoEngine with my new Django site.

I will be walking through this tutorial to start with – http://docs.mongoengine.org/en/latest/tutorial.html. I will start where I left off with the skeleton site.

Getting Started

The suggested installing it with “$ pip install mongoengine”

(lazybower)rcstats@ubuntu:~/Desktop/lazybower/LazyBower$ pip install mongoengine
Downloading/unpacking mongoengine
 Downloading mongoengine-0.8.4.tar.gz (124kB): 124kB downloaded
 Running setup.py egg_info for package mongoengine
 0.8.4

 no previously-included directories found matching 'docs/_build'
Requirement already satisfied (use --upgrade to upgrade): pymongo>=2.5 in /home/rcstats/Desktop/lazybower/lib/python2.7/site-packages (from mongoengine)
Installing collected packages: mongoengine
 Running setup.py install for mongoengine
 0.8.4

 no previously-included directories found matching 'docs/_build'
Successfully installed mongoengine
Cleaning up...

I then went ahead and added it to the dev requirements (requirements/dev.txt).

Werkzeug==0.8.3
 flake8==2.0
 ipython==0.13.2
+mongoengine==0.8.4

python manage.py shell_plus
In [1]: from mongoengine import *

In [2]: connect('tumblelog')
Out[2]: MongoClient('localhost', 27017)

Complete the Tutorial

I ran through without incident just pumping the commands suggested in the tutorial into the django ipython shell.

(lazybower)rcstats@ubuntu:~/Desktop/lazybower/LazyBower$ python manage.py shell_plus

from mongoengine import *

connect('tumblelog')

class User(Document):
 email = StringField(required=True)
 first_name = StringField(max_length=50)
 last_name = StringField(max_length=50)

class Comment(EmbeddedDocument):
 content = StringField()
 name = StringField(max_length=120)

class Post(Document):
 title = StringField(max_length=120, required=True)
 author = ReferenceField(User, reverse_delete_rule=CASCADE)
 tags = ListField(StringField(max_length=30))
 comments = ListField(EmbeddedDocumentField(Comment))
 meta = {'allow_inheritance': True}

class TextPost(Post):
 content = StringField()

class ImagePost(Post):
 image_path = StringField()

class LinkPost(Post):
 link_url = StringField()

Mongo Cheat Sheet

http://docs.mongodb.org/manual/tutorial/getting-started-with-the-mongo-shell/

So you can double check your work, here is the cheat sheet I use:

Results

That gives an overview of my experience getting started with MongoEngine. I was fairly pleased and will move on to integrating it with a Django app in the following post.

 

Starting a new Django site

Overview

Start a new django site using one of the available skeleton/templates for django projects on a Linux Ubuntu system. This will be a record of the steps and modifications I needed to make this run on my developer system.

• Results available via github project https://github.com/AnthonyHonstain/LazyBower

Assumptions – this guide uses xubuntu 13.04 and assumes you already have django installed. This walkthrough was just intended to be a record of the steps required to use a skeleton project. You can find the django install guide here along with their excellent tutorial.

Django Memcache

Overview

I want to increase the performance of several pages on http://www.rc-stats.com, there are a few problematic pages that are taking several seconds to load. They have rarely changing content, but somewhat complex queries to the DB underneath them. I have decided to use memcache and will be using the following references as a guide:

• http://docs.webfaction.com/software/django/config.html
• http://stackoverflow.com/questions/2899178/using-memcached-in-python-django-questions
• http://www.djangobook.com/en/2.0/chapter15/

Fixing an bug in the race uploader (txt file to PostgreSQL)

Overview

When I implemented the automatic file uploader I neglected to add a test case for the time field that stores the race time (it comes in in the format <hour>:<minute>:<second>.<microseconds>). My first implementation had worked correctly and I had never had any tests to verify this particular field was getting process correctly.

Now I needed to fix the uploader and try to fix the incorrect data in the database without having to reprocess all the old results.

Performance Tweaks

Overview

This is simply a record of my investigation into merging and compressing my static resources (css and js files) on a personal website. What follows is the resources I looked at and how I hoped to use them on my site.

In summary, I have not found a good solution and I will revisit the subject in the future.

Django Basic File Upload

As part of the ongoing improvements to the web site http://www.rc-stats.com/, I have decided that I want contributors to be able to upload race results. Currently I am using a python script to upload the results myself.

Some Back Ground on The Current Process

Currently I use set of python scripts and pgdb to parse the race results and feed them into the postgresql database. The code to feed results into the database was a bit rough and certainly not robust enough enough for contributors to work with (it has been a bit of bootstrapping as I learned more about security with respect to django and postgresql).

This was a design decision I made early on. That I would prioritize getting results into the system quickly, the consequence being that I would need to manually run script to upload races results every time they became available. Now that the site is live and I am able to start getting user feedback, the time has come for me to clean up this process.

References To Start From

• Django guide: https://docs.djangoproject.com/en/dev/topics/http/file-uploads/?from=olddocs
• Stackoverflow description: http://stackoverflow.com/questions/5871730/need-a-minimal-django-file-upload-example

Initial Prototype

Using the django documentation and some stackoverflow examples I arrived at the following code.

upload_start.html

{% extends "base.html" %}

{% block pagetitle %}Upload Race Results{% endblock %}
{% block pagetitle_description %}Upload race results from the scoring system.{% endblock %}

{% block content %}

<section class="full_width">
<h2>{{ error_status }}</h2>
	<form action="/upload_start/" method="post" enctype="multipart/form-data">
	{% csrf_token %}
	<input type="file" name="file" />
	<input type="submit" name="submit" value="Upload" />
</form>

{{ form.errors }}
{{ form.non_field_errors }}
</section><!-- /.full_width -->
{% endblock %}

views.py

class UploadFileForm(forms.Form):
    #title = forms.CharField(max_length=50)
    file  = forms.FileField()

@login_required(login_url='/login')
def upload_start(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)

        # What causes the form to not be valid?
        if form.is_valid():
            # Need to make sure the key used for FILES[ ] matches up with the
            # form in the template.
            handle_uploaded_file(request.FILES['file'])
            return HttpResponseRedirect('/upload_start')
        else:
            error = "Failed to upload file:" + request.FILES['file'].name
            return render_to_response('upload_start.html', {'form':form, 'error_status': error}, context_instance=RequestContext(request))

    else:
        form = UploadFileForm()
    return render_to_response('upload_start.html', {'form': form}, context_instance=RequestContext(request))

def handle_uploaded_file(f):
    #print "name:", f.name
    #print "size:", f.size
    with open('/home/username/Desktop/rcstats/rcdata_media/fileupload/testfile.txt', 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)

Some of the initial stumbling points I hit that are worth a note:

• I needed to have {% csrf_token %} added to the form, several older examples do not mention this.
• I was uncertain where the file should be written to, for the time being I have created a folder in the django media folder (I suspect this will need some revision when I get ready to run this on the live server).
• Apache needs permission to write to directory you plan to upload in. I used this reference to make sure that www-root has permission.
  • http://stackoverflow.com/questions/5543641/django-media-upload-errno-13-permission-denied
• I was having trouble with invalid Forms, I was not getting much in the way of an actionable error message. Inserting {{ form.errors }} in the template helping understand the problem.
  • http://stackoverflow.com/questions/5516437/django-form-has-no-errors-but-form-is-valid-doesnt-validate

Result

At this stage, this is sufficient to result in a file getting placed in the assigned directory. I am in the process of designing the E2E user experience through this process, I will follow up with an additional post to cover the next steps.