stereochrome

Inklings in March 2013

Trademark Clearinghouse Automated Interface

Built on top of EPP, which is more convenient for me anyway.

A Brief Intro to Profiling in Python

Jazz that nobody asked for

Project minikrebs: Tiny TP-Link WiFi AP Goes Big with Webcam, NFC, USB/IP, Automagic Internets and Rick-Rolling

Unofficial Google Reader API documentation

Sherlockcat and the Case of the Poopy Pawprints, a Book in Two Volumes

Volume I opens in 221B Dodger Street, with Sherlockcat and Dr Watson preparing for dinner, when Ms Lawson, his landlady and sometime housekeeper, announces that Inspector Lestrade has asked for his assistance concering the appearance of poopy pawprints around London.

As Sherlockcat and Watson investigate, the evidence begins to point the blame at Sherlockcat himeself. However, Sherlockcat discovers that the actual perpetrator is none other than Professor Maowriarty!

With no solid evidence of Maowriarty’s guilt and the police closing in on him, Sherlockcat is forced to flee until he can find the evidence to clear his name.


Volume II opens with Dr Watson, who has been attempting to track down Sherlockcat ever since he absconded, discovering Sherlockcat in a catnip den in Shanghai. Fearing the worst for his friend, he attempts to intervene, but Sherlockcat convinces him that he is, in fact, only in the den as part of his investigation of Maowriarty, the actual cat in the mirror in the first book.

Sherlockcat discovers that the events of the first book were simply a rouse to make sure that Sherlockcat, as the only intellect Maowriarty considered his equal, was discredited and thus could not intervene in his true master plan: the cornering of the world yarn trade.

After barely escaping Shanghai with their lives, Sherlockcat and Watson track Maowriarty to Austria, where Sherlockcat confronts Maowriarty in a castle in the Austrian Alps. In the ensuing fight, both Sherlockcat and Maowriarty tumble over a balcony in to a raging waterfall, falling to their doom.

Or did they? Has Maowriarty been defeated? Will Sherlockcat return? Find out in the next book: “The Pups of the Baskervilles”.

GeoNames

The GeoNames geographical database covers all countries and contains over eight million placenames that are available for download free of charge.

Commercial access costs are pretty reasonable too.

Song of GitHub

Let me sing you the song of my contributions.

I haven’t listened to what mine sounds like, but it’s here.

Firefox and Chromium plugins I'm using

Since I’m doing a tab clearout of my browser in work, I’d might as well note the plugin’s I’m using:

Work (Debian 7.0)

Firefox/Iceweasel 10.0.12

wireframe.cc

Wireframing tool.


[It’s just occurred to me that when I’m coding up this site’s replacement, I should really include something for tagging.]

Alembic

Database migrations for SQLAlchemy.

pathod and pathoc

Pathological HTTP server and client designed for fuzz testing HTTP clients and servers.

Skeleton

Mobile-friendly CSS framework. Source is on Github.

IETF Tools

password-store

Simple password manager using gpg and ordinary unix directories.

Repo is here.

Setting up ReviewBoard without mod_wsgi

Yesterday I set up Review Board at work to see if it might be a better way to conduct code reviews than RhodeCode, which has code review functionality, but it’s never worked particularly well for us.

Owing problems we’ve had in the past1, we rarely use mod_wsgi these days and prefer run our applications as standalone daemons managed by Supervisor that listen on the loopback interface, with Apache2 acting as a reverse proxy using mod_proxy.

First thing I did was create a virtual environment for the application to run in:

# mkdir -p /opt/reviewboard
# cd /opt/reviewboard
# virtualenv --no-site-packages env

The server in question is running Debian Squeeze, which makes it necessary to use the --no-site-packages flag with virtualenv.

Review Board requires easy_install, so pip is a no-go:

# env/bin/easy_install ReviewBoard

It’s advisable to have a memcached instance running, so that was installed from APT.

As we authenticate off of a central LDAP server, I installed the python-ldap from PyPI into the virtual environment with pip:

# env/bin/pip install python-ldap

As we’ll need a WSGI server to run it under, I chose to use waitress. As of waitress 0.8.4, waitress comes with a command line runner I contributed called waitress-serve3. It makes starting instances of waitress on the command line easier:

# env/bin/pip install waitress

With that done, it’s time to generate the site with the rb-site command:

./env/bin/rb-site install site

After asking a few questions, that will create a directory called /opt/reviewboard/site to contain the site assets.

Out of the box, reviewboard doesn’t have a way to run standalone. To get around that, I had it generate a mod_wsgi site, given that was the closest option, and took a look at the .wsgi file generated. It looked like this:

import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = "reviewboard.settings"
os.environ['PYTHON_EGG_CACHE'] = "/opt/reviewboard/site/tmp/egg_cache"
os.environ['HOME'] = "/opt/reviewboard/site/data"
os.environ['PYTHONPATH'] = '/opt/reviewboard/site/conf'

sys.path = ['/opt/reviewboard/site/conf'] + sys.path

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

As we’ll be running this as a standalone server to be managed by Supervisor, I used this as the basis of the Supervisor configuration block for the application:

[program:reviewboard]
command=/opt/reviewboard/env/bin/waitress-serve
  --expose-tracebacks
  --url-scheme=https
  --host=127.0.0.1
  --port=8005
  --call
  django.core.wsgi:get_wsgi_application
user=nobody
stdout_logfile=/opt/reviewboard/site/logs/%(program_name)s.out.log
stderr_logfile=/opt/reviewboard/site/logs/%(program_name)s.err.log
environment=
  DJANGO_SETTINGS_MODULE="reviewboard.settings",
  PYTHON_EGG_CACHE="/opt/reviewboard/site/tmp/egg_cache",
  HOME="/opt/reviewboard/site/data",
  PYTHONPATH="/opt/reviewboard/site/conf"

Normally, waitress-serve expects a WSGI application object to be specified, but as none is created by default in a Django application, I used its --call flag so that a callable could be specified to call to get an application object.

With that set up, it’s time to set up Apache:

<VirtualHost *:80>
    ServerName reviewboard.example.com

    RewriteEngine on
    RewriteCond %{HTTPS} !^on$ [NC]
    RewriteRule . https://%{HTTP_HOST}%{REQUEST_URI} [L]
</VirtualHost>

<VirtualHost *:443>
    ServerName reviewboard.example.com

    DocumentRoot /opt/reviewboard/site/htdocs
    ProxyPass /static !
    ProxyPass /media !
    ProxyPass /errordocs !

    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:8005/
    ProxyPassReverse / http://127.0.0.1:8005/
    SetEnvIf X-Url-Scheme https HTTPS=1
    RequestHeader Set X-Forwarded-Proto https

    SSLEngine On
    SSLCertificateFile /path/to/certificate.crt
    SSLCertificateKeyFile /path/to/certificate.key
</VirtualHost>

The ProxyPass directives just following DocumentRoot ensure that any static content is handled by Apache. The block that follows that passes any other traffic through to the server we’ve bound to the loopback interface.

With all that in place, tell Supervisor and Apache to reload their configuration:

# supervisorctl update
# service apache2 graceful

And bingo! It should be working!

Caveat regarding LDAP authentication

When I set up LDAP authentication, I noticed it wasn’t working and there were some bizarre errors in the log:

WARNING:root:LDAP error: {'info': '(unknown error code)', 'desc': "Can't contact LDAP server"}

After some hunting and experimenting, I discovered that the problem was that python-ldap was doing certificate checking. To get around that, I opened up the reviewboard.accounts.backends module, found the LDAPBackend class, and in the authenticate() function, inserted the following statement following the import ldap statement:

ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW)

And it was able to authenticate off of LDAP.


2013-05-27: Updated for waitress 0.8.4: the waitress-serve package is no longer required as it’s been merged into waitress proper.


  1. YMMV: mod_wsgi is still a solid piece of software, but it just didn’t suit our deployment scenarios. 

  2. We may switch to Nginx for this for lower overhead, but it’s not a priority. 

  3. It doesn’t currently support WSGI middleware though. I didn’t feel strongly about adding support for middleware at the time, but I may change my mind in the future. It wouldn’t be a difficult thing to add.