Wednesday, March 21, 2007

Benchmarks: python. Django vs. perl-> Catalyst

Ok so i made the benchmarks proper way.

Some stuff to know:
Perl: v5.8.8 built for i486-linux-gnu-thread-multi
Python: 2.4.4c1 (#2, Oct 11 2006, 21:51:02)
Linux: ubuntu 2.6.17-11-generic #2 SMP Thu Feb 1 19:52:28 UTC 2007 i686 GNU/Linux
Apache/2.0.55 (Ubuntu) mod_fastcgi/2.4.2

Django was deployed in apache such way:
FastCGIExternalServer /path/to/fastcgi.fcgi -socket /tmp/mysite.sock

Catalyst was deployed that way:
FastCGIServer /path/to/script/ -processes 3

Catalyst was easier to run as a FastCGI - in app/script you have ready to use fastcgi script. With Django you need to find the proper script at their web page. There was also problem problem with template paths - I fixed it by changing to absolute in I didn't find on Django webpage the same deployment method as for Catalyst. I tried recipt from "Running Django on a shared-hosting provider with Apache" but got:

runfastcgi(method="threaded", daemonize="false")
TypeError: runfastcgi() got an unexpected keyword argument 'method'

All packages were "out of the box" - from system package manager, cpan or loaded from net (flup).

So how good are these frameworks? How much changed apache compared to own-servers benchmarks I made before?

Tested with: ab -c 25 -n 5000 http://adam/(django|catalyst)/; w
Numbers belowe are averages from 3 measurements/framework. Every measurement was started when 1m load average was below 2.0 (typically 1.75-1.9)


Concurrency Level: 25
Failed requests: 21 (in 15k reqs)
Average: 127 requests/sec.
1-min load after: 11.93


Concurrency Level: 25
Failed requests: 0 (in 15k reqs)
Average: 118 requests/sec.
1-min load after: 2.49

Django was little faster however this method of deployment give me quite big number of errors. Catalyst with always 3 fastcgi processes was less CPU consuming and 100% error free.


makoto tsuyuki said...

You should be use mod_python for testing Django.
mod_python does not give an error.
And it's faster than fastcgi several times.

Adam Bartosik said...

Probably yes - but mod_python is not an acceptable approach for typical shared hosting (without jails). The same problem is with mod_perl anyway.

Mike said...

"... mod_python is not an acceptable approach for typical shared hosting ..."

I'd say you need to find a better host. webfaction has awesome Django support.

If you are trying to choose between the two frameworks, you can't just remove one of the advantages of one framework (mod_python is really one of Django's pluses) and then claim it's deficient!

What you are actually comparing here isn't "Benchmarks" it's "FastCGI Benchmarks".


Adam Bartosik said...

Mike, true - this benchmark i was doing with easy hosting in mind. To be honest - Catalyst is also not so easy to deploy, because you need to install dozens of CPAN modules. But - Catalyst works almost perfectly with FastCGI, FCGI or mod_perl. More than one way to use it ;)

Staying with fastcgi/fcgi approach, Catalyst is even easier to configure than Rails (you don't need to use mod_rewrite for static content - there is Catalyst plugin).