Langs, tech, stuff

Grub, no count down after failed run, how to fix it

2011-11-08T11:57:00.000+01:00

This default grub behavior could be very annoying especially with virtual machines running headless or whatever should stand up after failure.It could be fixed by modifying make_timeout () in /etc/grub.d/00_header:Original code:****

make_timeout ()
{
    cat << EOF
if [ "\${recordfail}" = 1 ]; then
  set timeout=-1
else
  set timeout=${2}
fi
EOF
}

****here is how to fix it:****

make_timeout ()
{
    cat << EOF
if [ "\${recordfail}" = 1 ]; then
  set timeout=${2}
else
  set timeout=${2}
fi
EOF
}

****/I know it is a "secure programming" approach in terms of how this code looks like now... and yes, it is/The original hint is taken from here:http://forum.xbmc.org/showthread.php?t=64857

cssh - cluster ssh - do your cluster work easier

2011-10-11T18:31:00.000+01:00

This is a very simple but powerful tool for interactive work with many machines at once
Just log in to them

cssh node1 node2 node3

and do the same operations on all of them.

It needs x-window (mmm, just wonder if there is a pure-text replacement?) and it is worth to configure ssh keys and perhaps ssh-agent to make your logins easier but still safe.
cssh comes with packages for ubuntu/debian, not sure if redhat/centos rpms catch up here. This project page is http://sourceforge.net/projects/clusterssh/
Many thanks for this tool!

CPAN of the week

2011-09-27T21:38:00.001+01:00

Carp::Always - this is what you need when perl -MCarp=verbose doesn't work the way you expect it. Spotted in some post by JRockway.

Mock::Quick - mock whatever you want, but also takeover/partially override existing modules and classes. The beauty of certainty for TDD or just when escaping from too many dependencies.

Git ignore file

2011-06-05T21:48:00.002+01:00

I just find myself like searching for this again and again as I am not a very heavy git user now, but setting global core.excludesfile seems to be very similar to svn/cvs approach.

How to ignore certain types of files in git?

1) Put the excludes in your $GIT_DIR/info/exclude file (.git/info/exclude), if this is specific to one tree.

2) Run git config --global core.excludesfile ~/.gitignore and add patterns to your ~/.gitignore. This option applies if you want to ignore certain patterns across all trees.

/taken from StackOverflow by Emil

Back ticks equivalent in cmd / windows

2011-03-16T11:51:00.007+01:00

How to get a result of one command and use it in another command in Windows shell, like with backticks in Unix/Linux shell?

Here is a working example:


for /f "tokens=1* delims=" %%s in ('gnuwin32\date +"%%Y%%m%%d_%%H%%M"') do echo output_%%s.log

> output_20110215_1057.log

Works in Windows XP, Vista, Windows 7.

Remember:
1) The variable used in a for loop must be a one letter (at least in XP, Vista)
2) gnuwin32 is a set of GNU tools like grep/sed/awk/date which can be run natively in Windows without cost of additonal Cygwin-like layers

Mr Openoffice is happy with default A4 page

2011-02-10T22:31:00.003+01:00

This is that kind of thing that you know you want to change, but there is always no time to do it. But it is just a minute, really!

http://selinap.com/2008/12/how-to-set-openoffice-default-page-format-to-a4/

Simply:
1) change page format
2) save it as a template (File -> template)
3) set this template as a default (File -> template -> organize -> commands -> set as a default tpl)

Do it, do it!

Rails, flushing buffer log - what a waste of time to fix it

2010-04-05T22:18:00.004+01:00

It was so stupid...

There was a ruby on rails application. There were some changes in a production server. Some missing libraries needed to be added. You are fixing one thing then another, try and fix, try and fix and so on. That is fine as long as we know what needs to be done.

Finally this application was ready to work in development but not in production mode. There was nothing on screen nor in log/development.log even with enabled -u (debug) flag for script/server (you should have ruby-debug gem and ruby-dev package also as a dependency).

What is wrong as I am still getting "Application error
There is an error in this application. Please let us apologize for this inconvenience and help to fix it by reporting this problem to site-master", why there is NOTHING in the log or maybe - where is this f..$%&* log at all? log/production.log is untouched even after shutting down the server.

Ok, probably the error is somewhere in differences between environments so maybe this could be a database.yml, different place of connection socket but why I can not see even a simple error message about this?

After some time for browsing forums, groups and so on the proper solution was to add

Rails.logger.auto_flushing = 1

at the end of my config/environment.rb

My question is - even if this large buffer is understandable solution for heavy production environments why I cannot see anything after closing down my server? Should not this log be flushed then?

If you want to check your logger settings...

Run your rails console:

script/console

and then type:

RAILS_DEFAULT_LOGGER.inspect

so it looks like:

script/console production
Loading production environment (Rails 2.3.5)          
>> RAILS_DEFAULT_LOGGER.inspect

and check @auto_flushing= value.

If you have @auto_flushing=1 then you are at home.

Auto-complete cmd in Windows (like in bash)

2009-06-01T16:40:00.005+01:00

Mostly for Win-XP: It is really handy to have an auto-complete with standard windows shell, cmd. Surprise: it works! The default expand sequence is Ctrl-F.

To change it to more popular tab, open (win-R) regedit and go to HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor, change CompletionChar and PathCompletionChar to 9.

That's it.

Open your cmd window again and enjoy.

The more detailed description you cand find at http://support.microsoft.com/kb/310530

Matrix, Paweł Hajdan, Chromium and Chrome

2008-12-14T00:29:00.007+01:00

I just heard this news yesterday being in Warsaw for a couple of hours: Paweł Hajdan Jr, the member of "Matrix", became a first external commiter to Chromium Code.

See Chromium Blog

Congratulations!!!

"Matrix" is a name of the group that I started 5 years ago to improve the software development and internet services delivery for Polish Scouting Association, ZHR.pl. Paweł was the youngest guy at the first Matrix meeting 5 years ago. It was really nice to see him again with his talk about security in web applications given to the members of Matrix group.

Exif auto-rotate & scale your photos

2008-02-01T20:17:00.001+01:00

Typically after a journey to new place I finish with hundreds of photos from my camera. When I am browsing it on linux (or mac osx) everything is ok - gwenview can read EXIF rotation data and show them in proper direction (rotate on the fly). The problem begin when you upload such "good" portrait image or send it by email - it will be 90deg rotated. Ooops!

A small script can fix it and create subdirectories with 1280x1024 and 800x600 scaled images. It also rotates original images lossless preserving original EXIF data - don't be afraid about the quality!

To use this, simply add packages: imagemagick, exiftran & perl bindings Image::Magick.


#!/usr/bin/perl -w
`exiftran -aip *.JPG`;
`exiftran -aip *.jpg`;

use strict;
use Image::Magick;
use Cwd;

my $currdir = ( getcwd =~ m{([^/]+)$} )[0];

my $webdir = $currdir . '_to1280';
my $web800 = $currdir . '_to800';
print "Creating dirs: $webdir, $web800\n";
mkdir $webdir;
mkdir $web800;

print $| = 1;
foreach my $filename (<*.jpg>, <*.JPG>) {
    # read this
    my $image = Image::Magick->new();
    $image->read($filename);

    # scale it
    print "$filename ";
    $image->Thumbnail(geometry => '1280x1024>');
    $image->write($webdir . '/' . lc($filename));

    $image->Thumbnail(geometry => '800x600>');
    $image->write($web800 . '/' . lc($filename));
    print ".\n";
}

Reasons to switch from CVS to SVN

2008-01-31T23:00:00.001+01:00

Plenty of reasons to switch to SVN/subversion as soon as possible.

SVN is better - easier to usage, no hacks for typical development tasks like
changing file names, directory structure etc.

What developer can do in SVN compared to CVS:

- can change file names and their place in project directory structure preserving
history. ### It is very important if you can make lib named "foo", code it and
week later rename to more apropiate name. This is more agile.

- can store more than plain files: symlinks, file rights. ### There are
still some problems if you would like to store your server files with
exactly the same permissions, but it is much more better than in CVS.

- handling with tags and branches is more intuitive. SVN shows them as
separated directories. ### SVN propose structure for your project where
the default (HEAD) branch for development is named "trunk". Other branches
you have in subdirectory "branches" and tagged versions are under "tags".

- commits in SVN goes together as a "transaction". All files in commit must
pass to be added to SVN repository. ### This is more safer approach than
"every file goes own way" like we have in CVS.

- commits in SVN get unique revision numbers. Single commited files belongs to
such revision number. Single files don't have own version numbers. ###
This is just different than in CVS. It has one advantage: revision number
in SVN means "this exact snapshot of code", the same like tag in CVS.
Compared to CVS, you can obtain snapshot by date (same in CVS - but you
know, this is UTC or localtime, and which timezone, huh?) and also by revision
number, without tagging code.

- mayby this is admin problem but it is simpler to make SVN working through SSL
without unix accounts on remote server (SVN repo). Do you know how to do
it in CVS?

My photo-CV /yeap, I am looking for a new job/

2008-01-17T17:15:00.000+01:00

I used to work hard, in a stress and time-critical environment - it is always a challenge for me.

I like to work with others - to cooperate to solve complex problems! IT world is too complicated to go thorough really alone. RTFMs, lists, IRC, friends...

It is very practical to document my work...

... but it is even better to share my knowledge with other people! /last yapc::eu 2007, vienna/

It is not so easy to optimize code and databases, to make it better, faster, more efficient and reliable - better for users... but it just need to be done /all those apache/mod_rewrite/squid/non-forking-light-servers approaches.../

having my workplace and codebase really nice is also a pleasure to dig more /cvs, subversion, good directory layout, getting best of all /etc/*, .ssh/.rc files/

simplicity of architecture & design, good ERM models and libraries can give project a real power! /from 2NF, 3NF to fast ETL tools and database tuning/

As a good developer, I always try to learn something new and get more from my work /enjoy safaribooksonline, free sources, books to do what need to be done the best way/

No idea for new project?

2007-12-13T03:26:00.000+01:00

No idea how to solve hard IT problems? Try to do something new. Open your mind!

/Stanislaw Wyspianski, "The creation of the world", stained glass, Franciszkanow church, Krakow, PL/

What I hate in perl5 and Catalyst

2007-09-21T20:54:00.000+01:00

Perl - references

Let's remove all simple data-structures like @lists and %hashes. When we think "list" let use list-refs, ref-hashes instead of hashes. Let's keep only scalars and references with implicit dereferencing, like in Python or Ruby. No wantarray context-dependent behaviour! No @{$foo} and \@foo and other nasty code! Perl will be clear and more funny!


current perl:
$foo = [ map { ... } @$bar ]

better perl:
$foo = map { } $bar

As I see folks have problems why we write:

%hash = (foo => 42);
$hash{foo}   # where is hash sign, % ?

@list = qw/foo bar baz/;
$list[1]     # where is list sign, @ ?

and there is no much difference between lists and refs, but differences between various forms of using refs are probably bigger than list<->ref:


$list[1][2]      # wolf
$$list[1][2]     # dog1
$list->[1][2]    # dog2
$list->[1]->[2]  # dog3

Sure, there is only one dog, welcome to TIMTOWDI world!

There are also some troubles with

$ref = \@list;
$ref = [@ref]
# maybe it is @list?
$ref = [ $some->method ]
# or maybe it is %hash?
$ref = { $some->method }

# could be better to make something like
$ref = $some->method->return_as_hashref
$ref = $some->method->return_as_arrayref

Maybe I should stop writing perl code because I think too much about all those things? ;)

Catalyst - index is different than other actions

What is a difference between:

Controller app::foo;

sub index : Private {
    my ($self, $c, $param1) = @_;
}

sub show : Local {
    my ($self, $c, $param1) = @_;
}

?

Index will not catch arguments, will not work with /foo/123, although /foo/show/123 will work that way. To fix index, we need to use LocalRegex

sub index : LocalRegex('^(.*)$') {
    my ( $self, $c ) = @_;
    my ($param1) = @{ $c->req->captures };
}

YAPC::Europe 2007 in Vienna

2007-09-04T00:37:00.000+01:00

Larry Wall, Damian Conway, many great perl hackers, interesting events, all in one place in one time.

It was my first YAPC and also first YAPC talk (slides from Catalyst refactoring are with others from this yapc on slideshare - also check perlbuzz.com) and Bartosz Jakubski's YAPC::EU 2007 photos.

However more interesting was to see in real life how this OS world works, met many interesting people from the perl community. There is probably nothing more interesting than long talks about POE, Catalyst and other kinda-like-app-servers or any other stuff during evenings near the Dunau Kanal and in other nice places in Vienna.

Huh, many words to say how great and cool was this YAPC!!!

PS: I will try to remember difference between Dbix and (Dbic or Dbix::Class) at least to avoid bot activation on #catalyst /and mst should also be happy ;)

Ruby, Rails and Netbeans (compared to Catalyst)

2007-08-16T23:08:00.000+01:00

In current time I am working on my bike-"startup" site, rowerem.org (in Polish: by-bike org) which promotes using bicycles in Krakow and cities at all. This is a free project, no money just fun and opportunity to learn something new.

For long time I was not sure I should do it in perl/Catalyst, some kind of python framework or Ruby/Rails. From the hosting point of view it should be php but... never mind ;) Because I like python, this simplicity, explicit constructions I was checking python frameworks. Unfortunately Django goes own way of MTV, Turbogears is not so well designed (opinions), Pylons - not so popular to have mature quality of all web-related libs. Inspired by latest billboards ads in the city I chose Rails ;)

What is good about Rails?

- Good documentation. You have few good, printed books: Rails Recipes, Rails Cookbook and of course Agile Web Development with Rails. If you have a problem with something, just google for it or check in book. Howtos on rails/doc are also helpful. The step-in cost is very small compared to eg. Perl/Catalyst even when I code in Perl for 10 years when Ruby is still fresh for me.

- not too deep directory structure. It is simpler compared to Catalyst.

- model (ActiveRecord) is really simple to use. Maybe DBIx::Class is powerful but hard to use for night development (after typical day in work). Or I am just too lazy ;)

- good IDE. Have you ever tried Netbeans 6? There are some issues like hard-to-work-without-mouse, hard to (re)define shortcuts / compared to Kdevelop / but... you have good, free IDE for Ruby and Rails projects!

Now it is very funny to extend Catalyst knowledge based on Rails usage - "there should be something like this in Catalyst", this works also in the opposite direction, because both frameworks are quite similar.

Do you use Catalayst? Or maybe you use Rails? What do you think about them?

Mysql, unicode in latin1, bad but not lost!

2007-08-08T21:15:00.000+01:00

If you just created project, have some unicode data but have just found it is a unicode in latin1 fields - simple repair it in 2 steps:

1) change text fields (char/varchar/text) format to BLOB to loose coding info
2) change BLOB to varchar/text setting proper coding.


ALTER TABLE `news` CHANGE `body` `body` BLOB NULL DEFAULT NULL;
ALTER TABLE `news` CHANGE `body` `body` TEXT CHARACTER SET utf8 NULL DEFAULT NULL;

Description: if we have 2/3-bytes sequences as a one non-ascii character in latin1 field, it means we have unicode (eg. utf8) text but for mysql it is a latin1 string. It can work for some time if you do all operations outside database in programming language. But typically you would like to use SQL for sort/comparisons and have data properly coded.
If you did convert directly unicode-in-latin1 to unicode it would be 4-byte-noise. So we loose coding info by BLOB step to leave data untouched.

Perl, unicode, utf-8, mysql

2007-07-20T20:33:00.000+01:00

Handling unicode/utf8 in Perl is quite trivial when you understand "two string approaches".

Perl 5.8 by default can handle UTF-8 strings like a sequence of bytes (1-4 bytes per one char). You can "compress" them to unicode strings with

$wide_char_string = Encode::decode_utf8($octets)

Such encoded string has unicode flag, you can check it with:

Encode::is_utf8($checked_string)

If they have latin1/2/russian/etc. chars after unicode packing:

length($octets) > length($wide_char_string)

Remember: unicode flag does not mean you MUST have wide characters in string.
Wide characters (ord > 255) can be in such unicode string, but it can also has a set of unicode-octets, "unpacked" in unicode string.

Use Test::utf8 to check you have wide chars or not.

FAQ / typical problems:

- check all sources in your program are the same, coded as unicode (wide) or unicode-string. Typically wide-char-strings is a better approach then byte-string (see perldoc Encode). All inputs/outputs like files, DBI, network need to be converted to your choosen internal format.

- do not use use utf8 pragma unless you really need it. In this case all strings have unicode flag, but it does not mean they have wide chars!

(octet_string eq unicode_string) == false

You cannot compare such strings without decode/encode, they are natively different!

Perl, utf8, MySQL

Two approaches to get unicode working with MySQL:

1) after connecting to database, do("SET NAMES 'utf8');
All unicode strings will be octets, without unicode flag.

2) use DBI connection with flag: mysql_enable_utf8 (since DBD::mysql >= 4)
All unicode strings will have wide chars and unicode flag.

Only the second approach works correctly with AutoReconnect flag.

Good news: there is no difference for inserts/updates you use octets or wide_char_strings.

Linux - FreeBSD - Solaris - Windows commands

2007-07-20T19:07:00.007+01:00

Update: Added Solaris (both OpenSolaris and SunOS) and Windows (where applicable) coverage for this set of commands.
Update 2: Coverage for redhat-fedora-centos Linux line / yum package manager.

Linux groups:
* Redhat base: Redhat, Fedora, Centos (yum/rpm packages)
* Debian based: Debian, Ubuntu, Kubuntu (dpkg/.deb packages)

Accounts

Check what groups do I belong to?


{linux, freebsd, solaris}$ groups
{windows - active directory}> dsquery user -samid %USERNAME%|dsget user -memberof

Disk, filesystem

Disk usage

{linux}$ du -sh
{freebsd}$ du -sh

Count subdirectories in current directory:

{linux}$ du --max-depth=1
{freebsd}$ du -d1
{SunOS}$ du

Typical approach to find biggest directories/files on disk:

{linux}$ du --max-depth=1 -kx|sort -n
{freebsd}$ du -d1 -kx|sort -n

Find some kinds of files (regex is a mask for full path, no need for begin/end marks)

{linux}$ find . -regextype posix-extended -type f -regex ".*\.(java|class)"
{freebsd}$ find -E . -type f -regex ".*\.(java|class)"

Show open files and programs:

{linux}$ lsof
{freebsd}$ fstat

Real time disk usage (is there something which shows results for every disk in linux?):

{linux}$ vmstat 3
{freebsd}$ iostat 3

Swap info:

{linux}$ free
{freebsd}$ swapinfo

Networking

Show open ports and apps connected to them:

{linux}$ netstat -apne --inet
{freebsd}$ sockstat
{SunOS}$ netstat
{windows}$ netstat -b
netstat -b -v   # slower but with tree of dependencies

Kernel issues

Show loaded modules:

{linux}$ lsmod
{freebsd}$ kldstat

Load kernel module:

{linux}$ modprobe SomeModule
{freebsd}$ kldload SomeModule

Remove loaded module:

{linux}$ rmmod SomeModule
{freebsd}$ kldunload SomeModule

Program development

Trace the system calls of a program:

{linux}$ strace
{freebsd}$ truss   
(strace is also available in /usr/ports/devel/strace)

libraries - show all paths + libs:

{linux}$ ldconfig -p
{freebsd}$ ldconfig -r

Packages management

Different linux distros make it own way. I'll focus on debian-based distributions like Debian, Ubuntu, Kubuntu etc.

Find which package this file belongs to?

{freebsd}$ pkg_info -W /path/to/checked_file
{debian ubuntu}$ dpkg -S /path/to/checked_file
{redhat centos}$

Have we got a package like... (in (k)ubuntu you can use more friendly tools like synaptic, apt-get, kPackageKit)

{linux}$ apt-cache search your_name
{freebsd}$ cd /usr/ports; make search key=your_name
                          make search name=pear display=name,path
    you can also try simple locate (only in package names):
{freebsd}$ locate -i your_name | grep "/usr/ports/"
{redhat centos}$ yum search name
yum provides name

Install a binary package

{debian ubuntu}$ apt-get install package_name
{redhat centos}$ yum install name
{freebsd}$ pkg_add -r package_name
{windows}$ msiexec /i package.msi

In FreeBSD you have packages made in distribution release time - unfortunately there are no binary upgrades for released version)

Update binary packages

{debian ubuntu}$ apt-get update; apt-get upgrade
{redhat centos}$ yum update

Install a package from sources

{debian ubuntu}$ apt-src
{freebsd}$ cd /usr/ports/path/package; make install clean

...The more I see the less I know...

YAPC Vienna 2007 - my talk

2007-07-13T20:16:00.000+01:00

I will have a Catalyst talk at Yet Another Perl Conference in Vienna, 28th to 30th August 2007 in Web track.

More on conference site,
http://vienna.yapceurope.org/ye2007/talks#18

Yaml, yaml-syck, catalyst...

2007-06-30T14:42:00.000+01:00

If you have problem like:


Couldn't instantiate component "app::Model::app", "->config->{schema_class} must be defined for this model

check if you have YAML::Syck module also. Seems like pure YAML module have bugs and is not enough to parse Catalyst yaml configs.

Just to remember, you can make hierarchy of config files:


app.yaml
app_local.yaml

to override "default" values (eg. for production environment)

Ruby & rails - thanks for nice weekend ;)

2007-06-04T05:24:00.000+01:00

It is early Monday morning and I need to write it (however no one wants to read it).

My wife spent past 2 days in mountains (Tatry, never mind). I spent it with Rails. Or rather - with ruby. When I dig into Rails I sow all these :foo, :bar so I couldn't move forward without knowing WHAT DOES IT MEAN???

This is a time when books like "Programming Ruby" comes in hand. What I think now?

Ruby is really easy especially if you know perl :) I am not a fun of TIMTOWTDI and Ruby seems to be also... However there are ugly constructions like "reverse if/unless" so Ruby programmers can step to the Black World of Many-F^&*-Ways-To-Do-It... but they probably don't want, do they?

We need some ways in language to make different things looking different and Ruby allow us to do it. Oh, it looks like expressive programmers manifesto ;)

So once again:
Pretty nice language, with this object logic you thought "how it will be simple to have it done ONE WAY, objective". Really worth trying. Simple. Code looks nice.

I thought Ruby is too perlish and probably I should focus on clean Python syntax. However Ruby is better designed than Python. There are no "dirties" like:


$> Python
>>> foo = "bla bla bla"
>>> foo.upper()
'BLA BLA BLA'
>>> foo.len()
Traceback (most recent call last):
 File "", line 1, in ?
AttributeError: 'str' object has no attribute 'len'
>>> len(foo)
11

In Ruby it is simple:
(even Python's dir(object) in Ruby is objective: obj.methods or even better obj.methods.sort)

irb> foo.upcase
=> "BLA BLA BLA"
irb> foo.length
=> 11

You can write it "Pythonic/java" way with parentheses, but they are optional:
irb> foo.length()
=> 11

Ok, so what about :foo, :bar and @others?


var = 123   # string
$var = 123  # global variable
@var = 123  # class variable
@@var = 123 # package variable

Classes:


def class1
   attr_reader :duration
end

# which is a shortcut of:

def class1
   attr_reader(:duration)
end

# which is a shortcut of:

def class1
   def duration
       @duration
   end
end

# which is a shortcut of:

def class1
   def duration
       return(@duration)
   end
end

:duration - means you passing "object symbol" not "value of it" (aiui)

If you come from Java world, now you know we don't need "make setters/getters" refactoring option in our Ruby editor/IDE ;)

We can strip methods parentheses(), we can strip returns, we can focus on code logic - this is why frameworks written in Ruby can be pretty nice, easy to understand and to remember. I just wonder when I will see a book named "Ruby best practices".

Java or Ruby? Perl or C++?

2007-06-03T00:54:00.000+01:00

Which of the following positions would you say is most true at your company, assuming (for the moment) that you can only choose one of them:

Above all, we need stability. We have enormous scale and massive business complexity. To create order out of inevitable chaos, we need rigorous modeling for both our code and our data. If we don't get the model and architecture mostly correct in the beginning, it will hurt us later, so we'd better invest a lot of effort in up-front design. We need hardened interfaces -- which means static typing by definition, or users won't be able to see how to use the interfaces. We need to maximize performance, and this requires static types and meticulous data models. Our most important business advantages are the stability, reliability, predictability and performance of our systems and interfaces. Viva SOAP (or CORBA), UML and rigorous ERDs, DTDs or schemas for all XML, and {C++|Java|C#|OCaml|Haskell|Ada}.

Above all, we need flexibility. Our business requirements are constantly changing in unpredictable ways, and rigid data models rarely anticipate these changes adequately. Small teams need to be able to deliver quickly on their own goals, yet simultaneously keep up with rapid changes in the rest of the business. Hence we should use flexible, expressive languages and data models, even if it increases the cost of achieving the performance we need. We can achieve sufficient reliability through a combination of rigorous unit testing and agile development practices. Our most important business advantage is our ability to deliver on new initiatives quickly. Viva XML/RPC and HTTP, mandatory agile programming, loose name/value pair modeling for both XML and relational data, and {Python|Ruby|Lisp|Smalltalk|Erlang}.

- read all Steve Yegge's (previous Amazon.com, now Google developer) weak vs. strong typing (not working? check at wayback machine)

***{**}***
Hmm, interesting - i just realized Steve is the same person I suggested reading in first blog entry, "The Next Big Language".

When is no Perl 6, we can try Ruby

2007-05-11T05:48:00.000+01:00

You know perl, all the funny $_ / $& / q{} vars/ops, inline regexps but want to use object-oriented syntax with more typical dot, my_object.my_sub("blabla")?

There is Ruby, now probably more known as a one of dependency needed to fire up Rails MVC framework ;)

Start from here
http://www.ruby-lang.org/en/documentation/ruby-from-other-languages/to-ruby-from-perl/

look for some code examples
http://migo.sixbit.org/papers/Introduction_to_Ruby/slide-index.html

and of course compare solutions in Ruby/Perl from mythical Perl Cookbook, done by PLEAC folks,
http://pleac.sourceforge.net/

For me Ruby is too perlish - after years of perl programming I fill better when I can do something in more clean Python syntax, with TIOOWTDI. However it's nice to see that Ruby is so close to Perl and there are some hacks you can do in Perl, Ruby but not in Python.

Who use MySQL?

2007-04-02T21:10:00.000+01:00

This text is especially for folks who belive in Oracle/DB2/SQL_Server_XXXX and typically have some idea of MySQL like "maybe it works, but it is not real database", "you get what you pay for", "i used MySQL 3.x but it lacks foreign keys". Et cetera.

Google
Is using MySQL for its AdWords/AdSense system. As some xooglers sed they tried to migrate to some "real database" but after problems with speed and efficiency they come back to MySQL.
"People often ask - do you use Oracle? No, we use MySQL!" - sed Greg Badros at his explanation of Google Ads during Academic Computer Science Festival in Krakow, PL, March 2007.

Yahoo
is using MySQL for Yahoo Finance.

Cox Communications, (4th largest cable tv in USA)
is using MySQL for their data warehouses.

Sabre Holdings (travel/ticket systems)
is using MySQL for their online travel searches

Lycos Europe, myspace, Orbitz, neckermann, Zdnet... all those companies are using MySQL.

Every day on many, many pages you can see google ads. They are powered by MySQL. As you can see - they works. Imagine how many requests get these ads-databases every minute, every second...?

Benchmarks: python. Django vs. perl-> Catalyst

2007-03-21T00:05:00.000+01:00

Ok so i made the benchmarks proper way.

Some stuff to know:
Perl: v5.8.8 built for i486-linux-gnu-thread-multi
Python: 2.4.4c1 (#2, Oct 11 2006, 21:51:02)
Linux: ubuntu 2.6.17-11-generic #2 SMP Thu Feb 1 19:52:28 UTC 2007 i686 GNU/Linux
Apache/2.0.55 (Ubuntu) mod_fastcgi/2.4.2

Django was deployed in apache such way:
FastCGIExternalServer /path/to/fastcgi.fcgi -socket /tmp/mysite.sock

Catalyst was deployed that way:
FastCGIServer /path/to/script/mysite_fastcgi.pl -processes 3

Catalyst was easier to run as a FastCGI - in app/script you have ready to use fastcgi script. With Django you need to find the proper script at their web page. There was also problem problem with template paths - I fixed it by changing to absolute in settings.py. I didn't find on Django webpage the same deployment method as for Catalyst. I tried recipt from "Running Django on a shared-hosting provider with Apache" but got:


runfastcgi(method="threaded", daemonize="false")
TypeError: runfastcgi() got an unexpected keyword argument 'method'

All packages were "out of the box" - from system package manager, cpan or loaded from net (flup).

So how good are these frameworks? How much changed apache compared to own-servers benchmarks I made before?

Tested with: ab -c 25 -n 5000 http://adam/(django|catalyst)/; w
Numbers belowe are averages from 3 measurements/framework. Every measurement was started when 1m load average was below 2.0 (typically 1.75-1.9)

DJANGO


===============================>
Concurrency Level:      25
Failed requests:        21 (in 15k reqs)
Average:                127 requests/sec.
1-min load after:       11.93

CATALYST:


=============================>
Concurrency Level:      25
Failed requests:        0 (in 15k reqs)
Average:                118 requests/sec.
1-min load after:       2.49

Django was little faster however this method of deployment give me quite big number of errors. Catalyst with always 3 fastcgi processes was less CPU consuming and 100% error free.

First benchmarks: Catalyst vs. Django

2007-03-19T22:38:00.000+01:00

This comparison is little silly because I don't have time to deploy mod_python / fastcgi / lighttpd or anything like such-good-approach. I just wrote the same code (make SQL join, show template with same fields, same logic) and check it with ab (apache benchmark tool) against framework own servers.

Catalyst


Server Software:
Server Hostname:        localhost
Server Port:            3000

Document Path:          /gallery/
Document Length:        7185 bytes

Concurrency Level:      1
Time taken for tests:   64.568406 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      36675000 bytes
HTML transferred:       35925000 bytes
Requests per second:    77.44 [#/sec] (mean)
Time per request:       12.914 [ms] (mean)
Time per request:       12.914 [ms] (mean, across all concurrent requests)
Transfer rate:          554.68 [Kbytes/sec] received

Django


Server Software:        WSGIServer/0.1
Server Hostname:        localhost
Server Port:            8000

Document Path:          /
Document Length:        4981 bytes

Concurrency Level:      1
Time taken for tests:   61.358104 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      25625000 bytes
HTML transferred:       24905000 bytes
Requests per second:    81.49 [#/sec] (mean)
Time per request:       12.272 [ms] (mean)
Time per request:       12.272 [ms] (mean, across all concurrent requests)
Transfer rate:          407.84 [Kbytes/sec] received

With some modifications (eg: running together) the timings are like 100:96 for Django. A little faster. Will it differ much if we deploy it with Apache-and-the-fastest-way?

Django (python) vs. Catalyst (perl)

2007-03-18T15:07:00.000+01:00

Intro

I spend many days with Perl MVC framework - Catalyst. Catalyst for perl is like Rails for ruby but without all this marketing hype & noise (and unfortunatelly with lack of good books, tutorials and doc). I tried to use Rails for some easy-but-existing database. I was surprised that Rails ActiveRecord was not able to quote SQL names to get data from - as I remember - "lines" column in MySQL. Little stupid, hmm?

Anyway Ruby is too perlish. I know perl and know how much cost all this TIMTOWTDI when your project is growing up. Python is like clean, better Perl (however Python libraries are not at this quality level like Perl libs in CPAN - eg. unicode translations). So this is "why python".

Few days ago I started small project with Django. Here are first thoughts and differences:

Django view = Catalyst controller

Django call itself MTV framework (model-view-template). Going this way Catalyst is Model-Controller-(View)-Template - the only difference is: Django models == Rails & Catalyst controllers.

Django global switch

- In Django you need to define every URL action in urls.py and create appropiate "views"

Example of urls.py (aka "global switch"):


urlpatterns = patterns('',
    ('', 'hnl.photos.views.index'),                      # root page
    (r'^admin/', include('django.contrib.admin.urls')),  # admin
    (r'^photo/update', 'hnl.photos.views.update'),       # update
)

- In Catalyst, there is no such "global switch". You just create controllers (perl classes), put methods with "local" attribute inside - and they are visible in your URL namespace. Why you should define this namespace mapping if it is enough to do it only by proper view class-function naming? For me it looks like Catalyst dispatcher is more "pythonic" than Django.

Output validation

I created view to import some data from Yaml data-dump to MySQL database. Everything works however Django cannot display some rows because of problems with validation. Message error is not so helpfull -

ValueError at /
year is out of range

Hmm, but which field (i have 2 dates in every record) in which row? Exception is caught in template (first line)


{% for repo in gallery_list %}
    * {{ repo.galleryname }}
{% endfor %}

but as you see - I do not fetch any date-time values here!

After looking into data I found '0000-00-00' date. Who pasted this record into database without problems? Django db model. Who was not able to show it? Django! I think it's a better idea to validate/filter data at input rather than output. Or in both places.

Objects in perl 5

2007-03-14T19:00:00.000+01:00

There are many ways to do it. The old-school approach

blessed hash


sub new {
    my $classname = shift;
    my $rh = {};      # Reference to Hash, rh :)
    bless $rh, $classname;
    return $rh;
}

All variables can be created just by assigning something to blessed hash:


my $foo = my_class->new();
$foo->{bar} = 42;
print $foo->{bar};

Unfortunatelly there is lack of encapsulation - you have access to all methods and properties.

Get control of object properties - use fields


package my_class;
use fields qw(bar);     # declare used object vars
sub new {
    my $classname = shift;
    return fields::new $classname;
}

# usage:
my my_class $foo = my_class->new();
$foo->{baz} = 42;  # trying to use not declared field - error!
print $foo->{bar}; # works

We have control on object properties. Anyway, we still do not have private variables.

Private variables (scalar-bless)


package my_class;
{
   my %bar;  # All vars are declared as hashes
   sub new {
       my $classname = shift;
       my $rs = \do{ my $scalar }; # reference to scalar
       bless $rs, $classname;
       return $rs;
   }

   sub set_bar {
       my $this = shift;
       $bar{$this} = shift;
   }

   sub get_bar {
       my $this = shift;
       return $bar{$this};
   }
}

And now all variables are private!!!

Potential problem: it's not working with threads. To have a copy of such object instance in threads, you need to workaround this closure with get_all_data()/set_all_data() methods - and all object data need to be passed as additional parameters when creating new thread.


my $foo = my_class->new();
$foo->{bar} = 42;      # sorry, no bonus! Use set/get method instead.
print $foo->{bar};     # This is private variable, not accessible outside class.
print $foo->get_bar(); # works!

Private, public, protected - objects like in perl6:

like in manual:


use Perl6::Classes;

class Composer {
    submethod BUILD { print "Giving birth to a new composer\n" }
    method compose { print "Writing some music...\n" }
}

class ClassicalComposer is Composer {
    method compose {
        print "Writing some muzak...\n";
        $_->do_private;
    }
    method do_private is private { print "really private thing\n" }
}

class ModernComposer is Composer {
    submethod BUILD($) { $.length = shift }
    method compose() { print((map { int rand 10 } 1..$.length), "\n") }
    has $.length;
}

my $beethoven = new ClassicalComposer;
my $barber    = new ModernComposer 4;
my $mahler    = ModernComposer->new(400);

$beethoven->compose;   # Writing some muzak...
$barber->compose;      # 7214
#$beethoven->do_private;
compose $mahler;       # 8927586934796837469875

Pretty well, uhh? For me it was like "look, this is almost like in Java!" Just change is to extends.

Perl6::Classes works as a filter for code and do all this translation job. Well, we pay for it - it is 20x slower than hash/scalar blessed objects. If this latency doesn't matter - use it.

1, 2, 3, start

2007-03-14T01:21:00.000+01:00

For nice start - some text to read - what about language of the future - see Steve Yegge's The Next Big Language