A dynamic SQL query is a query that operates accordingly through parameters.

Sometimes it is necessary to query the database filtering the rows in base at the user input. Like in a search, an user would choose several filters and not others.

The examples are in PostgreSQL and Python, but can be easily adapted to any database and language.

Imagine a table of customers:

CREATE TABLE "customers" (
    "id" serial PRIMARY KEY,
    "name" varchar(64) NOT NULL,
    "active" bool NOT NULL,
    "country" varchar(32)
);

The classic approach

The classic approach is to compose the SQL joining strings and conditional constructs, for example:

# Function that performs the query
# The parameters have null defaults
def customers_filtered(active=None, country=None):
    # This is the beginning of the query
    query = '
        SELECT *
        FROM customers
        WHERE'

    filters = []

    # Add the active filter, if not null
    if active is not None:
        filters.append('active=%(active)s')

    # Add the country filter, if not null
    if country is not None:
        filters.append('country=%(country)s')

    # Join all the filter queries chunks with an AND
    query += ' AND '.join(filters)

    # Prepare the parameters
    parameters = {
        'active': active,
        'country': country
    }

    # Execute the query and return the results
    return execute(query, parameters)

The resulting code is long, the query takes times to write and is unclear to read.

The database have to make an execution plan before executing the query. The execution plan is re-used when the SQL query is static, but this query is dynamic so the execution plan will be not re-used a second time and making the query a bit slower.

Optional WHERE clause

There is a solution with a negligible impact on perfomances, much easier to write and to read. The trick is to use NULL conditionals, so the query will be:

# Function that performs the query
# The parameters have the null defaults
def customers_filtered(active=None, country=None):
    # This is the full query!
    query = '
        SELECT *
        FROM customers
        WHERE (%(active)s IS NULL OR active=%(active)s)
              AND (%(country)s IS NULL OR country=%(country)s)'

    # Prepare the parameters
    parameters = {
        'active': active,
        'country': country
    }

    # Execute the query and return the results
    return execute(query, parameters)

The trick is to validate the right part of the OR clause only if the parameter is not NULL.

Optional JOINs

A similar approach can be applied for the JOIN clauses. First, create the “orders” table:

CREATE TABLE "orders" (
    "id" serial PRIMARY KEY,
    "customer_id" int NOT NULL,
    "date" timestamp NOT NULL,
    "value" int NOT NULL
);

It is possible to use a dynamic JOIN in order to get or not expensive data to calculate. The following SUM will be performed only if needed, setting the “flag” as true:

SELECT u.id, u.name, u.active, SUM(o.value) AS total
FROM customers AS u
LEFT JOIN orders AS o ON (true=%(flag)s AND o.customer_id=u.id)
GROUP BY u.id

Also in this case the performance impact is risible and the query remains very clear to read.

Conclusions

The classic method of composing the queries can be a source of bugs and headaches, this should be avoided. These solutions can reduce the code and increase the readability, especially when the ORM cannot be used or with the CQRS pattern (Command Query Responsibility Separation).

A video tutorial

Nat Dunn has created a nice video tutorial inspired by this article, it is part of his Python course:

Discussion on
Hacker News: https://news.ycombinator.com/item?id=8842677
Reddit: http://www.reddit.com/r/Database/comments/2rh0ak/dynamic_pure_sql_where_clauses_and_optional_joins/
Twitter: https://twitter.com/davmuz_en/status/552295916385087489

In the CGI mode our program will be executed at every request. Go loads pretty fast so it may run fast like PHP, but it is possible to use a better approach.

FastCGI have the same protocol like CGI, but the web server loads the program one time then submit the requests. This is much more efficient, but the program have to be written better in order to keep a low use of RAM since the process will remain active for a while.

The web server can communicate with a FastCGI program in three manners: via the standard input/output, via a UNIX socket or via a TCP connection. In a shared hosting environment the standard input/output way is the most available and the web server will load the program for you automatically. Instead, using a UNIX or TCP socket, you will have to load and manage your program yourself monitoring it if it crashes (see Supervisord for example), because the web server can’t handle it for you.

The code

This example requires github.com/gorilla/mux and it’s tested on Golang 1.1.2, but it’s not mandatory:

go get github.com/gorilla/mux

The following is the source of the program (fcgitest.go):

import (
    "flag"
    "github.com/gorilla/mux"
    "io"
    "log"
    "net"
    "net/http"
    "net/http/fcgi"
    "runtime"
)

var local = flag.String("local", "", "serve as webserver, example: 0.0.0.0:8000")

func init() {
    runtime.GOMAXPROCS(runtime.NumCPU())
}

func homeView(w http.ResponseWriter, r *http.Request) {
    headers := w.Header()
    headers.Add("Content-Type", "text/html")
    io.WriteString(w, "<html><head></head><body><p>It works!</p></body></html>")
}

func main() {
    r := mux.NewRouter()

    r.HandleFunc("/", homeView)

    flag.Parse()
    var err error

    if *local != "" { // Run as a local web server
        err = http.ListenAndServe(*local, r)
    } else { // Run as FCGI via standard I/O
        err = fcgi.Serve(nil, r)
    }
    if err != nil {
        log.Fatal(err)
    }
}

The “flags” piks the command input. If “localhost” is set as a parameter, the program will run as a local web server and not in FCGI mode (see in your browser):

fcgitest -local=":8000"

The “init” function sets the number of processors to use at the same number of processors available by the machine.

The “homeView” function is a view that prints an HTML output to the user.

The main function sets up a URL router adding the “homeView” function at the root “/” path using the Gorilla handler.
The “if” part chooses if it runs as FCGI or as web server.

Pay attention at the “err = fcgi.Serve(nil, r)” line because it’s not documented well in the Go’s documentation: use “nil” for the standard input/output mode.

Run in TCP or Unix mode

Add two more flags for the TCP and UNIX socket modes:

var (
    local = flag.String("local", "", "serve as webserver, example: 0.0.0.0:8000")
    tcp   = flag.String("tcp", "", "serve as FCGI via TCP, example: 0.0.0.0:8000")
    unix  = flag.String("unix", "", "serve as FCGI via UNIX socket, example: /tmp/myprogram.sock")
)

…and change the “if” part like the following:

if *local != "" { // Run as a local web server
    err = http.ListenAndServe(*local, r)
} else if *tcp != "" { // Run as FCGI via TCP
    listener, err := net.Listen("tcp", *tcp)
    if err != nil {
        log.Fatal(err)
    }
    defer listener.Close()

    err = fcgi.Serve(listener, r)
} else if *unix != "" { // Run as FCGI via UNIX socket
    listener, err := net.Listen("unix", *unix)
    if err != nil {
        log.Fatal(err)
    }
    defer listener.Close()

    err = fcgi.Serve(listener, r)
} else { // Run as FCGI via standard I/O
    err = fcgi.Serve(nil, r)
}
if err != nil {
    log.Fatal(err)
}

The same code can be written with a “switch” statement.

In this case we are creating a TCP or a UNIX listener so it have to be closed at the end of the execution.

Deployment under Apache

This is an example of deployment under Debian/Ubuntu using the standard I/O mode, it should be compatible with Dreamhost and is a fast way to test it.

First install mod_fcgid, enable it and reload Apache. For example, this is under Debian/Ununtu:

sudo apt-get install libapache2-mod-fcgid
sudo a2enmod fcgid
sudo service apache2 reload

Compile, rename the file and deploy it on the web server:

go build fcgitest
mv fcgitest fcgitest.fcgi

Change the file access permissions removing the write flag from “group” and “others” or Apache will raise a 118 error:

chmod go-w fcgitest.fcgi

Add the following “.htaccess” file, it will redirect all the requests that are not a file or a directory to the Go program through the FastCGI protocol via the standard I/O mode:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d [OR]
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^(.*)$ fcgitest.fcgi/$1 [QSA,L]

Done, try it visiting the web site.

Conclusions

Personally I like to use this protocol via standard I/O because I run several web sites on my servers, like a hosting company; this allows me to keep the things simple and with no effort with a low RAM usage.

Download the full Go source code.

Post to Hacker News

Precise truncate words by chars

It truncates the text when it exceeds a certain number of characters and deletes the last word. Adds ‘…’ at the end of the text, only if truncated.

The difference between other similar filters is that this deletes the last word only if partial, so it is more precise and pleasant.

The snippet is here.

Divide a list into exact columns

It divides a list into an exact number of columns.

The difference between other similar filters it that the number of columns is guaranteed, even if there are not enough elements to fill all the columns.

The snippet is here.

I wrote this modifier for Smarty, a template engine for PHP that I am using for a long time.

The plugin can be downloaded from here.

The license is the New BSD.

The usage is very simple:
{$php_code|highlight}

So, an url like this:

http://www.foobar.it/index.php?page=articles

…could be converted like this:

http://www.foobar.it/articles/

This is very good for SEO and just for the simplicity of the url.

In order to activate this, is simple to add this code in the “.htaccess” file in the root directory of the web site:

RewriteEngine On

RewriteCond %{REQUEST_FILENAME} !-fRewriteRule ^(.*)$ index.php?page=$1 [QSA,L]

The first line activate the mod_rewrite. The second one control if the url is not a real file on the web server, and the last one convert the url.

Actually it is not so complicated. But try to add a sub-directory protected by password and some problems will arise.

Make a directory “foo”, put a file under it like “index.html” in order to test the behavior and protect it with this “.htaccess” file:

AuthName "Restricted Area"AuthType Basicrequire valid-userAuthUserFile "/home/foobar/passwd"

This code will protect that directory with a password (contained in the “passwd” file in another directory of security reasons).

Try to access to it and you will discover that it is impossible to use. The problem is the mod_rewrite because it rewrite also the url of this directory and it can’t see the index.html file because it is hidden by the password. Without the password protection the file is well displayed…why??

In order to solve this problem you have to add two lines to your root .htaccess file:

RewriteEngine On

RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-d [OR]RewriteCond %{REQUEST_URI} ^/$RewriteRule ^(.*)$ index.php?page=$1 [QSA,L]

The first line added control if the url is a valid directory into the filesystem, it add a OR condition, and the second line force the void url (in this case is “http://www.foobar.it/) to convert itself with mod_rewrite. The home page will not be displayed without that second line.

Now it is possible to see the directory “foo” and access to the index.html but…only if you are just logged in. If you are not logged in you will be redirect to a 401.html page redirected again to the index.php file (only a little debug system can show this to you); so the login form is not displayed, and this is a big problem.

In order to force the login form, it is sufficient to add this last line in the “foo/.htaccess” file:

AuthName "Restricted Area"AuthType Basicrequire valid-userAuthUserFile "/home/foobar/passwd"ErrorDocument 401 default

Now the login form is forced and you can use it without problems.

The project page is:
https://www.hosted-projects.com/trac/Design/pub/wiki/Nirvanapi

It include the most important API calls, is light (one file and less of 400 lines of code) and have not dependences.

The client support right now the following API calls:

LoginLogoutCreateFoldersDeleteFoldersDeleteFilesCopyFilesCopyFoldersMoveFilesMoveFoldersRenameFileRenameFolderSideuploadListFolderCreateHostedItemExtractFrames

The login is automatic also when the connection drop.

List a directory into Nirvanix in a little example:

from nirvanapi import NirvanixClientnc = NirvanixClient('XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'username', 'password')print nc.list_folder('/')

For dates I’m sticking with Paul Allen‘s recommendation: appointments and milestones goes in your agenda, the things to do that starting at a certain date goes in your tickler file or in your electronic agenda.

For the items, in order to remember certain dates, I use this syntax:
[since>update|completed]

An example: I’m waiting a CD-ROM from Paolo since 12 October, I will stress Paolo if I have not the CD for January.

- @Wait [12/10/2007>10/01/2008] Paolo's CD-ROM.

The “since” date remember you since you wait Paolo or since the item was added.
The “update” date is only for remember you when the item have to be updated (so contact Paolo and remember him that you are waiting the CD!).
The “completed” date is only for completed items: it remember you when the item was done.

Write dates is very useful just for waiting contexts.

If there isn’t a “update” date, but only a “since” date:

- @Wait [12/10/2007] Paolo's CD-ROM.

If there is only the deadline (see the caracter “>”):

- @Wait [>10/01/2008] Paolo's CD-ROM.

The completed date is optional for all the type of items, useful only if you like to know when you complete the item. It must be used only for completed items:

x @Wait [|9/01/2008] Paolo's CD-ROM.

I prefer to use day/month/year, but you can chose your preferred order. A suggestion in order to increase the readability: use always 2 digits for days and months and 4 for years.

Put dates only when is really useful for you, because you have to spend time for your job and not for your organization system. In my GTD text file, only 2% of items have a date.

Conclusion

With this model of text file it’s easy to write some tool or some syntax file for your preferred text editor.

I use Kate for GTD (I will write about this) and I written a tool, just few lines in Python, for sync my Palm with a short selection of contexts (if there are some requests I can publish it).

Now you have a complete system for GTD fast to write, easy to read, really portable and simple to expand with few line scripts. Good job!

In the first part you saw how to write todo items and check them. In this part I talk about projects.

A short description: for GTD, a project is a work to do which require more than one physical action to achieve. I added several other things to it (thanks Anthony Robbins!).

In order to write efficient and attractive plan, the project should have a purpose, some goals and some notes. I built my system over this. See an example:

== WRITE A GTD BOOK ==# Purpose:Help people that prefer read a paper book. Earn several euros.# Principles:Slim (less than 100 pages), very simple to read, sell at minimum 1,000 copies.# Displaying the result:People write me how the book improve their lives. The book sell more than 2,000 copies.# Brainstorming:Syntax, Kate, call the publisher, set the price (10€ or 9.90€?), syntax of dates.

The title is between “== TITLE ==” and is all uppercase.

Some suggestions in order to increase the readability and waste less time:

• For the “purpose” is preferable to write only one line.
• For the “principles” is preferable to write only one line or a short realistic list.
• For the “Displaying the result” write a short good scenario, not a dream one! Write less lines as you can, one if possible.
• For the “Brainstorming” write all on a one line comma separated, if possible.

So you should to write less as you can.

Add your todo items after a blank line:

== WRITE MY GTD BOOK ==# Purpose:Help people that prefer read a paper book. Earn several euros.# Principles:Slim (less than 100 pages), very simple to read, sell at minimum 1,000 copies.# Displaying the result:People write me how the book improve their lives. The book sell more than 2,000 copies.# Brainstorming:Syntax, Kate, call the publisher, set the price (10€ or 9.90€?), syntax of dates.

- @Tel Irene for a review of the draft.- @Write the second part.- @Email a draft of the cover to the designer.

Write only the todo items that you can do in the same time. So, if you have a things to do after write the first part of the book, you should: add to the brainstorming list, or add “Next write the second part.” to the end of line. See the example:

- @Write the second part. Next write the third part.

Add your project in the same GTD.txt file after the todo items without project and separate it by two blank lines. A fast example:

@Buy @Home @Office @Tel @Web

- @Buy 1l milk, 2 lemons, chocolate.- @Buy An iPod.- @Buy 2 new Moleskine (one slim and one big) and a flicker.

- @Home Repair the chair.

- @Tel Elisa for the appointment of Monday 13 November.- @Tel Marco in order to set a meeting with Company S.p.A.

- @Web Renew the Backpack login.- @Web Make a review of www.ontiles.com

== WRITE MY GTD BOOK ==# Purpose:Help people that prefer read a paper book. Earn several euros.# Principles:Slim (less than 100 pages), very simple to read, sell at minimum 1,000 copies.# Displaying the result:People write me how the book improve their lives. The book sell more than 2,000 copies.# Brainstorming:Syntax, Kate, search a publisher, set the price (10€ or 9.90€?), syntax of dates.

- @Tel Irene for a review of the draft. Next send an email to Elena with some Irene's opinions.- @Write the second part.- @Email a draft of the cover to the designer.

== REPAIR THE CAR ==

[...]

Remove the project when you have done it, use an old_projects.txt file if you like.

With this system you can store a large number of projects and items to do in an easy to read (and very fast to write) file. If you use Vim or Emacs for this you can really be a lot faster than any program or web application can do!

In the next part I will talk about dates.

Before that I read a lot of organization books, tips and articles on internet. When I discovered GTD of David Allen I changed my mind and now I see the results, the best results.

I’m a fan of simplicity and I love text files because I can read it whit all devices, open it in an instant and build fast scripts to automate some operations.

I tried a lot of programs, both off and on line (web applications), like Remeber The Milk, Nozbe, Vitalist, pyGTD, Backpack (my preferred) and others.

The 20% of web applications are really good, like Backpack or Vitalist, but you need a connection to the internet if you want to work.
The desktop applications are slow to open, not multiplatform and needs installations on all your devices: there are not an application that run in Linux, Windows, MacOS and on my Palm or cellular phone.
All text systems that I tried are complex, too complex for the real life: I want to write a todo item in a couple of seconds, not in minutes.

So I created my text system: simplest to remember, faster to write and easiest to read.

Well, if you just know GTD take a file called GTD.txt. If you hear GTD for the first time, give it a try: you will discover how is easy to organize your life and your million of things to do!

So, create a simple text file with your preferred program. Avoid MicrosoftWord or some other heavy program, use the Notepad or Edit Plus or Vim or whatever else: you have to create a .TXT file.

Put all your contexts in your first line, see the example:

@Buy @Home @Office @Tel @Web

This line is for reference. You can use only one word for each context, I suggest you to capitalize the first letter.
The context is a tag that indicate in what place do the thing, or when do it. So, if you are in a store you have to read all the @Buy items.

Each item must have a context and only one one.

Make a blank line then write your @Tel items, then make two blank lines (this increase the readability when you have a lot of things to do) and write your @Web items, ecc…

See the example:

@Buy @Home @Office @Tel @Web

- @Buy 1l milk, 2 lemons, chocolate.- @Buy An iPod.- @Buy 2 new Moleskine (one slim and one big) and a flicker.

- @Home Repair the chair.

- @Tel Elisa for the appointment of Monday 13 November.- @Tel Marco in order to set a meeting with Company S.p.A.

- @Web Renew the Backpack login.- @Web Make a review of www.ontiles.com

Every item start with a line, a single space, only a single context and another space. Yes, only one context for item. If you have more contexts for each item you have not efficient contexts, so reorder them.

Use only one line for each item; this is very important because it is easiest to use and it obligates you to write good items: the better todos are short!

When you do an item you can simply eliminate it or put an “x” before:

- @Buy 1l milk, 2 lemons, chocolate.x @Buy an iPod.- @Buy 2 new Moleskine (one slim and one big) and a flicker.

If you like to remember old things done, copy it in another file (history.txt). In my honest opinion using an history file is not useful and distract you: the mind have to look at the present, not at the past.

In the next part I will tell you about projects.

The new features will be:
- multi list management;
- all objects of the pages can now be ordered;
- the old page descriptions are now notes;
- page links have been deprecated;
- search function;
- separators;
- the tags bug could be now resolved.

Some problems still seem not to have benn taken in consideration, like the export of filed and images. Together with others developers we are making pressure in order to have the possibility to manage them through API, but 37signals still does not answer to our demands.

Da poche ore è stato aggiornato Backpack ed è disponibile la nuova API che corregge diverse problematiche che lo affliggevano.
Il client per Python che ho preparato sarà presto aggiornato e pubblicato.

Le nuove funzionalità saranno:
- gestione di più liste;
- posizionamento degli oggetti nella pagina;
- la descrizione delle pagine è ora diventata una nota come le altre;
- i link tra le pagine sono ora deprecati;
- funzione di ricerca;
- separatori;
- il bug dei tag ora dovrebbe essere risolto.

Alcuni problemi sembrano non essere ancora stati presi in considerazione, come l’esportazione di immagini e file. Assieme ad altri sviluppatori stiamo facendo pressione per avere la possibilità di gestirli tramite API ma 37signals ancora non risponde alle nostre domande.