Xt

Xt is a rapid XML templating language. It allows you to create dynamically generated XML/XHTML content from data stored in a database. All Xt documents are also valid XML documents and thus may be authored using and processed by standard XML/XHTML tools. It is designed with simplicity and data integrity in mind.

Language

The Xt language is embedded in XML as XML processing instructions (PIs). The language is fairly simple, owing to its narrow focus. The primary construct is the query block. In its basic form, a query block performs a query on the backing database, and repeats a block of XML once for every result returned. The repeated block may contain variables which are substitued for the values returned by the query.

For example, this query block:

<?xt query SELECT name, type FROM pokemon?>
<?xt text name?> is of type <?xt text type?>. <br />
<?xt end?>

will produce this output:

Charizard is of type fire. <br />
Blastoise is of type water. <br />
Bulbasaur is of type plant. <br />

assuming that the pokemon database contains those rows.

Xt automatically escapes special characters to ensure proper XML formatting.

Xt currently supports ASCII, UTF-8, and UTF-16 encodings for input, and outputs as UTF-8 (or ASCII if only that subset is used). Each backend ensures to the best of its ability that the results are encoded as UTF-8, but for some backends this is not possible (e.g. encoding info is not present). It is up to the user to ensure that data is encoded as UTF-8 when using these backends.

Xt program output is “streamed”: content is sent to the client as soon as it is generated and is not stored in memory. That this is possible is a guarantee of the language (not just the implementation), and ensures that Xt can handle large result sets without overloading your server.

Syntax

Query blocks

The general form of a query block is as follows:

<?xt query query?>
Passes query to the database backend. query is interpreted in a backend-dependent manner.
prefix
<?xt pre?>
Optional. If the query returns at least one result, then prefix is emitted exactly once. No variable substitution is performed in prefix.
content
content is repeated once, with variables substituted, for every result returned by query.
<?xt sep?>
separator
Optional. separator is repeated between each repetition of content. No variable substitution is performed in separator.
<?xt post?>
postfix
Optional. If the query returns at least one result, then postfix is emitted exactly once. No variable substitution is performed in postfix.
<?xt else?>
default
Optional. If the query returns no results, then default is emitted. Variable substitution is obviously not performed.
<?xt end?>
Marks the end of the query block. Variable substitutions are not available outside of the block.

Note that prefix and postfix need not be valid XML snippets on their own; rather, only their concatenation need be valid. For example, prefix may be <div> and postfix may be </div>.

Queries may be nested. In the case of overlapping variable names, those of the innermost query have priority.

There is also a conditional expression, if:

<?xt if expr?>
Passes expr to the database backend. expr is interpreted as a Boolean expression in a backend-dependent manner.
content
content is emitted exactly once if and only if expr evaluates to true.
<?xt else?>
default
Optional. If the expression evaluates to false, then default is emitted.
<?xt end?>
Marks the end of the if block.

Note that if is merely a shorthand for a query with no data source and no bindings. e.g. in the SQL backend, it is fully equivalent to (and in fact is implemented as) the query SELECT NULL WHERE expr; or in the QLC backend, [{} || (expr)].

Variable substitution

Two PIs exist to access variables from results:

<?xt text name?>
Replaced with the value of name from the current result. XML-escaping is performed as necessary.
<?xt xml name?>
Replaced with the value of name from the current result. XML-escaping is not performed. The value must, however, be valid a XML snippet, or an error is raised.

Inline references

Xt also supports inline variable references, similar to XML entities. Inline references may be used in attribute values as well as in text:

$name;
Replaced with the value of name from the current result. XML-escaping is performed as necessary.
$dol;
Replaced with a literal dollar sign ($).
$:name;
Same as $name;, but $:dol; has no special meaning.
$uri:name;
Acts as $:name;, but escapes the (Unicode) value as a (UTF-8) URI (component) before escaping as XML. Equivalent to $percent:UTF-8:name;.
$encoding:name;
Encodes binary value using encoding. E.g. data:image/png;base64,$base64=:thumbnail; or magnet:?xt=urn:sha1:$base32:sha1;.
Currently accepted encodings are hex, base32, base32=, base64, base64=, and percent.
$encoding:charset:name;
Encodes text value using charset followed by encoding. E.g. data:text/plain;charset=UTF-8;base64,$base64=:UTF-8:snippet;.
Currently accepted charsets are US-ASCII, ISO-8859-1, and UTF-8.
$if:name;
Evaluates to the empty string. Inside an attribute, causes the attribute to be emitted if and only if name has a true value (as defined by the backend).

Variables may also be accessed within queries, but the syntax is backend-dependent.

Recursive templates

Templates may reference themselves in a recursive fashion using the following PIs:

<?xt named-query name query?>
Behaves as <?xt query query?>. name may be referenced by one or more enclosed rec instructions.
<?xt rec name?>
Replaced verbatim with the source text of the innermost enclosing named-query block named name (including the named-query, rec, and end instructions). The replaced block is then evaluated as usual.

Warning: recursive templates present the opportunity to create an infinite loop which will continually allocate memory. To prevent this, you must ensure that the template’s query references at least one variable which is rebound within the query template, and that successive queries eventually result in one with no results (or one with results, if the rec instruction appears in the else clause).

Processing instructions

<?xt pi args…?>
Encodes a backend-specific processing instruction, such as file inclusion, or backend configuration.

Backends

xt_qlc (xt-erl only)

xt_qlc is the QLC/Mnesia backend. Queries are QLC queries (list comprehensions) whose left-hand side is either a variable, a constant, or a tuple or unification thereof. Queries may reference variables from enclosing queries using the function get/1, with the variable name represented as an atom (e.g., get('SomeVar')). Frontend-defined tables may be accessed with the table/1 function.

(under construction…)

XtODBC (xt-erl only)

(under construction…)

XtPg (xt-ocaml only)

XtPg is the PostgreSQL backend. Queries are PostgreSQL queries. Queries may reference variables from enclosing queries using PostgreSQL parameter syntax, that is, a dollar sign ($) followed by the variable name. Proper escaping will be performed. For example:

<dl>
<?xt query SELECT musician FROM musicians?>
  <dt>$musician;</dt>
  <?xt query SELECT instrument FROM musicians WHERE name = $musician?>
    <dd>$instrument;</dd>
  <?xt end?>
<?xt end?>
</dl>

This will result in the following output:

<dl>
  <dt>John Linnell</dt>
    <dd>Accordion</dd>
    <dd>Organ</dd>
  <dt>Sufjan Stevens</dt>
    <dd>Banjo</dd>
    <dd>Guitar</dd>
    <dd>Trumpet</dd>
    <dd>Vibraphone</dd>
</dl>

XtPg connects to the default PostgreSQL server and database. This may be influenced by using the standard PostgreSQL environment variables. For example, in Apache, make sure mod_env is enabled and add the following lines before your XtPg configuration:

SetEnv PGHOST host
SetEnv PGDATABASE database
SetEnv PGUSER user

This will instruct XtPg to connect to database on host as user.

XtPg initially creates three temporary tables holding CGI environment information, query, index, and path_info:

CREATE TEMP TABLE query (name varchar NOT NULL, value varchar NOT NULL);
CREATE TEMP TABLE index (index int PRIMARY KEY, value varchar);
CREATE TEMP TABLE path_info (index int PRIMARY KEY, path varchar NOT NULL);

query contains the CGI query name-value pairs. index contains the 0-indexed components of an “isindex” query. path_info contains the 0-indexed path components following the name of the Xt file. (If the Xt file is being used as a directory index, then path_info will contain exactly one row, (0, '.').)

XtPg ensures that string values are properly transcoded to UTF-8 by using SET NAMES 'UTF8'.

All queries in the Xt file are run in a single READ ONLY ISOLATION LEVEL SERIALIZABLE transaction. If the query aborts for any reason, the error message will be inserted into the XML stream and processing aborted. This may change in the future to simply write errors to a log file.

Apache configuration (xt-ocaml only)

The various Xt backends may be run in one of two modes with Apache, CGI mode or HTTP mode. CGI mode is easier to set up but slower; HTTP mode typically requires admin priviledges but is more efficient.

The following sections are written assuming XtPg for clarity, but they are equally valid for the other backends as well.

CGI mode

For CGI mode, I find it works best to name your XtPg-enabled files with a .xtpg.html extension (or .xtpg.xml, etc.). This plays well with MultiViews (which I strongly recommend enabling), and allows XtPg to correctly guess the MIME type. The following instructions are written with assuming this naming convention.

  1. Place the xtPgCGI binary in a location where it can be run as a CGI script; e.g. /usr/lib/cgi-bin/xtPgCGI.
  2. Use a2enmod to enable mod_actions, mod_cgi, and mod_mime, if they are not enabled already.
  3. Add the following directives to your configuration, either site-wide or in an .htaccess file:
    Action x-xtpg /cgi-bin/xtPgCGI
    AddHandler x-xtpg .xtpg
    (Of course /cgi-bin/xtPgCGI will also change if you install the binary somewhere other than /usr/lib/cgi-bin/xtPgCGI.)
  4. (optional) Add MultiviewsMatch Handlers to your configuration. This allows Xt to integrate with Multiviews if you have it enabled.
  5. Restart Apache.

Of course you may need to tweak this example to match your local configuration. See the Apache docs on the above directives for more info.

HTTP mode

HTTP mode does not require the same naming convention as CGI mode, as it treats all files as Xt-enabled. XtPg will still use the extension as a means of guessing MIME types, and will automatically append extensions, similarly to Apache's MultiViews. However, all files served by XtPg must reside in a single directory tree.

  1. In the toplevel directory of your XtPg document tree, run xtPgHTTP. This will run the server on localhost port 8080. Run xtPgHTTP --help for more options.
  2. Use a2enmod to enable mod_proxy and mod_proxy_http, if they are not enabled already.
  3. Add the following directive to your site-wide configuration, where /web/site/path is the path you want your document tree to be visible from:
    ProxyPass /web/site/path http://localhost:8080
  4. Restart Apache.

Of course you may need to tweak this example to match your local configuration. See the Apache docs on the above directives for more info.

Download

NOTE: I am in the process of reimplementing Xt in Erlang. It is mostly complete but not heavily tested. You can obtain the code via darcs get http://hub.darcs.net/squirrel/xt-erl. The original OCaml implementation at (obtainable via darcs get http://hub.darcs.net/squirrel/xt-ocaml) is no longer maintained. OCaml is not a good server language.

Contact

with bugs, questions, etc.