TextMate, Emacs and META indent-region

[cross-posted from the Desert Moon blog.]

I haven't used GNU Emacs very much since switching to TextMate in 2005. One Emacs feature which I really miss in TextMate is indent-region. It lets you take an entire region of code, whatever its language, whatever its mix of tabs and spaces and indentation widths, and re-format it using your preferred indentation style.

But wait! Emacs has a batch mode, and you can drive it from TextMate. Many thanks to Gragusa's Things for showing the way.

The post on Gragusa's Things is specific to R code, but I'm more interested in re-formatting C and C++ code. Here's my first cut at a general TextMate Bundle to re-format code regardless of the source language:

#!/usr/local/bin/python2.6
"""
Use Emacs to re-indent regions of the current buffer.
Inspired by 
http://gragusa.wordpress.com/2007/11/11/textmate-emacs-like-indentation-for-r-files/
"""
import tempfile
import os
import sys
import subprocess

# Use the same filename extension so Emacs will know which
# mode to use. 
ext = os.path.splitext(os.environ["TM_FILEPATH"])[-1]
outf = tempfile.NamedTemporaryFile(suffix=ext, delete=False)
pathname = outf.name

outf.write(os.environ["TM_SELECTED_TEXT"])
outf.close()

args = [
    "emacs", "-batch", pathname, 
    # Assume no emacs-startup.el
    "--eval", "(setq indent-tabs-mode nil)",
    "--eval", '(c-set-style "java")',
    "--eval", "(setq c-basic-offset 4)",
    "--eval", "(indent-region (point-min) (point-max) nil)", 
    "-f", "save-buffer"]
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode:
    print(err)

inf = open(pathname, "r")
sys.stdout.write(inf.read())
inf.close()

os.remove(pathname)

NB:

1. Due to the use of the delete=False keyword argument to tempfile.NamedTemporaryFile, this command bundle requires Python 2.6+.
2. TextMate on OS X 10.5 won't, by default, have /usr/local/bin in its path; hence the pathetic shebang.

Anyway, install this as a new TextMate command bundle, assign a Key Equivalent such as ⌘-Shift-R, and enjoy.

Creating an 'hg ignore' extension

[cross-posted from the Desert Moon blog.]

I often wish Mercurial had an 'hg ignore' command similar to 'bzr ignore'. Turns out it's pretty easy to add one:

#!/usr/bin/env python
"""Ignore pathnames and patterns"""

import os

def ignore(ui, repo, *pathnames):
    """Ignore the given pathnames and patterns."""
    outf = open(os.path.join(repo.root, ".hgignore"), "a")
    for p in pathnames:
        outf.write(p + "\n")
    outf.close()
    return

cmdtable = {
    'ignore': (ignore, [], "hg ignore pathname [pathname]"),
}

To use this, save it to a file such as ${HOME}/platform/independent/lib/hg/ignore.py. Then add the extension to your ${HOME}/.hgrc:

[extensions]
~/platform/independent/lib/hg/ignore.py

Running TileCache within a Django Application

Punchline

Here is how to serve TileCache tile images from within a Django application.

from TileCache.Service import Service

_service = Service(...)

def get_tile(request):
    global _service

    format, image = _service.dispatchRequest(
        request.GET, request.path, request.method, 
        request.get_host())
    result = HttpResponse(str(image), mimetype=format)
    return result

Scenario

You're building a low-traffic Django-based GIS application, and you need to serve your own map layers. You're using TileCache to improve your application's performance. But installation and configuration are hassles.

• All of your servers must run with the right user and group IDs, so the Django app can expire the tile cache when necessary.
• Your Django app needs to understand the structure of the tile cache, so it can remove the correct tile images when the underlying data changes.
• Etc.

This would all be much easier if you could serve TileCache requests from within your Django application. They're both Python-based; why not?

The TileCache code base includes sample code that shows how to run TileCache as a CGI or a FastCGI service. I couldn't find any sample code for running TileCache within a Django application, but it was easy to convert the cgiHandler code for use with Django's HttpRequest objects.

Installation Prerequisites

In order for TileCache to generate its own tiles, instead of delegating to a separate mapserver instance, you must already have compiled and installed mapserver's Python mapscript bindings. For instructions on compiling the bindings see the mapscript/python/README file in the mapserver source distribution.

Configuring TileCache

import os
thisdir = os.path.abspath(os.path.dirname(__file__))
def relpath(p):
    return os.path.abspath(os.path.join(thisdir, p))

from TileCache.Service import Service
import TileCache.Layers.MapServer as MS

# Create the service 'singleton'.
_mapfile = relpath("../mapserv/data/mapfile.map")

_service = Service(
  _cache,  # See "Cache Invalidation", below
  { 
    "basic": MS.MapServer(
        "basic", _mapfile, layers="basic", debug=False),
  }
) 

Handling Tile Requests

This is the sweet part. It's derived from the cgiHandler() example in the TileCache source code, but Django's HttpRequest class makes the implementation very simple:

def get_tile(request):
    global _service

    format, image = _service.dispatchRequest(
        request.GET, request.path, request.method, 
        request.get_host())
    result = HttpResponse(str(image), mimetype=format)
    return result

What About Feature Info Requests?

I don't know much about the required web API of a WMS server, but it appears as if the same URL must serve both tiles and feature info requests; the type of request is determined by the Request querystring parameter.

Django's dispatch system is based on URL pathnames; I'm not aware of any way to dispatch based on query string parameters. So you'll need to either configure your web server (e.g. Apache) to rewrite WMS requests to distinct URLs provided by your Django app, or you'll need to do some dispatch within your Django app.

Suppose you opt for the latter. Then your urls.py might look something like this:

    ...
    url(r'^wms/$', 'world.views.wms', name='wms'),
    ...

and in world/views.py you might have this:

def wms(request):
    if request.GET.get("request") == "GetFeatureInfo":
        return get_feature_info(request)
    return get_tile(request)

Cache Invalidation

For my web app, several of the tile layers are derived from a Django model which is updated via the admin interface. Whenever the model changes, the tile cache for the corresponding layer(s) needs to be invalidated, so the images can be regenerated.

The TileCache Cache interface doesn't provide for invalidation. Since I'm using a filesystem-based cache, I subclassed TileCache.Caches.Disk to create a Disk cache which does support invalidation.

import shutil
from TileCache.Caches.Disk import Disk

class InvalidatingDisk(Disk):
    """A Disk cache which can invalidate its contents, 
       layer by layer."""
    def invalidate(self, layerName=None):
        if self.basedir:
            pathname = self.basedir
            if layerName is not None:
                pathname = os.path.join(self.basedir, 
                                        layerName)
            shutil.rmtree(pathname, ignore_errors=True)

Introducing Google Chrome Frame

Introducing Google Chrome Frame:

"With Google Chrome Frame, developers can now take advantage of the latest open web technologies, even in Internet Explorer. From a faster Javascript engine, to support for current web technologies like HTML5's offline capabilities and <canvas>, to modern CSS/Layout handling, Google Chrome Frame enables these features within IE with no additional coding or testing for different browser versions.

To start using Google Chrome Frame, all developers need to do is to add a single tag:

<meta equiv="X-UA-Compatible" content="chrome=1">
"

I guess that's good news. Makes you wonder why the target audience wouldn't just install Google Chrome. But I suppose this lets people continue to use IE while using modern web facilites on sites which require them.

TR: China Wind Energy Potential, HVDC

(Just taking notes, trying to understand what HVDC is, what it has to do with variable power sources such as wind, and why it makes buried transmission lines convenient.)

Technology Review: China's Potent Wind Potential:

"The major grid upgrades already under way in China are making extensive use of continental-scale high-voltage direct-current (HVDC) lines, which remain the stuff of supergrid blueprints in Europe and the United States. 'They are leading the world in implementing long-distance transmission schemes,' says Bjarne Andersen, director of U.K.-based consultancy Andersen Power Electronic Solutions and an expert in the ultra-efficient HVDC technology."

Technology Review: Europe Backs Supergrids:

"This summer [2008], for example, a negotiator appointed by the EC convinced France to accept a new transmission connection with Spain, breaking a 15-year impasse over expanding power exchanges between the countries. Use of high-voltage DC (HVDC) technology will enable planners to bury the new line and thereby overcome local opposition to conventional overhead AC transmission lines."

Wikipedia explains how HVDC can have low power losses:

"Power in a circuit is proportional to the current, but the power lost as heat in the wires is proportional to the square of the current. However, power is also proportional to voltage, so for a given power level, higher voltage can be traded off for lower current. Thus, the higher the voltage, the lower the power loss."

"The advantage of HVDC is the ability to transmit large amounts of power over long distances with lower capital costs and with lower losses than AC."

Lots of interesting stuff in the Wikipedia article. This excerpt seems to explain the connection between HVDC and variable power sources:

"Because HVDC allows power transmission between unsynchronised AC distribution systems, it can help increase system stability, by preventing cascading failures from propagating from one part of a wider power transmission grid to another. Changes in load that would cause portions of an AC network to become unsynchronized and separate would not similarly affect a DC link, and the power flow through the DC link would tend to stabilize the AC network. The magnitude and direction of power flow through a DC link can be directly commanded, and changed as needed to support the AC networks at either end of the DC link. This has caused many power system operators to contemplate wider use of HVDC technology for its stability benefits alone."

ActionScript: TileList, deleting, scrolling backwards

I have a Flex app which shows users a TiledList of chemical structure depictions. Since it can be a large list, it's lazy-loaded from the server as the user scrolls through the list.

Users can delete items from the list, with undo. In order to do this with reasonable performance, once the app has received confirmation from the server that an item has been deleted, it clears out the single deleted item from its local lazy-list.

To undo the deletion locally, the Flex app fills the correct lazy-list entry with an ItemPendingError; that error gets thrown as soon as the TileList tries to retrieve the item.

All of this works okay when the row containing the undeleted item is already visible. On the other hand, if the user has scrolled away from the row where the undeleted item will reappear, then when (s)he scrolls back the TileList simply empties out that item and all of the successive items in the row. Ugly!

Workaround

When the item is undeleted, immediately try to retrieve it via getItemAt(itemIndex). Catch the resulting ItemPendingError and register an ItemResponder. When the ItemResponder's result or fault method is called, tell the TileList to invalidateList(). If the undeleted item actually contains a value, the TileList will repaint correctly -- no more unsightly gaps.

import mx.collections.errors.ItemPendingError;
import mx.collections.ItemResponder;
[...]
try {
    structures.getItemAt(offset);
} catch (e:ItemPendingError) {
    e.addResponder(
    new ItemResponder(
        function(result:Object, token:Object = null):void {
            tilelist.invalidateList();
        },
        function(error:Object, token:Object = null):void {
            tilelist.invalidateList();
        }));
}

I like Mike -- General Michael Collins, That Is

This year's John Glenn Lecture Series featured Sen. Glenn, Chris Kraft and the three Apollo 11 astronauts. Michael Collins was as smart, funny and humble as in "When We Left Earth." His talk starts roughly 55 minutes in.

Apologizing for the lecture-unfriendly layout of the IMAX theater, which he helped approve:

"I'm down here in the bottom of a black hole about to be sucked in by gravity..."

After putting up this picture, which he took as the LEM began its descent to the lunar surface:

"I like that photo, it's my favorite one. You see in the little thing there are 3 billion people, and then in the big thing there are two people..."

About the glistening blue earth in the background:

"Serene it is not. Fragile it is. The world population when we flew to the moon was 3 billion people. Today it's over six and headed for eight, so the experts say. In my view this growth is not wise, healthy or sustainable[...]

"Our economic models are all predicated on growth. They require it. Grow or die, or maybe both: the dead zone created by the runoff from the Mississippi into the Gulf of Mexico is now larger than the State of New Jersey, and still growing...

"We need a new economic paradigm that somehow can produce prosperity without this kind of growth."

The video: http://www.youtube.com/watch?v=w9fCPhspOCQ