Beginner's Mind

Moving to Tumblr

jcrites@jcrites.com (Justin Crites) — Sun, 28 Jun 2009 17:34:00 +0000

I've moved my blog to Tumblr. Check it out at http://blog.jcrites.com

Cross-applicable

jcrites@jcrites.com (Justin Crites) — Sun, 21 Dec 2008 14:34:00 +0000

Vigorous writing is concise. A sentence should contain no unnecessary words, a paragraph no unnecessary sentences, for the same reason that a drawing should have no unnecessary lines and a machine no unnecessary parts. This requires not that the writer make all sentences short or avoid all detail and treat subjects only in outline, but that every word tell.

From "The Elements of Style" by William Strunk Jr

Semantic Patches

jcrites@jcrites.com (Justin Crites) — Tue, 09 Dec 2008 00:37:00 +0000

One of the goals of my programming language Ix is to support the evolution of programs in a first-class way. I haven't figured out all of the details, but the idea is for the language to know about version 1 and version 2 of the program, and to provide warnings when certain kinds of dangerous transformations occur.

Having a static type system also provides similar functionality without any explicit notions of transformations through time; if you change a function to remove a parameter, then client code which needs to be changed will fail to compile. Thus the static type system provides implicit support for verifying program changes.

Could a language support something more powerful? Some others were thinking along these lines and developed Semantic Patches:

Our goal is to document and automate the kinds of collateral evolutions that occur in code. Because programmers are accustomed to manipulating program modifications in terms of patch files, we base our transformation language on the patch syntax, extending patches to semantic patches. But as opposed to a traditional patch, a single small semantic patch can modify hundreds of files, at thousands of code sites. This is because the features of our semantic patch language (SmPL) make a semantic patch generic by abstracting away the specific details and variations at each code site among all drivers. Semantic patches, and the associated transformation engine spatch, abstract away:

Differences in spacing, indentation, and comments

Choice of names given to variables (use of metavariables)

Irrelevant code (use of '...' operator)

Other variations in coding style (use of isomorphisms) e.g. if(!y) <=> if(y==NULL) <=> if(NULL==y)

A natural extension of this technology is to perform validation along with the transformation.

An Introduction to Ix

jcrites@jcrites.com (Justin Crites) — Sat, 01 Nov 2008 01:05:00 +0000

For years, I have had a dream. This dream has been a waking dream, a fantasy that preoccupies my waking thoughts. It is a mundane dream, in a sense: I dream of my ideal tool, the tool I use to practice my trade. Mundane in the sense that a carpenter dreaming of the ideal hammer is mundane.

The tool I use is more abstract. It’s the tool man uses to control machines, a programming language. Unlike natural languages, such as English or Spanish, programming languages are invented by men deliberately to serve their purposes. There are many of these languages, thousands. Why so many?

I started thinking about developing my own language in my first college programming languages course. “Wow, this is easy,” I thought after implementing my first interpreter. It’s not easy to make it run fast, but it’s relatively easy to make it exist.

My design of Ix has been influenced by the best of all the programming languages I’ve used. In a sense I’ve been most impressed by Lisp, where code is data, and programs can rewrite themselves. However, in order to achieve the code is data objective, Lisp had to give up a lot of readability and conciseness of which other languages are capable.

I felt that Lisp came very close to hitting the mark, the mark for which I myself am aiming (“the perfect programming language”), but ultimately missed. One of its failings is its choice of primitive data structure, the cons cell or the pair. During the development of Ix, I’ve made it a priority to select the ideal fundamental structure.

The language’s fundamental construct should be one with which we can build any data structure known to computer science. The cons cell fails at this. It can’t, for example, be used to implement arrays. Although Lisp’s syntax was designed so that “there is only one sort of stuff” (the cons cell), and all syntax is expressed with it, I think it’s the wrong primitive construct with which to approach computer science, and the wrong fundamental construct even for language syntax. You can see it break down when Lisp needs to represent arrays or associative structures. (‘a 1 ‘b 2) is not a clear way to represent that ‘a maps to 1 and ‘b to 2.

Having decided the ideal primitive construct can’t be cons, I set out to find it. Since in the language I am building data is code, that also means that the primitive with which I represent language syntax will also be the primitive with which to build data structures. The first structure I considered was the array. Indeed, an array of length 2 is a cons, so surely arrays are at least as good a choice.

I came very close to deciding upon the array as the basic structure, but what held me back was the feeling that the language should be able to support named parameters. Yes, that’s all. I hacked together a clunky implementation where the basic syntactical unit had both array- and map-like properties before I finally put things together after reading Steve Yegge’s blog post called The Universal Design Pattern. I don’t necessarily think he was meaning to convey the idea I took away from it. Steve intended to convey that (property) maps are a useful data structure with which to build programs; what I ended up realizing is that they are actually the ideal fundamental data structure for programming language syntax, and are the most fundamental structure in computer science.

To make this connection it helps to think of data structures in an abstract mathematical sense. Consider arrays and linked lists. What are the most basic properties of these structures?

I would argue that fundamentally, these data structures are associations, specifically functions. They associate position (index) to an element value. A linked list with a section of the Fibonacci sequence {5, 8, 13, 21, 34} is a function mapping the indexes {1, 2, 3, 4, 5} to the values, so:

Index	Value
1	5
2	8
3	13
4	21
5	34

An array with the same contents represents the same association. Looking at the situation with the greatest degree of abstraction, arrays and linked lists are two implementations of a function; being a function is their first property. Only when we move down a rung on the abstraction ladder does the performance of the two structures come into play, such as the lookup or insertion time.

Lisp-style functional programming has taken a rather different approach on the situation, looking at the cons cell as a general recursive type, which it is. But without the language having a generalized concept of collections, it’s hard to get there.

So, Ix is Lisp-like but starts with the premise that the basic unit of syntax is the map. The key is taking the realizations from earlier, and noting that elements in a map don’t have to be “named” in any sense. The key might just be an integer which is the element’s position.

Ix’s syntax starts out like Lisp. (f a b) represents calling the function f with arguments a and b. In many contexts, like at the beginning of a statement, a function call is implied, and so the parentheses may be omitted. Assuming I can properly avoid ambiguity, C-style function call notation may be used: f(a b).

For clarity, the symbols “;” or “,” can be used to delimit arguments, so our recurring example could be written (among other ways):

· f(a,b)

· f(a;b)

· (f a; b)

Function call syntax is a map. A named argument can be specified with the maps-to operator, which is the symbol “:”. Example: open-socket(host : “example.com”; port : 22).

Aside from the function call syntax “( )”, Ix has two other primitive notations: (explicit) maps, and blocks.

Collections

Maps, primarily known as collections, are created by placing an expression inside “[ ]”. Elements in the maps may be identified by name (explicitly) or by position (implicitly). For example, a map associating the symbol A with 1 and B with 2 can be written explicitly with the aforementioned maps-to operator:

[ a : 1; b : 2]

Elements which are not explicitly assigned a name are implicitly assigned a position. For example, the portion of the Fibonacci sequence mentioned earlier can be written:

· [ 5 8 13 21 34 ]

· [ 5, 8, 13, 21, 34 ]

· [ 5; 8; 13; 21; 34 ]

Ix presents this variety of syntax meaning the same thing to allow the expression of connotations – generally, for the programmer to use the syntax that “looks right”.

The syntax of function calls and the syntax of collections (maps) is thus the same; the only difference is that function calls indicate function-call evaluation, while the collection syntax indicates the creation of a collection.

Blocks

The remaining type of “object group” notation is the block, which uses “{ }”. Block syntax indicates the creation of an anonymous function; the contents of a block is a collection of expressions. For example, (+ a b) represents the sum of a and b (evaluated presently), while {+ a b} represents an anonymous function taking parameters a, b and returning their sum. When a block is evaluated, its constituent expressions are evaluated sequentially.

By default, parameters are not positional, so one would have to call such a function with named parameters. For example:

({+ a b} a : 1, b : 2)

The final primitive piece of Ix notation is the function binding operator, written “=>”. The block notation creates an anonymous function, and any contained variables are free variables. In order to evaluate the function, the variables must be assigned values. This can be done by calling the function with named arguments, or can be done by using the binding operator, which assigns function arguments positions. For example:

[a b] => {+ a b}

The result of the previous example expression is a function with two positional arguments.

Macros

So far, we’ve seen how to call functions, how to create collections of values, and how to create anonymous functions. We haven’t yet seen how to create variables or anything else.

A design goal of Ix is for the syntax to be unified, following the Lisp “code is data” mantra. In Ix, code is indeed data, and the creation of variables and other standard constructs is through special function calls.

An Ix macro is a function which takes as its arguments program fragments, called expressions. It may choose to evaluate these expressions, or may not, or may transform them to some other meaning. One important Ix macro is def, which creates named variables. The following example creates a variable n with value 3:

def n 3

How does it make sense for “n” to be an argument to a function call if the variable doesn’t exist? The answer lies in the concept of the macro: the macro isn’t a function of values, but rather program expressions. Thus, the def macro actually takes in the expression “n” in an abstract sense, and by inspecting it creates a variable binding.

Another macro is if, which takes in a value and two blocks, and acts as one would expect. Assuming some basic comparison operators, we now have enough tools to implement the Fibonacci function:

def fib [n] => {

if (> n 2) {

+ fib(- n 1) fib(- n 2);

} {

}

What we have now is a syntax very similar to a high-level language, one that is much friendlier to humans, yet every component is formed from the simple primitives explained herein.

I haven’t yet fully designed the syntax and semantics for higher constructs like classes and interfaces, but I do have some ideas how they’d work. First, consider an additional operator :: which acts like Haskell’s “$”, which right-associates the remaining expression. In other words, assume the following expressions are congruent:

· g (f a)

· (g (f a))

· g :: f a

I'm hesitating t to post this publicly, just because I know the language is likely to undergo rapid evolution as I apply increasing scrutiny to the syntax, and begin implementing samples in the language. The precise syntax needn't be decided with finality this early on, partially because I'm designing the interpreter in a very abstract manner (working only on "expression tree" objects) so as to function as an API exposing the syntax for other consumers, but also because one of my goals is to allow the language to be "skinned" in the sense that translation to/from various syntaxes should be pluggable.

You can't insure what has already happened

jcrites@jcrites.com (Justin Crites) — Fri, 31 Oct 2008 03:37:00 +0000

I like Obama. He seems like a moral, intelligent guy who thinks a lot about the decisions he has to make. I would expect his policies to be well thought-out, reasoned from first principles.

That's why I was surprised to read the details of his proposed health plan. They make no sense. I'm not speaking of my opinion -- I'm speaking of logical, objective sense.

Obama's page on healthcare states the following:

Require insurance companies to cover pre-existing conditions so all Americans regardless of their health status or history can get comprehensive benefits at fair and stable premiums.

Before I respond to this proposal specifically, let's review what "insurance" is. Bad things happen in the world: houses burn down. A lot of people invest a significant portion of their savings into their houses, and they'd be in a very bad place if the house were to burn down.

Disasters are unpredictable, and we don't know whether any particular house might burn down. We can estimate general risk, based on general factors, such as the temperature and the surroundings (dry brush nearby?). Consider the two possible outcomes after purchasing insurance:

Your house burns down, in which case you come out even.

Nothing happens, in which case you've suffered a loss of the premium.

The idea behind insurance is that we're betting something bad will happen. We're placing a bet such that we come out ahead if the bad thing happens. We're probably risk-averse, meaning we'd prefer to pay a few dollars a month to remove the (financial) risk, even if we don't expect it to happen. Or perhaps we're betting that the odds of our house burning down are large enough to warrant the cost of the insurance premium.

The insurance company charges a little bit more than the statistical cost of someone's house burning down, and so takes in a profit. The customers get piece of mind, knowing that there is no chance of losing their savings. This works out fine -- but it's entirely based on the fact that we don't know whether the house will burn down. That's the concept of risk.

If your house is currently on fire, will you be able to get any insurance company to sell you an insurance policy? Of course not. Since the house is already on fire, there is nothing to "insure" against. It's going to burn down, and someone will have to suffer that loss.

"Insuring" against events which have already happened is not only bad business, it simply makes no sense. When the house has already burned down, it makes no sense to after the fact ask an insurance company to sell you a policy. Hopefully now the direction of this essay is obvious.

Obama's healthcare plan is senseless because it requires healthcare insurance companies to give up their "pre-existing condition" clauses. But without those clauses it's not insurance!

The reason you can pay a premium of (say) $200 per month to have your healthcare costs covered in case you get sick and have a $10,000 hospital bill is precisely because it's not known whether you will get sick and incur the cost. If it's known in advance that you will get sick for sure, there is no basis with which to make a bet. The only thing to "insure" is other, unforeseen illnesses.

Forcing healthcare companies to remove pre-existing condition clauses is tantamount to forcing housing insurance companies to sell insurance to people whose houses are currently on fire. Some people currently can't get healthcare insurance because they're already to sick and there is nothing to insure. Requiring healthcare insurance companies to cover them only passes their financial burden on to everyone else, in the form of their premiums going up.

Insurance is about things that haven't happened yet. Anyone who disagrees is welcome to sell economic insurance against George W. Bush winning the 2004 Presidential elections, or against the housing crisis.

I strongly support healthcare reform in America. I do believe the system is bad, and I haven't made up my mind, but I probably support government-provided healthcare rather than insurance. I have no problem with that; I believe that people who understand such things well should go to work and build something better. We can discuss such a system and reason about it, again from our morals and first principles. I support working towards that goal, knowing from the beginning that we're operating outside the free market, because market efficiency is not our goal.

But I can't get behind government interference with the market in a way that is senseless and in staunch opposition to the principles the underlie that market. If you're creating a market, don't interfere with it. An insurance market is for insurance; there's nothing to insure when you already know the outcome.

Refactoring Expression Trees

jcrites@jcrites.com (Justin Crites) — Sun, 12 Oct 2008 03:41:00 +0000

In the first version of my Ix interpreter library I had two expression tree types: AST and AST (Abstract Syntax Tree and Abstract Semantic Graph, respectively). The idea was that syntax would be turned into an AST just after parsing, then by successive transformations become an Abstract Semantic Graph, where nodes in the graphs link to each other in appropriate ways (and so isn't really a tree at all).

The problem with this approach was that there are many transformation processes: the one that parses syntax to create AST, some transformations from AST to AST, and more from ASG to ASG. Where should which transformations take place? The idea, initially, was for all variable resolution to have taken place by the time the graph was an ASG, but this takes place in stages, and there is no easy way to keep track of "an AST with this kind of partial resolution".

Instead, an ideal language would allow the creation of custom types for each state in the process. One might imagine the series of transformations being expressed through a linear type system.

One could approximate a linear type system simply by giving stages names, for example a freshly parsed tree is of type ASG while the variable-resolution result is of type resolved ASG or something similar. After each transformation process we can "annotate" the value type to explain what's been done to it. It would only be meaningful to the user, but type compatibility prevent one from trying to execute, say, a tree which has not performed variable resolution or type checking. Immediately before the execution of the interpreter, we'd check for the existence of resolution, type checking, and other annotations.

I aim to have Ix support such features. We'll see how it works out.

XML as a front-end syntax

jcrites@jcrites.com (Justin Crites) — Sat, 04 Oct 2008 04:09:00 +0000

(This is the first in what I hope to be a series of many more blog posts written about the programming language I am developing called Ix (pronounced like Styx).)

I'm sure that the title of this post has already inspired horror in the readers who have understood its true implication.

In the course of working on the Ix internals, the VM and evaluation logic, I've had a need to test various properties of the system. In test fashion, I often need to mock or stub out pieces I have not yet implemented. Since I haven't yet come close to nailing down the syntax, I've needed a means to specify tree-structured data, to specify the syntax tree to internal components. Viola! XML seemed like the right fit.

Indeed, XML seemed the right fit as a choice of Intermediate Representation; the idea being to parse the front-end Ix language, convert it into an Abstract Syntax Tree, then process it from there. If one wanted to serialize any of these steps to disk, a language like XML seems the natural fit.

Having been toying with the concept of using XML for the IR and specifying test programs in IR directly, it dawned on me to consider using XML as a front-end language. Surely this has been tried? Many people are familiar with Ant -- would they want to write a whole program this way?

On first impression there are many downsides from writing code with XML. Things can rarely be expressed concisely. However, I think there may be some advantages too:

Improved Block Clarity

When in the middle of a large chunk of code, the reader may be better able to understand it due to the closing syntax precisely explaining which blocks are closing. For example:


   <fn name="something">
       <if ...>
           ...

           ...
       </if>
   </fn>

When the developer comes to the end of the block, there is no confusion which block is closed. While the close of a straightforward function is easy to understand, I believe there will be many cases, the equivalent of Java code like the following, where readability is actually improved in this manner. Here is a snippet of Java code from the test Ix evaluator:

          public RuntimeException apply(Val val) {
                   return new InvalidValueTypeException(context, expected, val);
               }
           });
       }
       public FnVal visit(FnVal fn) {
           return fn;
       }
   }

That's a lot of closing blocks! Can you tell what's what? Well no, because I omitted the top portion -- but even looking at the full code, you may be scrolled down and cannot see the top section either. I posit that some code like the following may be more readable:

    (You get the idea...)

   </fn>
   <fn name="visit"><public/><in name="fn" type="FnVal"/><out type="FnVal"/>
       <return>fn</return>
   </fn>
</class>

This property has been touted as an advantage of XML before; whether it actually improves programming readability is up for grabs, but seems in XML's favor. With a decent editor, the engineer won't actually end up typing more than "</", and so won't require more typing than typical languages' syntax.

Domain-Specific Languages are More Obvious and Better Supported

Here I am using an unusual definition of the term "Domain-Specific Language", which is traditionally a specialized foreign language, often as embedded within a general language. SQL is an example of a domain-specific language for database queries.

When I am working with a language, I often think of it as comprised of many smaller languages. Consider C++, for example. One has the "language of types", where T[N] means "array of type T, lengh N", such as int[10]. This is an entirely different language (in my thinking) than the "language of expressions", wherein T[N] means "access the Nth element of array T".

One of the interesting and novel aspects of C was the parallel syntax of arrays and types, seen in arrays and pointers and many other places. I am uncertain whether this kind of "analogy" is beneficial or detrimental, though I probably lean towards the latter, simply for the reason that a single lexical form having multiple meanings can be confusing because context is required to differentiate them (as is the case for type inference and other forms of conciseness-improving features which I believe harm readability -- but that's for another post).

In any case, I think of programming languages as compositions of these "domain-specific languages". Modern languages have many, including:

The language of types ( T* )

The language of expressions ( a + b )

The language of hierarchy ( a.b.c ), often overlapping language of expressions

The language of classes and structure; classes contain methods; methods contain the language of expressions

Thinking of languages in this way helps me to conceptually organize the syntax and semantics -- what kind of operators can be invoked in various circumstances. It also provides an obvious path to a modular language implementation.

Getting back to the point, as a language designer and a language user, I find it helpful for the circumstances to be very specific when we "context switch" into a domain-specific languages.

Some expressions absolutely require custom language support. For example, given a generic function-call syntax like Lisp S-expressions, one would not prefer to write ((. abc def) ghi) instead of abc.def.ghi; it's less clear. The author of Clojure, a Lisp-like language for the JVM, added a special syntax to parallel Java's dot operator because sequential, hierarchical access is unpalatable with S-expressions. In Clojure, (. target member ...) can be written (.member target ...), making the form more Lisp-like, and the language has primitive straightforward support for a Java-like dot operator too: new java.util.HashMap translates to Clojure (new java.util.HashMap).

With an XML syntax, such specific languages manifest as actual XML-embedded domain specific languages. While we could perform a literal translation of the S-expression form ((. abc def) ghi) into XML, the result would be poorly readable. Can we do better, but with XML? Perhaps something like this: <dot><o>abc</o><o>def</o><o>ghi</o></dot>. Ouch :-( No one is going to want to use a language like that. We fundamentally need a form more concise than XML can provide.

Instead, we can embed our language within XML, something like this: <dot>abc.def.ghi</dot>. It's "embedded" in the sense that the document has structure which is not represented in the XML -- generally an XML no-no as I understand it. However, for a programming language, our goal is to provide a workable syntax, and the embedded form is almost in every way preferable. (The only manner in which it's not preferable is that the expression requires a custom, non-XML parser.)

This is effectively a disadvantage of XML, in that one needs to work outside the XML framework to accomplish something, but that's only a problem if one sets out to do everything in XML, rather than use it for its strengths. The advantage of this kind of embedding is that it makes the embedding very obvious to the user; the engineer can be come directly familiar with the various sorts of embedded languages as he sees the various XML nodes invoking them. He understands which cases invoke which kind of syntax and there is little room for a lack of clarity in this matter.

An XML programming language syntax also naturally supports user extensions for domain-specific languages, since all it needs to do is permit the embedding of arbitrary XML trees in the language structure -- the syntax of the language is in this manner naturally extensible. One could imagine an implementation that allows language-engine-plugins to work on the abstract document, transforming custom DSL nodes into standard language nodes until no DSL nodes are left; and if any are left, there is a syntax error.

An XML-based language could support this much more easily, and in a more first-class manner, than others.

Interoperability

This property is simple. XML is a well-known, very well-supported standard. It would be easy for proprietary programs to read and write the language if its syntax is XML, since so much software already speaks XML.

The simple, parsable syntax would facilitate additional machine processing of the syntax itself -- for things like refactoring or bug finding. Handling the syntax is no longer the arduous task of writing a parser.

Support for Literate Programming

I imagine an XML language would provide excellent support for Literate Programming in Knuth fashion:

I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be works of literature. Hence, my title: "Literate Programming."

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other.

In short, Literate software contains extensive documentation and other elements meant mainly for human, not computer, consumption. XML would obviously provide incomparable support for this, as HTML and other XML-based documents could be embedded directly in the programs.

Overall, the use of XML syntax presents both opportunities and inherent disadvantages. It's difficult to conceive of a pure XML language, which represents all language structure as XML document structure. I wouldn't want to use that. However, there are some apparent significant advantages that are making me carefully look at the option of providing a front-end syntax for Ix which is primarily XML.

Regardless of whether XML ends up being a permanent front-end for the language, I think taking a critical look at an XML representation of the language tree will give me more familiarity with the deep structure and relationship between language elements, which should improve the ultimate syntax, whatever it may be.

Andromeda Strain

jcrites@jcrites.com (Justin Crites) — Mon, 02 Jun 2008 01:54:00 +0000

In the grand hacker tradition of writing book reviews and the personal goal of improving his communication, this geek has decided to regularly review the books he reads, starting with an explanation of his weekend experience reading Michael Crichton's Andromeda Strain:

Seeing that the SciFi channel is airing a miniseries film adaptation of Andromeda Strain, I thought I had better read it again. I'm glad I did.

What has always struck me about Crichton's writing is his ability to write very technically on any topic as an expert would. He is helped by the fact that he is an expert on many of the topics: he is a medical doctor, graduating from Harvard Medical School, and an excellent scholar in general (graduating summa cum laude from Harvard's undergraduate program). It his his ability to thoroughly research any topic and present its technical details convincingly that makes his writing so compelling. It's science fiction, but believable.

The Andromeda Strain's plot revolves around a space program Scoop looking for extra-terrestrial life, and the associated Wildfire Project, a group headed by five scientists (the protagonists) responsible for responding to any extra-terrestrial biological threats brought back by Scoop spacecraft. The book begins with a Scoop satellite unexpectedly crashing in a rural town in Arizona. Immediately thereafter, the majority of the town is found mysteriously dead in the streets.

The book describes in great detail the military procedures surrounding the event, including the chain of command, flow of information, and especially failures in that system. Wikipedia points out:

A notable recurring theme in Crichton's plots is the pathological failure of complex systems and their safeguards, whether biological ("Jurassic Park"), military/organizational ("The Andromeda Strain") or cybernetic ("Westworld").

The book is written as a single large false document; it appears to be an overview document written in the fictional world by an author surrogate narrator, scientifically documenting the events that took place and the systemic failures that worsened the situation. It is written in the style of the document one might expect to read tracing the events leading up to the
shuttle Columba disaster. The narrator's view subtly segues into first-person accounts of the protagonists.

The plot follows the chain of events set in motion beginning with the satellite's crash, the military's notice of it and subsequent clandestine issuance of a "Wildfire Alert", the activation of the scientists. The scientists travel to a secret hyper-modern (for the 1970s) computer-automated laboratory consisting of five separate underground levels of increasing clean-room strictness. Crichton details the uncomfortable, intrusive medical procedures the scientists must endure to sterilize themselves, their bodies, for experimentation on a hostile alien bacterium. The laboratory had much thought put into it and is prepared for the most dire of pathogenic villains; it comes equipped with a nuclear device that will computer-detonate if the facility detects that it has been compromised.

The narrator details the history of the structure and the scientific and political concerns surrounding its construction in great detail. Many false documents are presented to the reader describing contingency plans and possible interactions with Cold War-era Soviet Union in the event of a (test-ban-treaty-breaking) deployment of the nuclear device. It goes so far as to describe the command structure of the complex and explains psychological theories that advise who should be trusted to make the decision to stop or deploy the weapon.

When the scientists finally reach the fifth, most-clean level of the laboratory, they are faced with the monumental task of treating the two survivors and characterizing the nature of the plague; containment in the Arizona desert being nearly impossible, the team is in a race against time to develop an antidote before it spreads to the surrounding population and, presumably, the entire country.

As they work around the clock (having decided not to take amphetamines unless there is a *real* crisis), the characters and narrator explain the medical investigative process and experiments used to characterize the pathogen. Crichton's expert medical knowledge and hard scifi writing ability shine as he explains the various techniques and instruments, such as X-ray crystallography, and how it all fits together in the (although it is not explicitly mentioned).

Although the book is a great example of hard science fiction, Crichton only develops the characters enough to be passably believable; little time is spent on their motivations or emotions except as cogs in the larger system. The book's theme is clearly centered around the failure of large systems despite their planner's best efforts, and in general has a pessimistic view of applied science, explaining the herculean research efforts across decades needed to produce the most basic understanding of biology, and how futile this knowledge can be when confronted with the unknown (such as alien bacteria).

Andromeda Strain is quite a pessimistic book, but it is not gloomy. The conclusion of the book is merely that it is easier for things to go wrong than to go right, but beneath the surface of negativity is realism, a social commentary meant to counter what Crichton sees as blue-sky optimism. Ultimately, the book advocates a humanist philosophy: that humanity can control its own fate against adversity if we put in enough careful effort.

Overall, the book was quite enjoyable, a stroll into medicine and military policy that this software engineer found refreshing and intriguing. Andromeda Strain has the feel of a grand thought experiment, exploring both the depths and interactions between the military, US government, politics, science, and individual psychology. Although probably not the best example of Crichton's writing, Andromeda Strain has cemented its place on my list of must-read books in hard science fiction.

Thoughts about Software Transactional Memory

jcrites@jcrites.com (Justin Crites) — Mon, 26 May 2008 22:22:00 +0000

I've been reading and talking about Software Transactional Memory (STM) a bit. Commentators are conflicted as to whether it can perform well, with some claiming its performance degrades terribly under heavy load.

I was wondering, can we provide the same tools that STM provides at the language level with a smarter underlying implementation? The performance complaint centers around the fact that STM is implemented as an optimistic system, where each transaction attempts to perform its work, and detects conflicts with others and retries. With a high level of contention this can mean a lot of wasted effort; it can also make the character of the system hard to understand, with surprise jumps (down) in performance as load approaches certain levels.

Pessimistic locking systems, on the other hand, have predictable performance. A process has to lock a series of locks, does the work, releases them. There's no uncertainty or (much) wasted effort -- just time wasted acquiring locks. (Of course some CPU cycles are spent actually in the locking code, but this shouldn't be significant). The time spent acquiring the locks isn't really opportunity cost if the locking system is well-designed. We would expect with a good system that even if some threads are blocked on locks, overall the system is efficiently used. At least we assume this insofar as the algorithm that the program embodies actually is parallizable in the first place.

So, I wonder, can any regular language provide an atomic { } block? Let's say I provide this feature in a language. I need to generate locks around the blocks to provide correct semantics; I need to optimize so that it runs efficiently.

(We can't do this kind of reasoning in a SQL database environment because database servers don't know in advance what kind of queries they will be asked to evaluate. However, a compiler does know all about the code it will be evaluating, so we would expect the optimal locking algorithm for programs to be different and better than a database server's approach.)

A simple algorithm can provide correct semantics: every atomic block locks a single global lock. This is trivially correct but not very efficient. Can we do better?

Let's say that our processes are represented as {A, B, C} where the block operates on some shared variables A, B, C. The particular operations or order is unimportant since we're guaranteeing that process has a transactional view of all variables. (Well, we could in theory optimize blocks down for the programmer, by detecting that (say) one atomic block actually could be split up into two. Let's ignore that for now or just assume a separate and correct algorithm for minimizing the data shared in atomic blocks.)

We have some set of processes like {A, B} {B, C} {C, D} {D, A}. We humans know that, for example, both process {A, B} and {C, D} could execute in parallel with no conflict -- but they don't, because a process can only execute its atomic block if it's holding the global lock. We want our STM system to be at least smart enough to run parallel tasks in parallel, so this isn't good enough.

Let's approach from another direction. Rather than having a single global lock, let's give every variable its own lock. Thus the process {D, A} must get the lock for D and A before proceeding.

This is efficient, because we allow {A, B} and {C, D} to run in parallel; they will each lock their respective locks and go on their way.

What about guaranteeing progress? Let's say we run all the processes in parallel. Perhaps {D, A} runs far enough to get the lock for D; next the process {C, D} runs, gets C, blocks on D; next {B, C} runs and blocks on C; and {A, B} runs and blocks on B. Let me describe each process and the locks it currently holds:

{A, B}: A (waiting on B)

{B, C}: B (waiting on C)

{C, D}: C (waiting on D)

{D, A}: D (waiting on A)

It's easy to see that we have a full deadlock -- all the processes are waiting on one another. Of course, it is not a particularly likely circumstance, but we as professional software engineers don't build our systems upon wishful thinking -- right?! We need a different way to do things.

Really, the problem as described above is the Dining Philosophers problem, and only a small adjustment is needed to prevent deadlock. The trick is in that process {D, A}. What we need is a locking convention; as long as there is a global total order on the order in which processes acquire locks, we are ensured that there is no deadlock. This formulation of the problem naturally lends itself to one: we can lock all locks in alphabetical order. This will work even if processes require many locks, like {A, B, C}, {B, C, D}, etc.

It sounds to me like this solution would be viable, but probably not efficient. One could imagine locking in order based on the memory locations of the objects (assuming they're constant) or imposing some other ordering scheme on object locks.

This would work if the amount of shared state is small (a couple of objects accessed per atomic), but what if the number is large? Could we use this for a game, if we need to (say) iterate through an array of all objects in the world and update their physics? This system breaks down if blocks need to access large numbers of objects.

From here we could imagine improving our algorithm, aggregating locks together to reduce their number but preserve the same atomic semantics. For example, we might say that we only need to lock an array to access objects contained in the array, rather than locking each individual object. (This would require assuming that the objects cannot be referenced any other way.) Where common locks exist, we could imagine coalescing them into a single lock. For example, if many processes access exactly two variables, we could imagine the algorithm determining that they both will be protected by the same lock. Another alternative is to have a single global lock which a process locks, then may obtain at once any number of other locks at once quickly. There are a lot of possible optimizations we could research to make this locking system faster.

A quick poll of some PhD friends suggests that there is probably not much research in this area. There's a lot to explore! We could even imagine a locking system that dynamically profiles itself and re-organizes the locking structure to maximize throughput. If the environment is low-level enough, the scheduler could even be aware of the performance characteristics and could schedule accordingly (schedulers already do know which threads are blocked on locks, e.g.).

When the programming language I am designing gets a bit further along I will attempt to implement a simple atomic block construct. One could imagine a simple naive global lock implementation for development followed by later optimization as necessary; it might be the case that a naive implementation is sufficient for many simple cases (like basic GUIs). If it is a useful way to program, hopefully we can discover solid implementations of the construct.

Is STM the solution? STM is much easier when one is working in a pure functional programming language with little or no shared state in many places. However, given that procedural languages are used widely in the industry, I hope other researchers are as interested in these problems as I am!

Overall, STM feels a lot to me like Garbage Collection (GC). The goal of it isn't to make programs run more efficiently; the goal is to make programming easier and thereby use programmer time more efficiently. Working with locks is difficult and it's easy to make mistakes; if STM can alleviate this with only a 20% performance decrease, it would be a huge win. I say it's like GC because it's a similar tradeoff: we'll give up memory efficiency and even runtime performance, and even yet suffer possible random freezes as the GC kicks in, but in allowing these things we make the much larger gain of freeing programmers from the concern of memory management. Managing memory and managing locks: both meticulous tasks well suited for a computer.

By adopting GC we've freed ourselves from the tiring troubleshooting of obscure memory issues, and become more productive. I wonder whether freeing ourselves from similar concurrent bugs would bring about a similar but even greater productivity increase. If we gain a powerful Software Transactional Memory model, not only will developers free themselves from troubleshooting concurrency bugs, but we may find developers generally being more willing to write concurrent software.

Think about it! Writing concurrent software is hard -- why bother if it will only provide a minor speedup? Commentators often note that software like Microsoft Word won't speed up on multiple processors. It's not because this software couldn't be sped up by multiple processors, it's that the gain in performance is not worth the extra development cost (which is likely to be small for most desktop software). But if the added development cost is low, suddenly concurrent implementations become more viable.

That's where I see things heading -- mainstream programming languages make concurrency easy, not just in the Big Server platform (Erlang) but in the everyday desktop platform. Once there is first-class language support, concurrent software will become a lot more available.

However, developers will still need to develop concurrent algorithms, which would take extra effort. And there's still the question of whether compilers that optimized as I've described above could ever be implemented. Donald Knuth, in a recent interview with Andrew Binstock, expressed doubt about whether the highly concurrent hardware will be well-used:

To me, it looks more or less like the hardware designers have run out of ideas, and that they’re trying to pass the blame for the future demise of Moore’s Law to the software writers by giving us machines that work faster only on a few key benchmarks! I won’t be surprised at all if the whole multithreading idea turns out to be a flop, worse than the "Itanium" approach that was supposed to be so terrific—until it turned out that the wished-for compilers were basically impossible to write.

Let me put it this way: During the past 50 years, I’ve written well over a thousand programs, many of which have substantial size. I can’t think of even five of those programs that would have been enhanced noticeably by parallelism or multithreading. Surely, for example, multiple processors are no help to TeX.

How many programmers do you know who are enthusiastic about these promised machines of the future? I hear almost nothing but grief from software people, although the hardware folks in our department assure me that I’m wrong.

I know that important applications for parallelism exist—rendering graphics, breaking codes, scanning images, simulating physical and biological processes, etc. But all these applications require dedicated code and special-purpose techniques, which will need to be changed substantially every few years. ...

The machine I use today has dual processors. I get to use them both only when I’m running two independent jobs at the same time; that’s nice, but it happens only a few minutes every week. If I had four processors, or eight, or more, I still wouldn’t be any better off, considering the kind of work I do—even though I’m using my computer almost every day during most of the day. So why should I be so happy about the future that hardware vendors promise? They think a magic bullet will come along to make multicores speed up my kind of work; I think it’s a pipe dream.

Cheers to all the researchers out there working hard to make this pipe dream reality.

[Edit: I have recently found an interesting paper relevant to the topic: Advanced Contention Management for Dynamic Software Transactional Memory. I'll write a follow-up if it's particularly relevant.]

Trying out Twitter

jcrites@jcrites.com (Justin Crites) — Mon, 26 May 2008 04:59:00 +0000

Rather than set up some kind of "tumblelog" I thought I'd give Twitter another shot.

If I remember correctly, Twitter came out a couple of years ago, when I was working for Scott Ruthfield at Amazon.com (or at least launched a public beta). I wasn't enthused at the time, but I also hung out on AIM and chatted all the time. Now that I'm more, I suppose, professional, perhaps I will get some functionality out of a site that allows frequent chat-like communication without the distractional burden of being on IM.

I haven't frequented IM in a while; somewhere around the middle or late years of college I realized that it's more of a net negative than positive for getting things done. That's not always my focus, of course (if it were I would have a pretty bleak life!), but having IM open is an invitation for distraction and just a way to make excuses to myself to procrastinate what I would otherwise be doing.

Mingling work and chat seems like a less efficient way to work and a less fun way to chat. Twitter is like IM but cheating; I can send you IMs but you can't directly send them back (I would have to check some page to see them ;-)

Anyway, check out my Twitter page!

Google Calculator

jcrites@jcrites.com (Justin Crites) — Sat, 24 May 2008 23:07:00 +0000

[Note: This post uses ASCIIMathML.js, a script that makes it easy to typeset mathematical formulas within HTML using MathML. If your browser doesn't support MathML, you will probably just see the text representation of formulas.]

Some friends and I were discussing the feasibility of a mass driver for travel to the Moon (for fun). A mass driver, or catapult, can theoretically launch objects into space by propelling them at such a high speed they escape the atmosphere; no rocket necessary, just a high launch velocity. I thought I would describe how Google Calculator helped me reason about the problem easily (especially all the unit conversions).

Google Calculator (as I am calling it) is a feature built into the regular search on Google.com. If you include a mathematical expression as your search term, Google will evaluate it for you. Here's a good reference. It can do basic arithmetic, but its most helpful feature is its unit conversions. The calculator will convert basic units as necessary to evaluate the arithmetic, but can also be instructed to convert the answer into some specific units. For example: "60 miles per hour to kph": 60 miles per hour = 96.56064 kph.

To catapult into orbit, the launch velocity would have to be greater than escape velocity, of course, or the Earth's gravity would pull you back down. Earth's escape velocity is about 12 km/s, or roughly 26 800 miles per hour. That's pretty fast!

A mass driver would need a track of some kind: imagine a sort of vertical roller coaster that releases you when it comes to the end of its track. It would have to accelerate you pretty strongly! How strongly? Let's say we want to get to a bit faster of a speed than `12 " km/s"`, so we can settle on `13 " km/s"`.

Well, let's say we have a `5 " km"` track. How much acceleration is necessary to get us to 13 km/s? We need some physics to help us solve this one, as classic formula from AP Physics:

`v^2 = v0^2 + 2a(x-x0)`

In our case, `v0 = 0`, `x-x0 = 5 " km"`, and `v = 13 " km/s"`. So we have `v^2 = 2*a*(5 " km")`. Solving this, `(13 " km/s")^2/(2*5" km") = a`.

We can solve this with Google! Enter "(13 km/s)^2/(2*5 km)=". Google returns: ((13 (km / s))^2) / (2 * 5 km) = 16 900 m / s^2.

We can also ask it to express the result in terms of "g": (13 km/s)^2/(2*5 km) in g= produces the answer ((13 (km / s))^2) / (2 * 5 km) = 1 723.3204 g. Ouch! We couldn't survive that kind of acceleration.

How much energy would it take to accelerate us? The physics formula `E = 1/2mv^2` describes the energy of objects with velocity. Let's say the object we're launching is about a ton, the size of a small car. After all, it doesn't need an engine since we're catapulting it into space! Google "1/2*(1 ton)*(13 km/s)^2=" gives us 76 657 110 530 joules.

That's a lot of energy -- about 21,000 kilowatt hours! To help visualize the result, let's convert it to some more tangible quantity. How much water could this energy vaporize? That is, if we have some quantity of water, how much water could this energy bring from 0 degrees C to 100 degrees C?

Wikipedia tells us that water has the highest known specific heat of any compound after ammonia, and a very high heat of vaporization. The specific heat is 4.186 joule/(gram degree C). Thus, to change a gram of water from 0 C to 100 C we need 418.6 joules per gram.

Earlier we found the energy is 76,657,110,530 joules. With that and our 418 joules per gram number, we find that the energy can heat up about 183000 kilograms. We know via references that water's density is 1000 kilograms per cubic meter, so finally we can solve for the volume of water that can be raised from 0 C to 100 C by this amount of energy! About 180 cubic meters or 48,000 gallons.

Unit conversions are an almost immeasurable help when doing tricky physics calculations. Google Calculator's abilities surpass even the respectable TI-89 calculator (which, if you need a calculator, is an excellent purchase).

Shoshin

jcrites@jcrites.com (Justin Crites) — Sun, 18 May 2008 05:30:00 +0000

I'm christening my blog Beginner's Mind, a term from Zen Buddhism. Beginner's Mind, as described by Wikipedia, is "an attitude of openness, eagerness, and lack of preconceptions when studying a subject, even when studying at an advanced level".

Also known as Shoshin or Zen Mind, the concept encourages all practitioners of any art to consider themselves students, no matter their skill level. This reflects the wisdom that a person can always learn more; an expert should not become smug in his knowledge, less he sabotage his own efforts to continue his mastery.

Software development is an exciting field because it is always changing. Tools invented only a decade ago, such as the programming languages Ruby and Python, revolutionize sectors of the field today. Development feeds back upon itself exponentially: when a programmer releases a tool, it becomes easier and faster for the rest to make more tools, and so they do. We are just now approaching what may be the golden age of programming, where techniques, tools, and computers are widespread and the industry is mature enough to be taken seriously.

Beginner's Mind is essential in the software industry precisely because of this high rate of change. Not only are the tools changing, but also since software development is such a young field, best practices for the same tools are being discovered and discussed continually. A good idea ten years ago might be a terrible anachronism today.

As software developers aspiring to master our craft, we must always knead and fold over our knowledge, ruminate over it, masticate it; occasionally, we must defenestrate it.

Writing today about software development is frequently shortsighted and partisan; we have made a religion of our tools. It is common to see camps of people strongly in favor of some particular tool write tirades in defense of it, or attack another tool. A Buddhist master would say that this is attachment, a mental block one erects to protect his view of the world and avoid cognitive dissonance.

The goal of my writing in this blog will be to treat software development (and other topics worth discussing) with Beginner's Mind as I navigate its nooks and crannies in my daily life as a software developer. More broadly, I aim to take a neutral look at the industry, its tools, and its people.

(While trying to come up with a clever conclusion for the end of this post, I came up with "I hope to offer crumbs of insight from the cake of enlightenment". While it is an evocative metaphor, and I definitely don't plan to write this blog with a deadly serious tone, I'll try to spare you, gentle reader, from the wrath of wearisome wit.)