Things I think about when starting a new project that, surprisingly, many people appear not to.

I switched projects at work and now I’m going through the struggle of trying to figure out how to build other people’s projects. It’s always a struggle, of course: a lack of familiarity always makes things harder than they should.

But there are some common things that I realized that don’t seem to be as universal as they should be.

So here are some things I think about when I start work on a brand new project, that I wish other people would also prioritize.

How will you debug the project?

Seems weird to put this at the top of the list. But there are quite a few web applications being built where, in the list of things people worry about, “debugability” isn’t even in the top 5. Oh, sure; they think about configurability (how will I configure the tool dynamically?), and logging (how will I get information about problems off a server that is locked down from me?), and management (how will I remotely manage something that has elements of it locked down from me?). But for whatever reason, the simple ability to reach up in the toolbar, and press the ‘debug’ button and have their product (in one automated process) build, locally deploy, launch, and stop on a locally set breakpoint doesn’t even appear in the top 10 of things people seem to worry about.

All too often debugging a web application seems to be “oh, sure; just kick off Maven with some magic combination of things, then copy this file there, and run that script–oh, and connect your debugger to the running process.” Some of those steps, of course, are more lore than they are documented processes. And it’s a terrible substitute for just checking out the project, and hitting the “debug” button.

I’m a huge fan of spending a few days and figuring out which integrated tools you need to allow for one-button debugging within Eclipse, my personal favorite IDE. Remote debugging should only be used as a last resort, in order to attach to a running remote process, in order to diagnose a problem with a deployed application. But ideally all debugging should be done by just hitting the little bug icon.

In fact, I’d suggest this is the first thing that should be done when setting up a project, before figuring out how you’re going to deploy the project, before figuring out how you’re going to package the project, or before you figure out which libraries you intend to use. And this should be the first thing on the top of your mind as you incorporate new technologies.

Because, in my opinion, if you check out a project onto a brand new computer, and it doesn’t just get sucked into the IDE, ready to be debugged in one step by pressing the debug button, your build is broken.

How will you bring new team members up to speed or replace them?

In part this is a technical question, and it has to do with documenting your processes. How does a new team member check out the sources from the source repository? Which tools does he need to install in his IDE in order to run your project? Where can he get the latest documentation?

If you’ve made sure that, once the IDE is installed, checking out the source from the source repository simply “works” (see the step above), then you’ve saved yourself a lot of documentation time.

But any document is also a product that must be debugged and maintained. So who will maintain that document? Who will debug the document (run through the steps to make sure they’re clear and correct)?

And if you’re in a shop which supports multiple IDEs, then you have an extra step you must perform: first, figure out how you will allow multiple IDEs to co-exist and operate on the same source base. And second, maintain the projects across different IDEs. Now that could be as simple as having a culture of constant communications when new files are incorporated into a project. But it does mean the project files will need to be maintained as well as the sources–and it does mean the documentation will also need to be maintained as the project evolves.

How will you distribute the resulting product?

For whatever reason, in the three web shops I’ve worked for, web distribution was always treated a bit cavalierly: since we maintain the severs, naturally we don’t need to give as much thought towards distributing the resulting product as we do if we’re distributing a CD-based product or an App Store distributed product.

And while this is true on the surface, every web shop I worked for also has an operations team and a support team who are expecting a “run book” or other documentation on how to install the product and how to maintain the product.

In other words, they’re asking for a software user’s manual.

Just like the ones we used to print for a CD distributed product.

So you can’t escape the following questions, even if you work for a web shop.

How will you distribute your product? What installer will you use to install the product? What are the distribution products, and who will maintain those distribution products? What about the documentation; who will maintain it? Who will test the installation process? Who will make sure that on first boot it’s obvious what to do?

And how will you alert the user when things go wrong, and how will you tell the user what to do when things go wrong–and when to call you for support, and when to handle things themselves?

How will you get diagnostic information back when things go wrong? And what form will that diagnostic information take? Is it sufficient to help you pinpoint the failure?

It’s surprising to me how little thought seems to be given to these basic questions. What’s even more interesting to me are the number of developers who treat these questions with disdain: I know a few who are proud of the fact that their “IDE” is vi and their debug process is printf(). Okay, if that is what chimes your bells–but most of us are more interested in tightening the edit/compile/debug cycle and in making the project more manageable for real human beings.

*sigh*

How long before Facebook is back? I use Facebook for more personal posts and politically oriented stuff, and I use this blog for more technical-oriented topics. So clearly, Facebook is now down for my account, especially when I have a long winded politically oriented post I want to write. (*sigh*)

Little killers: channel abuse.

Channel abuse.

When you use a magic value in a particular field (such as a text field) in order to signify something else. For example, if you use a magic text pattern (like ‘XXX’) to signify the last record.

This can kill you because (a) what if the user types in “XXX” into that field? And (b) if you’re storing the record as an explicit marker, what happens if your magic record goes away?

Better off either creating some sort of conversion protocol to allow out-of-band information to reside within the text (like the & escape character in HTML), or to create a separate control channel to contain control data. And if you’re using the symbol as an end of array marker, you’re much better off using the built-in OS routines (end of file marker, end of database marker, etc) to determine you’re at the end of the list.

A surprising observation, or finding your own voice.

One thing that surprised me was the percentage of people who, on reading my code from my last post, took umbrage to the following line of code:

if (null != (ret = cache.get(n))) return ret;

(Okay, in all fairness, only two people out of 20 or so commenters made a comment–about one in 10, but on a trivially small data set.)

It fascinates me.

First, a disclaimer. I don’t mean to use my comments to disparage anyone. I don’t know the people who made their comments, and I’m sure if they’re here they’re clearly very intelligent people of the highest caliber whose code quality is undoubtedly impeccable. If it wasn’t, they wouldn’t be here, right?

My comments go to current development trends, however, which motivate people to be far more interested in form over function.

There is nothing new here, by the way, as I go into in my comments below. But I’m fascinated by the devotion of form over function that is being taught to our developers which sometimes makes people blind to the reasons why the form exist.

I started tinkering with computers in 1977 when I played with the BASIC interpreter on a TRS-80 in the 8th grade. I don’t observe this to suggest that my experience trumps other people’s experience or to present myself as some sort of code guru or expert that people should not argue with. I only note this to note that I’ve been around a while and have seen different trends ebb and flow over the past 34 years of hacking, and over the past 23 years or so of doing this professionally since getting my B.S. in Math from Caltech.

It’s context, nothing more.

But in that time, I’ve noticed a few shifting trends: things that at one time was considered “best practice” are is now considered poor practice or visa versa.

Take, for example, the statement above that started this whole article. One suggestion I saw was to rewrite the code:

MyClass ret = cache.get(n);
if (null != ret) return ret;

We could even go so far as to rewrite this statement with the allocator statement reserving the variable on a separate line:

MyClass ret;
ret = cache.get(n);
if (null != ret) return ret;

When I started writing software, we edited our code on an 80 x 24 character display. This means you could only see 24 lines of code at any one time. Back then, the two statements below would have consumed two or three of those 24 lines of code, and so would be considered inferior to the one line statement:

if (null != (ret = cache.get(n))) return ret;

Back then, the limit on the number of characters in a line also favored shorter variable names. Setting aside, of course, that earlier C compilers could only match variable names of 6 characters or less (so that, for example, ‘myVariable’ would match ‘myVari’ or ‘myVariFoo’), which was imposed partially for memory reasons, but partially because of a lack of need–variable names were kept short because:

if (null != (foundFactorialReturnValue = factorialReturnStorage.get(n))) return ret;

This could get pretty unwieldy.

It gets worse when discussing formulas, such as the distance between two points:

double dx = p1.x - p2.x;
double dy = p1.y - p2.y;
double dist = Math.sqrt(dx * dx + dy * dy);

is easier to follow than:

double deltaXCoordinate = point1.xCoordinate - point2.xCoordinate;
double deltaYCoordinate = point1.yCoordinate - point2.yCoordinate;
double distanceBetweenPoints = Math.sqrt(deltaXCoordinate * deltaXCoordinate + deltaYCoordinate * deltaYCoordinate);

Of course programming styles change over the years. We’re no longer constrained by the 80×24 character limits of a green ADM-2A terminal. My computer display at home is 30 inches in diagonal, and capable of displaying several hundred lines of code with documentation in a separate window. Even the smallest MacBook Air has a pixel resolution of 1366 x 768 pixels; at 12 pixels per line, this means you can easily display 55 lines of code with room left over for the window tile and decorators and the menu bar.

And of course in the desire to cram more and more code into an 80×24 character display, C programmers took some “liberties” that took the whole drive towards putting as much information within a single line of code waaaay to far, writing such abominations as:

for (ct=0,p=list;p;++ct,p=p->next) ;

which counts the number of items in a linked list. (The count is in ct, the list in list.)

(In fact, this drive for “clarity” through compactness was one of the inspirations that led to the creation of the International Obfuscated C Code Contest.)

Today, I believe the pendulum has swung too far in the other direction. We’re so hell bent on the proper form (out of concern that, by putting a compound statement in a single line of code it will make it ‘harder’ to understand) that we even have tools (such as Checkstyle) which will enforce syntactic styles–throwing an error during the build process if someone writes an early return or a compound statement.

And while I’m not arguing anarchy, I do believe going so far as to break the build because someone dared to write a compound statement with an early return rather than writing:

MyClass ret;
ret = cache.get(n);
if (null == ret) {
    // some additional logic with a single exit point 
    // at the end of the if statement, using no returns, 
    // breaks or continues or (God help us!) gotos
}
return ret;

is going too far. (Setting ‘ReturnCount’ = 1 in Checkstyle.)

Imagine a world with only declarative sentences. There are no conjunctions. All sentences follow a proper format. All sentences start with a noun. The noun is followed by a proper verb phrase. The verb phrase is followed by a well structured object. The object of the sentence is a proper noun phrase.

Imagine this world with well written sentences. Sentences that follow the format taught in the third grade.

“I want you to be an uncle to me,” said Mr. George Wright. He leaned forward towards the old sailor.

“Yes,” said Mr. Kemp. Mr. Kemp was mystified. Mr. Kemp paused with a mug of beer midway to his lips.

Or:

The question is ‘to be or not to be.’

Is it nobler to suffer outrageous fortune?

Or should we take arms against a sea of rising troubles, and end them?

To die? To sleep no more?

Sorry, but I don’t think Shakespeare or Hemingway are helped by our rules.

Ultimately writing code has two goals.

The first is to accomplish a task, to create a software package that can be deployed which accomplishes the specified task with as few bugs (or no bugs) as possible.

The second is to produce maintainable code: code that, years from now, you can figure out. And code that, more likely than not, will be handed off to a maintenance developer–possibly overseas in India or China–who will be asked to understand and maintain the software that you wrote.

Now both tasks can be helped by writing simple code: that was yesterday’s post.

But code legibility can also be helped by thinking about the code you write.

To me, writing code is like writing an essay. Like an essay, which is full of sentences and paragraphs and sections, code is full of statements, and code groupings (like paragraphs) and classes or modules.

And like sentences, which has rules that we were all taught in the third grade (but then we later ignore as we learn the rules of the road and find our own voice), code too has rules of legibility that we should be able to break judiciously as we gain experience.

Each statement of code is, in a way, a sentence: it has a noun (the object being operated on), a verb (the operator or function call), subjects (parameters), and so forth. While we’re taught that a sentence must have a subject, a verb and an object, we learn later on that perhaps to express ourselves we can bend the rules.

So, for example:

if (null != (ret = cache.get(n))) return ret;

This may be a perfectly acceptable statement under the right circumstances: the idea is clearly expressed (get the cached value and return it if it was found), the logic is easy to follow.

And by putting it on one line, our focus is carried away from the logic of checking the cache and can focus on the multiple lines of calculating the cached value. We can concentrate, in other words, on the meat of the algorithm (the computational code), and the code which bypasses the check can be made into a functional footnote.

Of course there are places where this can be the wrong thing to write as well: if the emphasis in the routine, for example, is on checking the cache–well, then perhaps this deserves a multi-line statement.

Or perhaps when we find a value in our cache we trip off some other logic, then the logic deserves a line of it’s own. Perhaps the checking is sufficiently important enough that it needs to be called out separately, like:

ret = cache.get(n);
if (null != ret) {
	return doThing(ret);
}

It’s all a matter of communicating, with your own voice, what is and is not important, so future generations of code maintainers can understand what is and is not important, what goes together and what is separate.

Ultimately it’s about striving for a balance: creating working code that can be understood, by using idioms of programming which convey the subtext of the meaning of that code.

Sure, when you’re inexperienced and you haven’t found your voice, it’s appropriate to follow a strict “noun/verb/object” structure. It’s appropriate, in other words, to use simple declarative statements while you gain experience writing code, and to observe other common “best practices” such as using descriptive variable names.

But at some point you need to find your own voice. When you do, it’s also appropriate to break the rules.

And if you concentrate too much on the rules rather than on what’s being said, then perhaps you’ll also make the mistake both commentors did when commenting on my code style, when they failed to note the code itself was actually broken, with a dangerous infinite loop triggered by certain parameter values.

Complexity–or how hard is it to display a list of 3,000 items in a table on MacOS X anyway?

The laptop on my desk can do around 2,403 MWIPS, around 100 times faster than the Cray X-MP/24 of yesteryears, and has easily more than 100 times the memory. (For reference, NASA clocked the Cray X-MP/24 at 25 MWIPS, and the system came with either 16 or 32 megabytes of main memory, in 1 megabyte banks.)

So how friggin’ hard is it for Outlook Express to display the summary lines of 3,000 e-mails on my MacBook Pro?

separator.png

Here’s the thing. I suspect (though I don’t know because I don’t work for Microsoft) that rather than suck in the headers into memory from an e-mail file, storing those headers in a linked list and then, as the user scrolls up or down in the scroll bar, the system runs a pointer to the first point of the linked list, and simply does a “drawRow(…); ptr = ptr->next;” operation, instead the system is using something like SQLite for storage and doing a database query to the SQL engine, like “SELECT … ORDERED BY … LIMIT (n-rows) OFFET (scroll-amount)”.

Then, behind the scenes, the SQL statement is then compiled into a byte code for a SQL interpreter engine, the engine then interprets the byte code using a byte code interpreter, with each instruction pulling in elements of the table (stored in a way which is optimized for reducing disk space while arbitrary records are added and removed) into an on-board cache, which sometimes is missed because–well, we’re only interested in a dozen or so rows displayed in the summary table, right?

The end result is that a computer 100 times faster than the fastest computer in the world from 30 years ago (which, honestly, was pretty blindingly fast) and which has more than enough memory to store four encoded movies in RAM, hiccups and stutters when I scroll through my e-mails.

Really?

separator.png

I’d like to call this lazy, but I really can’t: wiring up the UI of Oracle to a back-end SQL engine, then maintaining the link of those systems in such a way which allows you to page through a few thousand e-mails is not a trivial task. Nevermind that SQL was designed in part to reduce its memory footprint for large datasets that cannot be stored in memory, and we’re using it on an e-mail database which could be stored in 1/100th of the available RAM on my system.

Instead, I think it’s because software development today tends to be more about marrying off-the-shelf components without thinking about what those off-the-shelf components actually do–or even if (as in the case of tracking 3,000 e-mails on a system that would have been considered 20 years ago a super-computer whose computational power made it a Federal crime to export overseas) the off-the-shelf components are even warranted.

Now if my e-mail box had, say, 1 million e-mail messages, and a total disk footprint of, say, 2 gigabytes–then I could see using a complicated relational database engine to manage those e-mails. But instead, in an effort to reduce the magnitude of the exponent in an O(n**e) operation, we forget that there is a constant C which can often dominate.

Which is why an e-mail system which simply parses all 3,000 e-mails into a linked list (and which would handle operations in O(n) time) would be far faster than a complicated system using a relational database that perhaps may run at a theoretical O(n**0.5) time, but whose constant C for each operation is measured in milliseconds verses nanoseconds for the former.

separator.png

I wish we would as an industry spend less time thinking about the tools and toys which we use to solve a problem, and more time thinking about the domain the problem resides in–and selecting the appropriate tools given the domain, rather than simply asserting the latest and greatest gee-wiz gadget is the right answer regardless of the question.

I’ve seen tons of examples of this sort of thing. Adobe Air used to build a user interface because, well, it’s pretty–without thinking for a second if it can even be deployed on the target devices used by people in the field. Javascript/AJAX web sites used for people who travel and whose devices do not have a consistent or reliable Internet connection. Sending and receiving full queries across a slow EDGE network connection when on-board caching would reduce the queries to status updates (or even allow the system to run without any queries whatsoever).

We don’t think of the domain of the problem or figure out the priorities of the components of that problem or pick the appropriate tools and technologies.

And so I’m stuck with a computer that is a marvel of advanced technology, a super-computer in a briefcase capable of running a simulation that can model the solution to Schrodinger’s Equation fast enough to allow a physicist to fiddle with the inputs and see the outputs in real time, run like a fucking dog when scrolling through my morning’s e-mails.

Insanity.

Insanity is having a two week scrum cycle and expecting to push working code to production at the conclusion of every single blasted cycle.

Insanity is expecting engineering to read, digest, and understand every single last bullet point on a 50 page product specification document in a matter of weeks, written by product managers who think they’re architects designing a product.

Insanity is doubling-down on the previous statement by creating 13 or 14 such 50 page documents, because engineering didn’t understand the first 50 page document.

Insanity is getting pissed off at said engineering because they notice conflicts on the bullet points in said 13 volume requirement and wonder how they should implement something that resolves the conflicting points.

Insanity is not shielding your direct reports from the vagaries and insanities of upper management, who believes their lack of understanding is your emergency.

Insanity is not pushing back at said upper management who thinks if they can buy something as complex as Microsoft Office at Amazon for a few hundred bucks and have it delivered in two days, then clearly you should be able to deliver something similarly complex in the same time frame and on the same budget.

Insanity is thinking that an admittedly poor manager managing an admittedly poor employee will somehow result in excellence.

Insanity is thinking making people look good both up and down the management chain is your first priority, or even your second or third. (Not to say cooperation isn’t vital–but cooperation is not the same as making people look good: the first requires getting along, the second requires pushing out mis-information.)

Insanity is stripping head count from a well functioning group and handing them to a poorly functioning group, on the theory that this will balance the workload.

Insanity is forgetting that everyone you manage has the potential of becoming an extremely talented person–some simply need to be taught, some you need patience with, and some who are naturally talented, if not recognized, may find greener pastures elsewhere. And even if at their maximum potential some people are far less productive than others, they’ll still be far more productive if encouraged.

Insanity is not being forward thinking.

Summarizing yesterday’s post on user interfaces.

If you have a system design that requires the operator to make 20 decisions in the back end in order to accomplish a task, don’t complain the guys building the front-end user interface that the user interface requires too many decisions.

The only thing the UI guys can do is either (1) hide the decisions that don’t need to be made (though if you’re asking for decisions that don’t need to be made, WTF?), (2) present the decisions in a way which is easier to understand.

But if you have 20 decisions that have to be made, no amount of UI goodness is going to reduce the effort of making 20 decisions.

So if you want a simpler UI, reduce the decisions that are required by the system.