Stating the obvious.

Just read another article telling us of yet another way to build a great product for yet another set of technologies. And I can’t believe I’m about to list some things that should be painfully obvious to any developer–things which were ignored by this article.

When building a web site or a mobile application which communicates with a back-end server:

Assume your wire protocol is completely insecure.

There are dozens of tools out there which allows a user of your app (your web site, your mobile app, your desktop app) to see transactions going back and forth to your back end server.

“But I’m using HTTPS, that’s secure.”

Sure, against third party snoopers–and not even there, if someone compromises the certificates, or if it’s a product designed to peek inside HTTPS packets. (I’m looking at you, Symantec.)

But it does not protect against someone with the right tools to create a proxy that allows them to decode traffic. And see exactly how your API works.

Corollary: If your business logic is in the front end, you don’t control your business logic.

Once someone has decoded your API protocol, it’s easy for them to then make calls into your API protocol. And if your logic to determine what a user can and cannot do is contained in your app, it’s easy for them to bypass those checks and (oh, say) ship a thousand dollars worth of product to their front door without paying a dime.

Corollary: If your security checks are in the front end, your site is insecure.

That basically follows from the above.

Interestingly we tend to forget this when attempting to implement ‘RESTful’ interfaces–that is, supposedly stateless user interfaces–by pushing security checks onto the client. But that subjects your app to a “replay attack”–where a bad guy snoops the web traffic, discovers the token representing security state, then fiddles that security state to obtain control.

A secure pure REST interface is impossible, if only because when a user logs in, that login state (such as an OAuth token) must be generated and transmitted to the front-end. More importantly it must be invalidated and a new login state token generated the next time the user logs in. You can’t issue the same OAuth token each time the user logs in (say, by doing an SHA-256 hash of the user’s ID plus a salt token), because that subjects your system to a replay attack.

It’s not to suggest outside of the authentication subsystem your interface shouldn’t be stateless.

But never allow the perfect to get in the way of the good–because there are unintended consequences.

And for God’s sake, don’t send the user an encrypted data structure which contains the access control entries they have access to! It’s just a matter of time before someone figures out how to decode that data structure, change the ACEs, and become a superuser.

Instead, admit your RESTful interface is not completely stateless–you have to manage access control lists as state–and move on.


Given all this, your app can be far more than basically a pretty presentation layer on a series of calls which return the contents of each app page as XML. There are times when it is appropriate, for example, to put some business logic in your front-end–but only to reduce the number of round-trips to the back end.

For example, if you have a page that accepts a credit card, you don’t need to do a network call just to see the user forgot to enter a date, a credit card number that passes the credit card number checksum, or to check if the user forgot the security code.

But this does not alleviate the need to duplicate these checks on the back end.

Ultimately, there is no such thing as a free lunch. There is no theoretical model which does a better job than existing models–just tried and proven concepts we should keep implementing, refining them only when there is an honest-to-God problem. We certainly shouldn’t be scrapping it all for the next blue-sky concept; that’s just giving into The Churn, one of the most monumental wastes of everyone’s time.

A Post For A Friend.

This is in response to someone asking for possible interfaces for building complex and/or queries.

Years ago I worked on a program called “BugLink,” and it had an interface for building complex and/or queries. The user interface worked like this:

When you first opened the application, the top of the screen had a search field that looked like this:

Image 1

The idea is simple: there is a field selector which selects a pre-defined field from a pop-down menu. There is an operator selector which picks from a list of predefined operators from a pop-down menu–and the type of operators change depending on the type of field. (For example, strings may include a “is substring of” operator that a number field does not.) And “value” can either be a pop-down for booleans (true/false), or an edit field or selector allowing you to enter a value. (If you allow dates, then tapping on the value field may bring up a calendar picker.)

Now if you clicked “Add Field”, you get a new row. But you also get a new column, associated only with the first row in the list of rows: the operator you wish to apply to the list of fields.

Image 2

The idea here is that now, you can pick how you want the rows to work: do you want all of them to be true before the query works? Or do you want any of them to be true? And or Or.

As before you can set the value of each of the rows–the field, operator and value. And fields can repeat; that allows you to search in a date range.

You can of course add more rows:

Image 3

And you can also selet rows:

Image 3S

Now here’s where the magic happens. You can also “group” and “ungroup” rows. When you do this, the selected rows indent to the right, and a new boolean operator appears:

Image 5

This allows you to create complex queries.

Naturally you need to be able to handle a bunch of fringe cases. For example, you need to decide how to handle grouping at multiple levels. (One possibility is to simply disable group and ungroup. Another is to pick an item–the first selected item or the deepest item–and move everything else into a group under that item, regardless of position in the tree.)

And you need to be able to handle other fringe cases that seem odd at first, such as having the same operator at different group levels:

Image 6

and having only one item in a group:

Image 4

But aside from handling the fringe case of grouping and ungrouping items that may be scattered across the query, there really are no fringe cases that can occur.

Further, I liked this user interface because it progressively reveals itself. The simplest query has no boolean operator. Adding a new row gives you a new option. And grouping and ungrouping reveals further complexity. (Of course you need a way to signal that queries can be grouped or ungrouped–one could handle this by showing a button that allows grouping or ungrouping. You can also explain there is a right-click pop-over menu that will show up–but I’ve never cared for pop-over menus because there is often nothing that suggests to the user that one could show up.)

But this was a fairly good interface for handling building complex boolean queries–and while it lacks a ‘not’ operator, one could theoretically add that to the operator pop-ups with a little extra work and a little extra consideration about how the user interface is to work. (For example, you could add “not” to the boolean popup if there is only one row in the group–or add a ‘not and’ and ‘not or’ if there are two or more items in that group.)

OCTools Update

I’ve taken the liberty to make a number of changes to the OCTools library to prepare for a first 1.0 release. Amongst other things I’ve updated the documentation, I’ve built sample parsers in Objective C and C++, and I’ve added support for generating Swift, along with an example Swift parser.

I’ve also taken the liberty to produce an installation package, which can be downloaded to install the tools in /usr/local/bin.

The GitHub library can be found here. Full documentation is here, and the algorithms are described here.

Don’t reuse the same buffer to pass parameters inside a loop.

So here’s a mistake I made with the Metal API.

Suppose you have a loop where you’re constructing multiple encoders, one encoder per loop.

And you need to pass a parameter–say, an integer–into each encoder.

So you write the following:

id<MTLBuffer> buffer = [self.device newBufferWithLength:sizeof(uint16_t) options:MTLResourceOptionCPUCacheModeDefault];

for (uint16_t i = 0; i < 5; ++i) {
    id<MTLComputeCommandEncoder> compute = [buffer computeCommandEncoder];
    ... blah blah blah ...
    memmove(buffer.contents, &i, sizeof(i));
    [compute setBuffer:buffer offset:0 atIndex:MyKernelIndex];
    ... blah blah blah ...
    [compute dispatchThread...];
    [compute endEncoding];
}

If you run this, I discovered that all five invocations of the kernel will result in the two-byte value at MyKernelIndex to be set to 4–the last value seen in i as we loop.

Why?

Because the same buffer is reused across all five invocations, and because the Metal code isn’t executed until after the entire buffer is committed–the last value passed in is the value that will be used across all invocations.

But if this is replaced with:

for (uint16_t i = 0; i < 5; ++i) {
    id<MTLComputeCommandEncoder> compute = [buffer computeCommandEncoder];
    ... blah blah blah ...
    [compute setBytes:&i length:sizeof(i) atIndex:MyKernelIndex];
    ... blah blah blah ...
    [compute dispatchThread...];
    [compute endEncoding];
}

Each invocation gets a unique value for i.


Just something to watch out for in a deferred execution model.

Final Metal Introduction Document

The best way to learn something is to try to explain it to someone else. So I wrote a document as a PDF file and a collection of examples using the Metal API.

Here’s the final document. Hopefully people will find it of use.
MetalIntro.pdf

Sample code can be found on GitHub.

And of course, like all examples, this one starts with… a blank screen.

ScreenShot

And ends with a slightly more complex demonstration:

ScreenShot

Updated “Metal Introduction” Document.

I’ve added a section on showing the processes in Metal for implementing Constructive Solid Geometry on the fly using the algorithms outlined in the paper An improved z-buffer CSG rendering algorithm, with example code uploaded to GitHub.

My goal is to eventually turn this into a CSG library for Metal.

The updated paper (with the additional section) can be downloaded from here: Metal: An Introduction, and updates the paper from my prior post.

And when you put it all together you should get:

ScreenShot2

Learning the Metal API

So I’m in the process of learning the Metal API, which is Apple’s replacement for OpenGL ES. The principles are fairly similar, though the Metal API is much lower level.

There are several web sites devoted to Metal, but my eventual goal is to implement image-based CSG (Constructive Solid Geometry) in Metal for an update to the Kythera application.

And that requires a deeper understanding of Metal than most introductions which seem to stop at drawing an object on the screen and perhaps adding a texture map to the object.


The best way to learn something is to try to explain it–so I’ve started writing a document showing how to build a Macintosh-based Metal API application in Objective C, and going from a blank screen to a deferred shading example.

In this case, the deferred shading example results in a rotating teapot with fairy lights and indirect illumination rendering at 60 frames/second:

Teapot Rendering Example

The sample code is uploaded at GitHub, with the different examples in their own branches.

And the first draft of the Metal Introduction Document (as a PDF with links to relevant documents) can be downloaded from here.

Feedback is appreciated.