We don’t need no stinkin’ CMS

This post is more playful than usual but it explores the question of how to achieve most of what one wants a CMS for with a few lines of Javascript and a few lines of server-side code. I’ve coined the term “Source Edit and Content Selection” for this functionality, but this was done primarily to permit me to make a pun that only viewers of a relatively unknown teen movie will understand. Let’s move on and please try to remember the safe word!

What is Wunder-SECS

Wunder-SECS is a framework that does two things:

  • Lets the user edit the placeholders marked in a page, from within the page itself, by specifying assets to fill them or by directly setting content in them,
    and also by setting style properties on elements of the page.
  • Lets the user save the finished page, which, from then on, is independent from Wunder-SECS and can be served by the server and rendered by any browser.

Wunder-SECS uses common Web 2.0 libraries and requires very little effort to work with sites built with standard HTML and CSS, either static or dynamic.

How Wunder-SECS composes pages

Marking an element to be editable (to be a placeholder) is a simple matter of adding a number of CSS classes to it. The mandatory class to add is editable. Other classes distinguish
different kinds of editing required, notably asset selection, of different kinds, or content editing. Wunder-SECS will recover all editable content in the page and apply the necessary
code to it.

Marking for Drag & Resize

Marking for Drag & resize is done with the draggable and resizeable CSS classes, which can also be used in combination.

Marking for Asset Selection

Marking for Asset Selection is done with a number of marker classes, which correspond to the asset folders where the assets reside.
The example below marks the element as a placeholder for assets of kind Flash, which also corresponds to the name of the folder. The placeholder is also draggable
and resizeable.

<h1 id="FlashID-container" class="editable _Flash draggable resizeable"/>

Wunder-SECS makes use of a JSON file called makeup.txt. Inside this file, there are directives that load assets or set content on placeholders, and
other directives that set style properties on elements of the page. This file is shared by all pages in a mini-site which are supposed to share look-and-feel.

Placeholder
An HTML element of the page that will be filled by Wunder-SECS, at page-load time. Any element can be a placeholder but, in order for the editing
facility to be used, the element needs to be marked (cf. Marking for Editing).
Asset
An HTML fragment that will be used to fill a placeholder. Can optionally contain its own inline <style> so that it be self-contained. Such interchangeable fragments
are typically created by Web designers, and need to be placed in asset folders with appropriate names.
Content
An HTML fragment (a.k.a. rich text) that was provided on-the-fly by the page editor. It is not reusable, since it cannot be used in other sites, and contains presentational
markup only.
Style properties
Any CSS property on any element that is useful to change in order to effect a change on a page. An example could be properties that need to be set to various elements in order to change
the page background in a meaningful way.

Editing the page

While entering edit mode, a splash screen is displayed briefly.

w-splash

When working in an editing session, you’ll see the Wunder-SECS toolbar in the page. The toolbar shows a select menu for the theme of the page.

w-toolbar

When you double-click on an asset placeholder, the toolbar will also show an asset select menu of the appropriate kind.

w-toolbar2

Selecting an asset from the select menu has an immediate effect.

w-asset-edit1
w-asset-edit2
w-asset-edit3

Double-clicking on a content placeholder, on the other hand, opens up the jWysiwyg editor.

w-wysiwyg

Placeholders are outlined by colored borders, sometimes two of them, to mark the placeholder and its being draggable at the same time. A resizeable placeholder has the familiar bottom-right handle, to
drag with the mouse, and also draggable bottom and right borders.

w-placeholder

The toolbar is also draggable, so you can move it around if it gets in the way.

Marking for Editing

Marking an element to be editable (to be a placeholder) is a simple matter of adding a number of CSS classes to it. The mandatory class to add is editable. Other classes distinguish
different kinds of editing required, notably asset selection, of different kinds, or content editing. Wunder-SECS will recover all editable content in the page and apply the necessary
code to it.

Marking for Drag & Resize

Marking for Drag & resize is done with the draggable and resizeable CSS classes, which can also be used in combination.

Marking for Asset Selection

Marking for Asset Selection is done with a number of marker classes, which correspond to the asset folders where the assets reside.
The example below marks the element as a placeholder for assets of kind Flash, which also corresponds to the name of the folder. The placeholder is also draggable
and resizeable.

<h1 id="FlashID-container" class="editable _Flash draggable resizeable"/>

Marking for Content Editing

Marking for Content Editing is done with the _Wysiwyg CSS class.
The example below marks the element as a placeholder for formatted text content.

<h1 id="text-container" class="editable _Wysiwyg"/>

Saving the page

When you are ready to save the finished page, you use one of the two “save” buttons.

w-toolbar3

Button “Save” saves the HTML and CSS that you have specified, while button “Save ESI” does the same thing but inside ESI markup. When not running inside a CDN that understands ESI
markup, the page is rendered essentially like before. When ESI markup is available and is interpreted by a CDN, then one can control the cache lifetime, expiration and upgrade of the individual assets inside a page through the facilities offered by the CDN.

<esi:include src="... asset URL ..." />
<esi:remove>... asset contents at time of save ...</esi:remove>

Integrating WunderSECS

To integrate WunderSECS in a page, you need to import the wundersecs.js file. It is written using the module pattern and exposes the WunderSECS global object. If you’re using Require.js or any other dependency manager, you might want to modify it to return the object instead of setting it at global scope.

It has a number of dependencies, which are listed below. If you rely on Require.js or any other dependency manager, remember to codify interdependencies within that list
correctly.

<script src="/appropriate_path/jquery-1.8.3.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.core.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.position.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.widget.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.mouse.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.draggable.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.resizable.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.ui.selectmenu.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.validate-1.9.0.min.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery-center.js" type="text/javascript"></script>
<script src="/appropriate_path/wundersecs/wundersecs.js" type="text/javascript"></script>
<script src="/appropriate_path/wysiwyg/jquery.wysiwyg.js" type="text/javascript"></script>
<script src="/appropriate_path/jquery.jeditable.mini.js" type="text/javascript"></script>
<script src="/appropriate_path/wysiwyg/jquery.jeditable.wysiwyg.js" type="text/javascript"></script>
<link href="/appropriate_path/jquery.ui.selectmenu.css" type="text/css" rel="stylesheet" />
<link href="/appropriate_path/jquery-ui-1.8.18-smoothness.css" type="text/css" rel="stylesheet" />
<link href="/appropriate_path/jquery.ui.theme.css" type="text/css" rel="stylesheet" />
<link href="/appropriate_path/wysiwyg/jquery.wysiwyg.css" type="text/css" rel="stylesheet" />

At normal rendering time, you don’t need to do anything. The page is self-contained.

If this is an editing session (which could be conveyed to the page using a parameter, or a cookie, or any other way), then you must call the initEdit function, in order
for edit functionality to be setup, and the WunderSECS toolbar to appear. You must pass the value of the “save” HTTP parameter, which is used by the “save” buttons.

WunderSECS.initEdit(saveFlag);

This is all that needs to be done at the client. At the server, right now the code expects a particular implementation of the AJAX call (WunderSecsEdit.ashx) for editing the makeup, but the specific URL used can be exposed as an initialization option, so as to be able to use it in any application.

Future enhancements might be:

  • Generalizing the background functionality to cover any combination of CSS.
  • Enabling recursive assets, where an asset might have placeholders of its own.

The code for wundersecs.js follows.

var WunderSECS = (function ($) {
    var my = {};
    var makeupData = {};

    function loadFragment(selector, url) {
        if ($(selector).hasClass('_Wysiwyg')) {
            $(selector).html(url);
        } else {
            $(selector).addClass("loading");
            $.get(url, function (data) {
                if (my.saveFlag == "esi") {
                    var esi = $('<esi:include src="' + url + '"/>');
                    $(selector).html(esi);
                    var rem = $('<esi:remove>');
                    $(rem).html(data);
                    $(selector).append(rem);
                } else {
                    $(selector).html(data);
                }
                $(selector).removeClass("loading");
            }, 'html');
        }
    }

    function startsWith(s, str) {
        return s.slice(0, str.length) == str;
    }

    function makeEditable(e) {
        var kind = '';
        if ($(e).hasClass('_Wysiwyg')) {
            var isResizeable = $(e).hasClass("ui-resizable");
            if (isResizeable) {
                $(e).resizable("destroy");
            }
            var wasResizeable;
            $(e).editable(function (val) {
                //                                var html = $.parseHTML(val);
                //                                $(html).remove('.ui-resizable-handle');
                //                                val = $('<div/>').append($(html)).html();

                $.post("/WunderSecsEdit.ashx", {
                    path: window.location.pathname,
                    key: '#' + $(e).attr('id'),
                    value: JSON.stringify(val)
                });
                if (isResizeable) {
                    makeResizeable($(e));
                }
                return val;
            }, {
                type: "wysiwyg",
                event: "dblclick",
                submit: "OK",
                cancel: "Cancel",
                onblur: "ignore"
                , onedit: function () {
                    wasResizeable = $(e).hasClass("ui-resizable");
                    if (wasResizeable) {
                        $(e).resizable("destroy");
                    }
                    return true;
                },
                onreset: function () {
                    if (wasResizeable) {
                        makeResizeable($(e));
                    }
                    return true;
                }
            });
            if (isResizeable) {
                makeResizeable($(e));
            }
            return;
        }
        if ($(e).hasClass('_Moto')) kind = 'Moto';
        if ($(e).hasClass('_TC')) kind = 'TC';
        if ($(e).hasClass('_MsisdnEntry')) kind = 'MsisdnEntry';
        if ($(e).hasClass('_Flash')) kind = 'Flash';
        if ($(e).hasClass('_Sub')) kind = 'Subtitle';
        if ($(e).hasClass('_Terms')) kind = 'SM';
        if ($(e).hasClass('_Button')) kind = 'Button';

        $(e).bind("dblclick", function () {
            $.get('/cms/assets/' + kind + '/files.txt', function (data) { //testing
                var selectlist = $.parseHTML(data);
                $('#editplaceholder').show();
                $('#editlist').html(selectlist);
                $(selectlist).selectmenu({
                    width: 200,
                    select: function (event, options) {
                        $.ajax({
                            type: "GET",
                            url: options.value,
                            success: function (value) {
                                makeupData[$(e).attr('id')] = options.value;
                                $.post("/WunderSecsEdit.ashx", {
                                    path: window.location.pathname,
                                    key: '#' + $(e).attr('id'),
                                    value: JSON.stringify(options.value)
                                });
                                var isDraggable = $(e).hasClass("ui-draggable");
                                if (isDraggable) {
                                    $(e).draggable("destroy");
                                }
                                var isResizeable = $(e).hasClass("ui-resizable");
                                if (isResizeable) {
                                    $(e).resizable("destroy");
                                }
                                $(e).html(value);
                                if (isDraggable) {
                                    makeDraggable($(e));
                                }
                                if (isResizeable) {
                                    makeResizeable($(e));
                                }
                                $('#editplaceholder').hide();
                                $('#editlist').html('');
                            }
                        });
                    }
                });
            }, 'html');
        });
    }

    function makeDraggable(e) {
        $(e).draggable({
            stop: function (event, ui) {
                var top = ui.position.top;
                var left = ui.position.left;
                //alert("Top " + top + ", left " + left);
                $.post("/WunderSecsEdit.ashx", {
                    path: window.location.pathname,
                    key: "position.#" + $(e).attr('id'),
                    value: JSON.stringify({ top: top, left: left })
                });

            }
        });
    }

    function makeResizeable(e) { //todo: look at the CSS and decide which dimensions should be resizeable
        $(e).resizable({
            stop: function (event, ui) {
                var width = ui.size.width;
                var height = ui.size.height;
                $.post("/WunderSecsEdit.ashx", {
                    path: window.location.pathname,
                    key: "size.#" + $(e).attr('id'),
                    value: JSON.stringify({ width: width, height: height })
                });

            }
        });
    }

    function makeEditableBg() {
        $.get('/cms/assets/bg/files.txt', function (data) { //testing
            var selectlist = $.parseHTML(data);
            $('#editbg').html(selectlist);
            $(selectlist).selectmenu({
                width: 200,
                select: function (event, options) {
                    var url = options.value.split('|');
                    makeupData.bg = url;
                    $.post("/WunderSecsEdit.ashx", {
                        path: window.location.pathname,
                        key: "bg",
                        value: JSON.stringify(url)
                    });
                    $('body').css("background", url[0]);
                    $('.oneColFixCtr #container').css("background", url[1]);
                    $('.oneColFixCtr #container .enterSomething.step1').css("background", url[2]);
                }
            });
        }, 'html');
    }

    function showSplash() {
        $('#wundersecs-splash').center().show().css("z-index", 1).css("-moz-box-shadow", "0 0 10px 10px #ffffff").css("-webkit-box-shadow", "0 0 10px 10px #ffffff").css("box-shadow", "0 0 10px 10px #ffffff");
        $('#wundersecs-splash').fadeIn(1000).delay(2000).fadeOut(1000);
    }

    function savePage() {
        var ready = $('.loading').length == 0;
        if (ready) {
            savePageInner();
            return;
        }
        setTimeout(function () {
            savePage();
        }, 1000);
    }

    function savePageInner() {
        var pageHtml = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml">\n' + $('html').html() + '\n</html>';
        var fullurl = window.location.href.replace(/save=[^&]*/g, '');
        var url = window.location.pathname;
        $.ajax("/WunderSecsEdit.ashx?path=" + encodeURIComponent(url), {
            type: "PUT",
            mimeType: "text/html",
            data: pageHtml,
            success: function (data, textStatus, jqXHR) {
                window.location.href = fullurl;
            }
        });
    }

    function compose() {
        $.get('makeup.txt', function (data) {
            makeupData = data;
            for (var key in data) {
                var url = data[key];
                //$(key).load(url);
                if (key == "bg") {
                    $('body').css("background", url[0]);
                    $('.oneColFixCtr #container').css("background", url[1]);
                    $('.oneColFixCtr #container .enterSomething.step1').css("background", url[2]);
                } else if (startsWith(key, "position.")) {
                    key = key.slice("position.".length, key.length);
                    $(key).css(url);
                } else if (startsWith(key, "size.")) {
                    key = key.slice("size.".length, key.length);
                    $(key).css(url);
                } else {
                    loadFragment(key, data[key]);
                }
            }
            if (my.saveFlag) savePage();
        }, 'json');
    };

    my.initEdit = function (saveFlag) {
        my.saveFlag = saveFlag;
        compose();

        if (my.saveFlag) return;

        $.get('/stat/wundersecs/wundersecs-toolbar.htm', function (data) {
            var toolbar = $.parseHTML(data);
            $('body').append(toolbar);
            $(toolbar).find('button.savebutton').bind("click", function () {
                window.location.href = window.location.href + '&save=true';
            });
            $(toolbar).find('button.saveesibutton').bind("click", function () {
                window.location.href = window.location.href + '&save=esi';
            });
            $(toolbar).draggable();
        }, 'html');
        $.get('/stat/wundersecs/wundersecs-style.htm', function (data) {
            var style = $.parseHTML(data);
            $('head').append(style);
        }, 'html');
        $.get('/stat/wundersecs/wundersecs-splash.htm', function (data) {
            var splash = $.parseHTML(data);
            $('body').append(splash);
            showSplash();
        }, 'html');

        $('.draggable').each(function (i, e) {
            makeDraggable(e);
        });
        $('.resizeable').each(function (i, e) {
            makeResizeable(e);
        });
        $('.editable').each(function (i, e) {
            makeEditable(e);
        });
        makeEditableBg();

    };

    return my;
})($);

And this might be an implementation for WunderSecsEdit.ashx, purely for demonstration. Feel free to substitute your own.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Web;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace Controller
{
    /// <summary>
    /// Summary description for WunderSecsEdit
    /// </summary>
    public class WunderSecsEdit : IHttpHandler
    {

        public void ProcessRequest(HttpContext context)
        {
            var path = context.Request["path"];
            if (context.Request.HttpMethod == "POST" && path!=null)
            {
                var key = context.Request["key"];
                var value = context.Request["value"];
                if (key != null && value != null)
                {
                    var filepath = path.Substring(0, path.LastIndexOf('/')) + "/makeup.txt";
                    var mappedPath = context.Server.MapPath(filepath);
                    var jsonSerializer = new JsonSerializer();
                    var streamReader = new StreamReader(mappedPath);
                    var makeup = jsonSerializer.Deserialize(new JsonTextReader(streamReader));
                    streamReader.Close();
                    if (makeup is JObject)
                    {
                        var jObject = (JObject) makeup;
                        var jvalue = jsonSerializer.Deserialize(new JsonTextReader(new StringReader(value)));
                        if (jvalue is JToken)
                        {
                            jObject[key] = (JToken) jvalue;
                            var streamWriter = new StreamWriter(mappedPath);
                            jsonSerializer.Serialize(new JsonTextWriter(streamWriter), jObject);
                            streamWriter.Close();
                            //context.Response.ContentType = "text/plain";
                            //context.Response.Write("Hello World");
                        }
                        if (jvalue is string)
                        {
                            jObject[key] = (string) jvalue;
                            var streamWriter = new StreamWriter(mappedPath);
                            jsonSerializer.Serialize(new JsonTextWriter(streamWriter), jObject);
                            streamWriter.Close();
                            //context.Response.ContentType = "text/plain";
                            //context.Response.Write("Hello World");
                        }
                    }
                }
            }
            else if (path != null && context.Request.HttpMethod == "PUT")
            {
                context.Request.InputStream.Position = 0;
                var reader = new StreamReader(context.Request.InputStream);
                string payload = reader.ReadToEnd();
                var mappedPath = context.Server.MapPath(path);
                using (var streamWriter = new StreamWriter(mappedPath))
                {
                    streamWriter.Write(payload);
                    streamWriter.Close();
                }
            }
        }

        public bool IsReusable
        {
            get
            {
                return false;
            }
        }
    }
}
Advertisements

Lamino, or What the Heck Do I Know About Dependency Injection

Recently, someone asked me if I knew what Dependency Injection was. Well, Dependency Injection and I go way back, I have to say. Or Inversion Of Control. Or whatever it was called at the time, I can’t remember. The time, however, was shortly after the turn of the centrury. Enter nostalgic guitar, please.

I was working at Velti, building a system for SMS interactions. Not content with the state of configuration management, I had developed a rudimentary facility to instantiate objects and set their properties. Today, you’d call it a DI/IoC container. Because reinventing the wheel is always fun, but not always productive, as we were growing this facility I was looking around for other similar stuff and for people with similar ideas. There weren’t any, except for Rod Johnson and his book “Expert one-on-one J2EE design and development” describing something called Interface21. But it was enough. The approach Rod was describing was very much to my liking. I immediately switched the XML grammar to that of what came to be known as Spring and adopted most of its concepts.

However, I did not switch to Spring, because it lacked two things, both of which we had come to rely upon.

  • One of them was the ability to separate the DI part from the instantiation part. We were using XmlWeb2, which I might blog about someday, which was instantiating MVC actions and views using a reloading classloader, so we needed these two phases to be separate. (BTW, it’s very easy to write a reloading classloader in Java, once you realize the following: a classloader can never reload a class!).
  • The other was a mechanism to build configuration in layers. It it mostly the latter that the rest of this post will be occupied with. Or rather its descendant, which is Lamino.

Lamino

The blurb: Lamino is a high-level configuration metalanguage that enhances basic IoC (Inversion of Control a.k.a. Dependency Injection) with configuration layering and encapsulated components. It is hosted by an IoC language and it “compiles” to it. Using configuration layering, the application developer can create editions or extensions by expressing the differences of each configuration layer from the layer below it. Using encapsulated components, the application developer can define reusable configuration chunks that import and export specific services, while hiding internal contents.

You should understand that, in my puritan mind, there is no program structure outside of the configuration. The whole object graph of the singletons that comprise the logic layer, should be there. That object graph and the collaborations it describes is the application. Configuring a different edition or deployment can involve a lot more than modifying some properties. A whole portion of the object graph could need be redefined. That’s why relying on a separate layer of property files, or something, can only be half a solution. And doing a different build of a system just to change how some classes collaborate, is not a solution at all.

I will not replicate the information in the project Web site here. But I will show you what Lamino does by walking you through the examples in the distribution, which is not covered by the site. They are the same for all three containers covered (you’d better get the latest files from SVN (here, the DotNet distribution), by the way, because the WebContextHandler for .Net is only to be found there). WordPress insists on mangling large XML snippets containing “object” elements, so you’ll have to view the files from there.

Example

To cut a long story short the resulting system can be found in Derived-derived.xml, which is the actual output of Lamino. To make the point that Lamino is purely transformation and did not need a full-blown programming language to implement it, it is written in XSLT (which, BTW, is the first mainstream functional language).

Modify an object with extends=”extends”

Let’s have a look at object “ObjectToBeExtendedFromMoreBase”, keeping in mind that element “x-object” is really “object” but I had to rename it to keep WordPress from slurping it.

This object is defined in MoreBase.xml:

	<x-object id="ObjectToBeExtendedFromMoreBase">
		<property name="PropertyToBeCopiedAsIs" value="foo"/>
		<property name="PropertyToBeReplaced" value="value to be replaced"/>
		<property name="PropertyToBeRemoved" value="foo"/>
		<property name="PropertyToBeAppendedTo">
			<list>
				<value>val1</value>
				<value>val2</value>
			</list>
		</property>
		<property name="PropertyToBePrependedTo">
			<list>
				<value>val1</value>
				<value>val2</value>
			</list>
		</property>
	</x-object>

And extended in Base.xml using extends=”extends”.

	<x-object id="ObjectToBeExtendedFromMoreBase" extends="extends">
		<property name="NewProperty" value="foo2"/>
		<property name="PropertyToBeReplaced" combine="replace" value="new value"/>
		<property name="PropertyToBeRemoved" combine="remove"/>
		<property name="PropertyToBeAppendedTo" combine="append">
			<list>
				<value>newval1</value>
				<value>newval2</value>
			</list>
		</property>
		<property name="PropertyToBePrependedTo" combine="prepend">
			<list>
				<value>newval1</value>
				<value>newval2</value>
			</list>
		</property>
	</x-object>

The resulting configuration is the following. Note that I couldn’t coerce either MSXML or Xalan to produce whitespace to my liking, so no whitespace remains in the output. Lamino annotates the output with some comments to make it easier to trace the output.

<!--Extended-->
<x-object id="ObjectToBeExtendedFromMoreBase" scope="application" visibility="public" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="NewProperty" value="foo2"></property>
<!--Replaced-->
<property name="PropertyToBeReplaced" value="new value"></property>
<!--Property PropertyToBeRemoved was removed-->
<property name="PropertyToBeAppendedTo">
<list>
<!--Original elements-->
<value>val1</value>
<value>val2</value>
<!--Appended elements-->
<value>newval1</value>
<value>newval2</value>
</list>
</property>
<property name="PropertyToBePrependedTo">
<list>
<!--Prepended elements-->
<value>newval1</value>
<value>newval2</value>
<!--Original elements-->
<value>val1</value>
<value>val2</value>
</list>
</property>
<property name="PropertyToBeCopiedAsIs" value="foo"></property>
</x-object>

Property “NewProperty” is added in Base.xml.

Property “PropertyToBeReplaced” starts off with value=”value to be replaced”, but this value is overriden using combine=”replace” in Base.xml with value=”new value”, which is the value that remains in the end.

Property “PropertyToBeRemoved” is removed using combine=”remove” in Base.xml.

Property “PropertyToBeAppendedTo” contains two elements to begin with, and two more are added in Base.xml using combine=”append”. A similar modification is done to property “PropertyToBePrependedTo”, where two elements are prepended using combine=”prepend” in Base.xml.

Finally, property “PropertyToBeCopiedAsIs” is not modified at all.

Replace an object with extends=”overrides”

Let’s focus now on object “ObjectToBeReplacedFromMoreBase”.

	<x-object id="ObjectToBeReplacedFromMoreBase">
		<property name="Property2" value="bar to be replaced"/>
	</x-object>

Things are simpler in this case, as this object is overriden in Base.xml.

	<x-object id="ObjectToBeReplacedFromMoreBase" extends="overrides">
		<property name="Property2" value="new value"/>
	</x-object>

The resulting object is, indeed, what Base.xml prescribes.

<!--Overriden-->
<x-object id="ObjectToBeReplacedFromMoreBase" scope="application" visibility="public" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property2" value="new value"></property>
</x-object>

Importing components

Now, let’s look how one can define DI components. Derived.xml imports the same component twice.

	<import resource="Component.xml"/>
	<import resource="Component.xml">
		<hiding object="OtherPublicComponentToBeHidden"/>
		<renaming object="ExplicitlyPublicComponent" as="Compo1"/>
		<renaming object="ImplicitlyPublicComponent" as="Compo2"/>
		<providing object="NewObject" as="External"/>
	</import>

Note that the component itself employs configuration inheritance in its definition.

ComponentBase.xml:

<x-objects xmlns="http://www.springframework.net"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.net
lamino-objects.xsd">
	<x-object id="ExplicitlyPublicComponent" visibility="public">
		<property name="Property3" value="foobar original"/>
		<property name="Property4">
			<ref local="Hidden"/>
		</property>
	</x-object>
	<x-object id="ImplicitlyPublicComponent">
		<property name="Property6" value="foobaz original"/>
		<property name="Property7" ref="External"/>
	</x-object>
	<x-object id="OtherPublicComponentToBeHidden"/>
	<x-object id="Hidden" visibility="private">
		<property name="Property5" value="baz original"/>
	</x-object>
</x-objects>

Component.xml:

<x-objects xmlns="http://www.springframework.net"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.net
lamino-objects.xsd" extends="ComponentBase.xml">
	<x-object id="ExplicitlyPublicComponent" visibility="public" extends="extends">
		<property name="Property3" value="foobar" combine="replace"/>
	</x-object>
	<x-object id="Hidden" visibility="private" extends="extends">
		<property name="Property5" value="baz" combine="replace"/>
	</x-object>
</x-objects>

As you see, the first importing does not specify anything in particular, while the second importing specifies the following.

  • Object “OtherPublicComponentToBeHidden” is to be hidden.
  • Objects “ExplicitlyPublicComponent” and “ImplicitlyPublicComponent” are to be renamed.
  • Object “NewObject” is to replace references to object “External”.

The effects of these (and the comparison to the unmodified component import) is seen in the output.

<x-object id="ExplicitlyPublicComponent" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property3" value="foobar"></property>
<property name="Property4">
<ref local="Hidden-IDAQJLVIDASJLV"></ref>
</property>
</x-object>
<x-object id="ImplicitlyPublicComponent" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property6" value="foobaz original"></property>
<property name="Property7" ref="External"></property>
</x-object>
<x-object id="OtherPublicComponentToBeHidden" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default"></object>
<x-object id="Hidden-IDAQJLVIDASJLV" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property5" value="baz"></property>
</x-object>
<x-object id="Compo1" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property3" value="foobar"></property>
<property name="Property4">
<ref local="Hidden-IDAKJLVIDASJLV"></ref>
</property>
</x-object>
<x-object id="Compo2" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property6" value="foobaz original"></property>
<property name="Property7" ref="NewObject"></property>
</x-object>
<x-object id="OtherPublicComponentToBeHidden-IDAXDEUIDAYDEU" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default"></object>
<x-object id="Hidden-IDAKJLVIDASJLV" scope="application" dependency-check="default" singleton="true" abstract="false" lazy-init="default" autowire="default">
<property name="Property5" value="baz"></property>
</x-object>

Note that the “visibility” attribute, meant to be used inside components, specifies whether the object will be publicly visible in the output. Hiding is done via a simple renaming of IDs (here, to “Hidden-IDAQJLVIDASJLV” and “Hidden-IDAKJLVIDASJLV”).

Importing
The component expects to find a reference to object “External”. The first importing does not modify this specification and resolution to that reference is done using object “External” defined in Derived.xml. Note that Lamino does not try to verify the validity or even the syntax of the configuration. If object “External” was a broken reference, it would stay a broken reference.

The second importing modifies the reference to demand an object named “NewObject”, also defined in Derived.xml. Same caveat as before regarding validity of references.

Exporting and Hiding
The component exports all objects that are explicitly or implicitly public. These are objects “ExplicitlyPublicComponent”, “ImplicitlyPublicComponent” and “OtherPublicComponentToBeHidden”, exporting with these names by the first, unmodified, importing. The second importing renames the first two objects and hides the third one. Had it not hidden it, the resulting configuration would contain two objects with the same ID, which would be… inconvenient but, as I have said, Lamino does not check your Spring.Net grammar and syntax (same for Spring for Java and Castle Windsor).

Embedding in Web.config

As my latest jobs where in .Net shops, I have paid particular attention to the Spring.Net distribution. In the latest code to be found in SVN, there is a WebContextHandler to help integrate Lamino in a Web application, in a context to be used instead of the one Spring.Net provides.

using System;
using System.Collections.Generic;
using System.Xml.Xsl;
using System.Linq;
using System.Web;
using Spring.Context;
using WCH=Spring.Context.Support.WebContextHandler;

namespace Lamino.Spring
{
    public class WebContextHandler : WCH 
    {
        protected override IApplicationContext InstantiateContext(IApplicationContext parent, object configContext, string contextName, Type contextType, bool caseSensitive, string[] resources)
        {
            var newResources = LaminoExtend(resources);
            return base.InstantiateContext(parent, configContext, contextName, contextType, caseSensitive, newResources);
        }

        protected string[] LaminoExtend(string[] resources)
        {
            var httpServerUtility = HttpContext.Current.Server;
            XslCompiledTransform transform = new XslCompiledTransform();
            XsltSettings xsltSettings = new XsltSettings();
            xsltSettings.EnableDocumentFunction = true;
            transform.Load(httpServerUtility.MapPath("~/lamino-objects.xslt"),xsltSettings,null);
            var newResources = (string[])resources.Clone();
            for (int i = 0; i < newResources.Length; i++)
            {
                var finalResource = newResources[i] + ".Lamino";
                var resourceFile = httpServerUtility.MapPath(resources[i]);
                var finalResourceFile = httpServerUtility.MapPath(finalResource);
                if (!System.IO.File.Exists(finalResourceFile) || System.IO.File.GetLastWriteTime(finalResourceFile).CompareTo(System.IO.File.GetLastWriteTime(resourceFile)) < 0)
                {
                    transform.Transform(resourceFile, finalResourceFile);
                }
                newResources[i] = finalResource;
            }
            return newResources;
        }
    }
}

You can be up and running with layered and componentized DI in no time!

Configuration layering

I want to see configuration layering become standard practice. It seems to me that any program meant to be deployed in various editions and deployments can benefit from having to specify just the delta from the base. The layers can be, for example: the core, interface-less part of the application, the different embodiments of the application in specific user interfaces and particular deployments. I would be thrilled if Spring and Spring.Net implemented these concepts themselves but, until they do, I offer Lamino as a lightweight solution to that need but, most of all, I intend it to demonstrate the need itself by showing you how easy it can be.

Fundamentum

Let me start by saying that ASP.NET MVC is terrific. Before ASP.NET MVC, one had to use ASP.NET, which I hope the universe will soon forget, so moving to ASP.NET MVC is like using a tablet instead of a clay tablet (except I am overly generous towards ASP.NET with the analogy). Really. There is no reason not to use ASP.NET MVC for your Web applications – if you enter a time machine and land ten years ago, when Struts on Java was already old news.

We live in 2013 now, folks, and the Web is moving forward. So, what is a dot-net Web programmer supposed to do? This post is a particular answer to that question I gave myself a few moons ago. Note that it is a particular, idiosyncratic even, answer and is meant to implement CRUD UIs, not any Web app in general. My answer was the Fundamentum framework, the code of which I will not open-source at this time.

Fundamentum assumes your Web application is built according to the client-side MVC or MVP pattern. It currently uses Backbone.js as its underlying Javascript framework, and offers a Backbone View and a Backbone Model to extend but it could be adapted to any other similar framework. Templating is done with the Handlebars.js library.

The application has no “dirty” state after each edit, meaning, there are never data that must be “submitted” to the server with an explicit later action (although it can have “stale” data, if other users are editing the same entities).

ASP MVC 4 is currently used to provide navigation, authentication, authorization and basic layout, while the content-centric parts of each page are rendered via Backbone views. It relies on server-side support for its Ajax calls, and the current implementation contains C# utilities and base classes to use to implement ASP MVC Controllers with the needed functionality.

Declarative annotations for editing

Fundamentum allows the declarative annotation of editable content in the page, whether in the “static” or “templated” parts of the markup. Annotation is done using special classes. Later on, a full grammar for the annotations will be presented. As an example, the following annotation instructs Fundamentum to create in-place editing functionality for whatever markup is generated by the Carrier partial template, and informs Fundamentum to create specifically a “select” list for the Carrier property of the model, and use the Carrier partial template to render each “option” in the list.

<span class="edit attr:Carrier select tmpl:Carrier">{{#with Carrier}} {{> Carrier }} {{/with}}</span>

Any property in the model can be edited, however deep in the object graph. Model properties are actually specified using Spring Expressions. An example of a more complicated expression is the following, which reaches inside the PurchasePlanPayment collection to find a particular element and refers to its ChargeUnit property.

PurchasePlanPayments.^{PurchasePlanPaymentID==new System.Guid('{{ PurchasePlanPaymentID }}')}.ChargeUnit

Fundamentum allows editing of single attributes, in preference to the usual Backbone practice of updating the whole model at once. It actually supplements the default Backbone Ajax API
(the set of calls that Backbone does by default and expects the server to understand) with a new one, which updates a single property and is caused by the saveAttribute(attributeName, value, callback) model method.

PUT model-url-root / model-id / property-name HTTP/1.1
... HTTP Headers ...

JSON rendering of the new value

It also supplements the default Backbone API with a call to return available options in the “select” list for a property, like in the initial example.

GET model-url-root /OPTIONS/ model-id / property-name HTTP/1.1
... HTTP Headers ...

A third addition is the support for cloning, using the following call, caused by the createClone() model method.

POST model-url-root  / model-id /Clone HTTP/1.1
... HTTP Headers ...

Finally, there is support for adding a new element in a collection property with the following call, caused by the pushAttribute(attributeName, value, callback) model nethod.

POST model-url-root / model-id / property-name HTTP/1.1
... HTTP Headers ...

Server-side support for responding to these calls will be presented shortly. You can read about standard Backbone model methods and associated HTTP calls at Backbone Tutorials.

Edit Annotation Grammar

The essential annotation for editable content is class edit. On its own, it invokes a simple, in-line editor, as in the following.

<span class="edit attr:SubscriptionPlan.DurationNumberofUnits">{{ SubscriptionPlan.DurationNumberofUnits}}</span>

Class edit must appear together with a class which is the property expression preceded by attr: (NB. the colon character). Since the whole must be a valid class name, and will also become part of a URL in the Ajax call, the property expression must be suitably URL-escaped, as in the example of a complicated expression above, where the space character inside the expression new Guid was replaced by %20.

Class select, as we’ve seen, invokes a “select” list containing available options fetched from the server. In most cases, this must appear together with a class which is the token tmpl: (NB. the colon character) suffixed by the name of the Handlebars partial template to use in order to render each option (which is returned as a JSON object). The latter is usually the same template used to render the editable content in the first place, unless a different presentation is needed. The tmpl: class is not needed if the OPTIONS call is overriden to return a character string, which will be taken as an already formatted option HTML text.

Class datepicker invokes a date-time picker.

Class checkbox invokes a checkbox control. In the application using Fundamentum, which uses a Metro-style, tile-based interface, the checkbox control has been overriden with an actionable tile activity flag.

Finally, class wysiwyg invokes a rich-text editor (jWysiwyg).

All in-place editing is done using jEditable, with some custom additions.

Fundamentum Controllers

A Fundamentum Controller (ASP MVC Controller) is needed per basic entity, where a basic entity is any entity which is at the root of an object graph that we deem necessary to manipulate as a whole. Except for those classes which happen to be basic entities themselves, and will end up with a Controller each, any classes that represent “value entities”, which don’t have an identity separate from the basic entity from which they stem, don’t (and shouldn’t) have a Controller dedicated to them.

I found AttributeRouting a very useful notation for HTTP routes, and it is used exclusively in all Fundamentum Controllers.

In fact, I have created a special attribute, FundamentumRouteConventionAttribute, inheriting from RouteConventionAttributeBase, that annotates controllers to give them convention-based routes (ignore the parts about tags, as I’ll not elaborate on them in this post).

        private static readonly List<FundamentumRouteConventionInfo> Conventions = new List<FundamentumRouteConventionInfo>
        {
            new FundamentumRouteConventionInfo("Index", "GET", ""),
            new FundamentumRouteConventionInfo("Get", "GET", "{id}"),
            new FundamentumRouteConventionInfo("Delete", "DELETE", "{id}"),
            new FundamentumRouteConventionInfo("Update", "PUT", "{id}"),
            new FundamentumRouteConventionInfo("Create", "POST", ""),
            new FundamentumRouteConventionInfo("Clone", "POST", "{id}/Clone",9),
            new FundamentumRouteConventionInfo("SetAttributeValue", "PUT", "{id}/{AttributeName}"),
            new FundamentumRouteConventionInfo("AttributeCollectionAddNew", "POST", "{id}/{AttributeName}"),
            new FundamentumRouteConventionInfo("AttributeOptions", "GET", "OPTIONS/{id}/{AttributeName}"),
            new FundamentumRouteConventionInfo("GetTagsAvailable", "GET", "TAGS/ALL",9),
            new FundamentumRouteConventionInfo("GetTags", "GET", "TAGS/{id}"),
            new FundamentumRouteConventionInfo("RemoveTag", "DELETE", "TAGS/{id}/{Tag}"),
            new FundamentumRouteConventionInfo("AddTag", "PUT", "TAGS/{id}/{Tag}")
        };

All Fundamentum Controllers inherit from GenericEditController, which offers basic functionality for implementing Backbone and Fundamentum calls. The GenericEditController is parameterized by the class of the entity and the class of its key. We also use what AttributeRouting calls Route Conventions to create routes automatically for methods, whether inherited from the GenericEditController or implemented in a Controller itself.

public abstract class GenericEditController<TKey, TEntity> : Controller where TEntity : class
{
    public virtual JsonResult Index() { ... }
    public virtual JsonResult Get(TKey id) { ... }
    public virtual JsonResult Delete(TKey id) { ... }
    public virtual JsonResult Update(TKey id, TEntity newEntity) { ... }
    public virtual JsonResult AttributeOptions(TKey id, string attributeName) { ... }
    public virtual JsonResult SetAttributeValue(TKey id, string attributeName) { ... }
    public virtual Tuple<object,object> SetAttributeValueHelper(string attributeName, object value, TEntity obj) { ... } // Used by SetAttributeValue but overrideable on its own, as well
    public virtual JsonResult AttributeCollectionAddNew(TKey id, string attributeName) { ... }
    public virtual T ReadStreamValue<T>() { ... }
    public abstract IRepository GetRepository();
}

Index

This is a convention-based method that responds to the GET /url-root/ call, that returns all entities.

Get

This is a convention-based method that responds to the GET /url-root/id call, that returns a specific entity.

Delete

This is a convention-based method that responds to the DELETE /url-root/id call, that deletes a specific entity.

Update

This is a convention-based method that responds to the PUT /url-root/id call, that updates a specific entity wholesale.

AttributeOptions

This is a convention-based method to respond to the OPTIONS call. It handles two cases.

  • When the property mentioned is a domain entity, in which case is fetches a list of available entities from the Repository.
  • When the property mentioned is an Enum, in which case it returns the list of available values. Integer properties which correspond to Enums are annotated with the UnderlyingEnumAttribute, as in the following example.
    [UnderlyingEnum(typeof(SpGw.Enums.PurchasePlanPaymentType))]
    public virtual int PurchasePlanPaymentTypeID { get; set; }

The Controller can inherit from a number of instantiations of the following interface.

public interface IQueryableRestrictor<T, TEntity>
{
    Func<IQueryable<T>, IQueryable<T>> Restrict(TEntity baseObject);
}

In that case, the appropriate Restrict method will be called to specialize the IQueryable computed by the Repository for the specific base entity involved in the call.

The call returns a JSON object of the form { "id1" : { ...JSON rendering... }, "id2" : { ...JSON rendering... }, ... }. Each entity can be called upon to return its own id by virtue of it implementing the Identifiable interface. This might be lifted in later versions, and the responsibility passed to the Repository directly.

SetAttributeValue

This helper method responds to the setAttributeValue model method, converting as needed. It basically calls SetAttributeValueHelper to do that, so it is the latter method that one should override to perform custom conversions etc.

AttributeCollectionAddNew

This helper method responds to the pushAttributeValue model method. The Controller can inherit from a number of instantiations of the following interface.

public interface IConstructorSpecializer&amp;lt;T, TEntity&amp;gt; where TEntity: class where T:class
{
    T Construct(TEntity baseObject);
}

In that case, the appropriate Construct method will be called to create the new collection element for the specific base entity involved in the call.

GetRepository

This method returns an IRepository, which abstracts away the data layer and provides many helpful generic methods which are called by reflection to make the magick happen. It will be
exposed in a later section.

ReadStreamValue

It is simply a helper method to call JsonHelper.ReadJsonValue<T> on the request stream.

The Repository

The Repository represents the data layer of the application. I had very little to do with its innards (George was responsible for it and I hope he’ll blog about it himself) but I present it here because it’s required by Fundamentum. It enables the GenericEditController and any application Controller to perform data operations. The Repository implements interface IRepository.

    public interface IRepository
    {
        T Load<T>(object id) where T : class; //gets an object without trip to the database
        T Get<T>(object id) where T : class;
        T SaveOrUpdate<T>(T entity) where T : class;
        void Delete<T>(T entity) where T : class;
        IQueryable<T> GetAll<T>() where T : class;
        PropertyInfo FindPrimaryKeyProperty<T>() where T : class;
        T CreateEntity<T>(object id, PropertyInfo property)  where T : class;
        object ConvertedPrimaryKey<T>(object id) where T : class;
    }

It also has the following helper method, as described in the section about property options.

    List<T> ToListRestrictable<T, TEntity>(object r, TEntity baseObject) where T : class { ... }

Currently, the application using Fundamentum implements two different data layers, one with NHibernate and one with EntityFramework, to evaluate the pros and cons of each in order to arrive at the best of them.

Don’t drink the Kool-Aid

I know it’s a beverage popular with developers working with Microsoft technologies. Don’t be afraid to give your own answers and, particularly, don’t be afraid to ask your own questions. You don’t have to believe anything just because an authority says so. Remember the dictum by Isaac Newton, “If I have seen further, it is by standing on the shoulders of giants”? Don’t do the converse, epitomized in the motto in Ketil Malde’s signature:

“If I haven’t seen further, it is by standing in the footprints of giants”.

SQL: The data model behind MongoDB

You know that MongoDB is a modern NoSQL database, meaning, among other things, that it is schemaless and its data model is more complex than what SQL can handle. However, in this post I will attempt to show you how SQL is sufficient to serve as a data model behind it. Hard to swallow? Follow me as I go through a list of misconceptions I need to debunk and then as I make my case.

Misconception #1. Let’s consider the claim that NoSQL is schemaless. Taking MongoDB as an example, clearly a database that can answer the following, is hardly as amorphous as this word implies.

db.somecollection.find( { foo: "foo" }, { bar: 1} )

This query expects all queried documents (not all documents in the collection, though) to have a “foo” field, which (some times) takes a string value, and a “bar” field (of whatever type). The net sum of all query/update code touching the database contributes to a definition of what data are supposed to be there. The data themselves, on the other hand, have a specific form, which does or does not correspond to this definition. In any case, this definition is a “schema”, by any other name. Consequently, it is more truthful (and some NoSQL material adheres to this truth) to talk about having flexible or dynamic schemas, instead of being schemaless. And keep in mind that the only flexibility is in extending the schema, as no facility exists for any other modification to it.

Misconception #2. Everyone knows that SQL represents “the relational model”. It might do, depending on what you mean by “SQL”. But you can’t possibly mean SQL99, whose data model explicitly encompasses “groups”, “repeating groups” and “nested repeating groups”, i.e. tree data. SQL99 transcends the flat relational model (in other ways, also, which I won’t go into in this post) so, confusing SQL as a standard with what vendors sell in the form of SQL is a classic “straw man” argument.

Nested relational models. Nested relational models can be queried using nested relational algebra. There is nothing extraordinary about nested relational algebra or its corresponding SQL. Its only difference with flat relational algebra is that attributes can be relations and, everywhere attributes can appear (in projection and selection), algebraic expressions can be used.

I’m going to use the example from the MongoDB documentation, which is “a collection named bios that contains documents with the following prototype” (note that it’s not a schema!).

{
  "_id" : 1,
  "name" : {
             "first" : "John",
             "last" :"Backus"
           },
  "birth" : ISODate("1924-12-03T05:00:00Z"),
  "death" : ISODate("2007-03-17T04:00:00Z"),
  "contribs" : [ "Fortran", "ALGOL", "Backus-Naur Form", "FP" ],
  "awards" : [
              {
                "award" : "W.W. McDowellAward",
                "year" : 1967,
                "by" : "IEEE Computer Society"
              },
              {
                "award" : "National Medal of Science",
                "year" : 1975,
                "by" : "National Science Foundation"
              },
              {
                "award" : "Turing Award",
                "year" : 1977,
                "by" : "ACM"
              },
              {
                "award" : "Draper Prize",
                "year" : 1993,
                "by" : "National Academy of Engineering"
              }
  ]
}

A query to return each document with awards from 1990 onwards would be the following:

select 
  _id, 
  name, 
  birth, 
  death, 
  contribs, 
  (select * from awards where year >= 1990) as recentawards
from bios;

A query to return only documents having awards from 1990 onwards would be the following:

select * from bios
where exists (select * from awards where year >= 1990);

It looks like normal SQL, doesn’t it? Unfortunately, it is not sufficient for everything you might want to do.

Shuffling information around. Consider what it would take to collect all distinct contribution names from all bios. Something more is needed to lift information from inside bios.contribs. This operation is called Unnest (μ) and its converse is Nest (ν). Unnesting contribs on a projection of bios that contains just _id and contribs, results in the following relation.

{
  "_id" : 1,
  "contribution" : "Fortran"
}
{
  "_id" : 1,
  "contribution" : "ALGOL"
}
{
  "_id" : 1,
  "contribution" : "Backus-Naur Form"
}
{
  "_id" : 1,
  "contribution" : "FP"
}

Note that the operation of Unnest can be achieved with a join like Sql Server’s CROSS APPLY, where the right joined table expression is evaluated in the context of each row of the left table, like in the following unnesting of awards.

select
  _id, 
  name, 
  birth, 
  death, 
  contribs, 
  a.award,
  a.year,
  a.by
from bios cross apply bios.awards a

Nesting _id is an operation akin to what GROUP BY does before aggregates are computed.

{
  "ids" : [ 1 ],
  "contribution" : "Fortran"
}
{
  "ids" : [ 1 ],
  "contribution" : "ALGOL"
}
{
  "ids" : [ 1 ],
  "contribution" : "Backus-Naur Form"
}
{
  "ids" : [ 1 ],
  "contribution" : "FP"
}

In fact, the operation of GROUP BY can be defined within this model: Nest using the grouping columns, producing a new relation attribute e.g. TheGroup; compute the aggregates on relation TheGroup, producing a single-row relation attribute e.g. TheAggregates; Unnest TheAggregates, producing a flat relation again.

Verso. There is also another viewpoint I found in a data model called Verso, where restructuring is a powerful operation in its own right. Some restructurings can be shown to preserve all information, others to lose some, but it’s a powerful operation that, combined with the rest of nested relational algebra, can express queries that are very cumbersome in flat relational algebra.

The transformation I presented earlier on a projection of bios that contains just _id and contribs would have been a single application of restructuring using the target schema (called “format” in verso). The initial “format” would be written as _id (contribution)*. Restructuring to contribution (_id)* would be done in a single step.

The data model behind MongoDB. Hierarchical databases, insofar as they hold useful data that we need to query, need a way for us to be able to express these queries. Making the conscious decision of not inherently supporting a full-blown query language and delegating this task to application code does not negate the need to be able to define the queries in a concise way. I posit that SQL, in its incarnation that targets hierarchical data, adapted to flexible and untyped schemas and by incorporating the rich amount of relevant research, is sufficiently powerful to serve as a data model for hierarchical databases, like MongoDb.