Migrating Higher Order Programming to the Web

My application, HOPE (Higher Order Programming Environment) is, in its simplest definition, a semantic publisher-subscriber application builder.  I’ve been wanting to move it to the web, but this is fraught with security concerns, hence my investigations into Docker technology and using a language like Python for the programming of “receptors” — the autonomous computational units that process semantic data.  Another stumbling block is my lack of experience with HTML5 canvas and graphics rendering.  Regardless though, there was no reason not to put together a proof of concept.

Here’s a quick walkthrough — no, this code is not yet publicly available.

Step 1: create a few receptors.

We’ll do some computations based on inputting a birth date.

Receptor #1: computing the age of a person in terms of years and days:

r1.png

Notice the class name computeAge.  We’ll talk about this later.

Receptor #2: compute the number of days to the birth day:

r2.png

Note the class name daysToBirthday.

Receptor #3: Get the interesting people born on the same month and day:

r3.png

Again, note the class name personsOfInterest.

Step 2: Add the Receptors to the Surface Membrane

r4.png

Step 3: Inject a semantic JSON object

Run the “membrane” and inject:

{"birthday": {"year": 1962, "month": 8, "day": 19}}

r5.png

And here’s the result:

r6.png

The full output being:

[
 {
 "age": {
 "years": 37,
 "days": 204
 }
 },
 {
 "daysToBirthday": 161
 },
 {
 "personsOfInterest": [
 "1724 Samuel Hood, 1st Viscount Hood, British admiral in the American Revolutionary War and the French Revolutionary Wars, born in Butleigh, England (d. 1816)",
 "1745 John Jay, American statesman, 1st US Chief Justice, born in New York City",
 "1863 Edvard Munch, Norwegian painter and print maker (The Scream), born in Ådalsbruk, Løten, Norway (d. 1944)",
 "1915 Frank Sinatra, American singer (Strangers in the Night, My Way) and actor (From Here to Eternity) known as 'old blue eyes', born in Hoboken, New Jersey (d. 1998)",
 "1932 robert pettit, American NBA star (St Louis Bombers/1959 MVP), born in Baton Rouge, Louisiana"
 ]
 }
]

What’s Going On?

Simply put, a preprocessor creates a mapping between semantic types and receptors (Python classes):

receptorMap = 
{
  'birthday':[computeAge(), personsOfInterest()],
  'age':[daysToBirthday()]
}

and the semantic processor engine routes the JSON semantic types to the Python class receptor’s process method.  When you inject “birthday”, it routes that type’s data to the computeAge and personOfInterest receptors.  The output of computeAge, which is of type “age”, is routed to the daysToBirthday receptor.

In this manner, one can create a library of small computational units (receptors) and build interesting “computational stories” by mixing and matching the desired computations.  Creating the receptors in Python makes this approach perfectly suited for running in Docker containers.  My ultimate vision is that people would start publishing interesting receptors in the open source community.

There’s still much more to go, but even as such, it’s a fun prototype to play with!  Some of the interesting problems that come out of this is, how do we let the end user create a visual interface (a UI, in other words) that facilitates both intuitive input of the semantic data as well as displaying real time output of the semantic computations.  Just the kind of challenging stuff I like!

Advertisements

MongoDB Grows Up – the $lookup aggregator

mongodbartSo far, I’ve been avoiding anything having to do with NoSQL because of the inability to do, in the classical RDBMS world, table joins.  The idea of having to pull into memory (well, my application’s memory) all the results of one table and then join them (with more code) to another table was appalling to me, particularly since I want my queries to be runtime generated by metadata schema.

However, as of the release of MongoDB 3.2, that has changed!  I can now do multiple table (collection) joins, which in my opinion, is critical for working with semantic data.  As Semag points out:

“Data is organized based on binary models of objects, usually in groups of three parts: two objects and their relationship.”

So now, in MongoDB, I can do something very simple (and very non-semantic in this example):

db.createCollection("Person")
db.createCollection("Phone")
db.createCollection("PersonPhone")

db.Person.insert({ID: 1, LastName: "Clifton", FirstName: "Marc"})
db.Person.insert({ID: 2, LastName: "Wagers", FirstName: "Kelli"})

db.Phone.insert({ID: 1, Number: "518-555-1212"})
db.Phone.insert({ID: 2, Number: "518-123-4567"})

db.PersonPhone.insert({ID: 1, PersonID: 1, PhoneID: 1})
db.PersonPhone.insert({ID: 2, PersonID: 2, PhoneID: 1})
db.PersonPhone.insert({ID: 3, PersonID: 2, PhoneID: 2})

Notice I have to tables, Person and Phone (please ignore my case style, I come from a different world, some may say planet).  I can now query the data using the $lookup aggregate function (plus a few other pieces):

db.PersonPhone.aggregate([
{ $lookup: { from: "Person", localField: "PersonID", foreignField: "ID", as: "PersonName" } }, 
{ $lookup: { from: "Phone", localField: "PhoneID", foreignField: "ID", as: "PersonPhone" } }, 
{ $match: {PersonID: 2} }, 
{$project: {"PersonName.LastName":1, "PersonName.FirstName":1, "PersonPhone.Number": 1, _id:0}} ])

and I get a lovely resulting dataset:

{
  "PersonName": [
    {
      "LastName": "Wagers",
      "FirstName": "Kelli"
    }
  ],
  "PersonPhone": [
    {
      "Number": "518-555-1212"
    }
  ]
}{
  "PersonName": [
    {
      "LastName": "Wagers",
      "FirstName": "Kelli"
    }
  ],
  "PersonPhone": [
    {
      "Number": "518-123-4567"
    }
  ]
}

So now I can use finally MongoDB to satisfy the foundational tenet of a semantic database: a relationship between two objects.

And the nice thing about using a NoSQL database is that schema is not fixed in concrete, as it is with a SQL database, which would normally require additional layers of manipulation to deal with the second foundational tenet of a semantic database: that your concept of, say, a person’s name might be semantically different (but still compatible) with mine — especially when we consider culture.

Strong Type Checking with Semantic Types in C#

Introduction

One of the drawbacks of common typed languages is loss of semantic information.  For example:

string zipcode = "12565";
string state = "NY";
WhatAmI(zipcode);
WhatAmI(state);

void WhatAmI(string str)
{
  Console.WriteLine("I have no idea what " + str + " is.");
}

This illustrates that the semantic meaning for zipcode and state is not known to the program — it is merely a convenient label for the programmer to hopefully populate with the correct value.

The goal of this article is to create a means for implementing immutable semantic types that wrap (usually) native types, providing stronger type checking of parameters, and to do so in a way that is easy to define the semantic type and easy to use the semantic type.

Source Code

Implementing semantic types is trivial — just copy and paste the interfaces and base classes described in the section “Behind the Scenes” and start implementing your own concrete semantic types.

“Stronger Typing” with Semantic Types

The implementation described below let’s us write this instead:

Zipcode z = Zipcode.SetValue("12565");
State s = State.SetValue("NY");
WhatAmI(z);
WhatAmI(s);

// Now we can pass a sematic type rather than "string"

static void WhatAmI(Zipcode z)
{
  Console.WriteLine("Zipcode = " + z.Value);
}

static void WhatAmI(State z)
{
  Console.WriteLine("State = " + z.Value);
}

Now we have “stronger” type checking as a result of using semantic types in place of native types.  Another benefit of this approach is that the semantic type is immutable — every call to SetValue instantiates a new semantic instance.  Using semantic types is therefore very advantageous in multi-threaded applications–in other words, semantic types implement a feature similar to functional programming.  Of course, this immutability can be easily defeated, but it’s not recommend!

Behind The Scenes

The implementation behind the declaration of a semantic type involves a couple interfaces and an abstract base class:

/// <summary>
/// Topmost abstraction.
/// </summary>
public interface ISemanticType
{
}

public interface ISemanticType<T>
{
  T Value { get; }
}

/// <summary>
/// Enforces a semantic type of type T with a setter.
/// </summary>
/// <typeparam name="T">The native type.</typeparam>
public abstract class SemanticType<T> : ISemanticType
{
  public T virtual Value { get; protected set; }
}

/// <summary>
/// Abstract native semantic type. Implements the native type T and the setter/getter.
/// This abstraction implements an immutable native type due to the fact that the setter
/// always returns a new concrete instance.
/// </summary>
/// <typeparam name="R">The concrete instance.</typeparam>
/// <typeparam name="T">The native type backing the concrete instance.</typeparam>
public abstract class NativeSemanticType<R, T> : SemanticType<T>
  where R : ISemanticType<T>, new()
{
  public T Value { get { return val; } }

  protected T val;

  public static R SetValue(T val)
  {
    R ret = new R();
    ret.Value = val;

    return ret;
  }
}

The interface ISemanticType is merely one of convenience when the type information is not available.

The interface ISemanticType<T> is another convenience — this allows us to pass instances of a semantic type without knowing, well, the semantic type.  In other words, it allows us to break the whole point of this article by passing a non-semantic interface instance, but sometimes that’s necessary.

The abstract class SemanticType<T> implements an immutable Value property.  We need a protected setter so that the concrete semantic type can be instantiated with a static factory method, but we don’t want the programmer to change the value once it’s been set.

The abstract class NativeSemanticType<R, T> is where the magic happens.

  1. This class derives from SemanticType<T>, allowing it access to the base class’ protected Value setter.
  2. The class takes R, the generic parameter of the concrete semantic type that is istelf derived from NativeSemanticType<R, T>. That’s really the fun part – a class that takes generic type that is itself derived from that class.

Regarding that last point, the compiler is very picky about what type R can be.  In order for:

ret.Value = val;

to work, ret (being of type R) must be able to access the protected Value setter.  For this to work, R must be of type NativeSemanticType<R, T> — it cannot be (though it would seem reasonable that it ought to be) of type SemanticType<T>.

Implementing Concrete Semantic Types

We can implement the concrete semantic types very easily.  In the example used earlier, the implementation is:

public class Zipcode : NativeSemanticType<Zipcode, string> { }
public class State : NativeSemanticType<State, string> { }

The only nuance to this is that the base class must specify the concrete class as the generic parameter R so that the base class’ SetValue function knows what type to instantiate.  Because this is a static “factory” method, we really can’t avoid this small awkwardness (at least I haven’t figured out how to avoid it.)

The second generic parameter is the backing native type.  Of course, this doesn’t actually have to be a native type — it could be any other class as well.

Additional Benefits of Semantic Types

Here’s a couple reasons to consider semantic types:

Validation

Another neat thing about concrete semantic types is that the type implementation can override the value setter and perform checks.  For example:

public class Zipcode : NativeSemanticType<Zipcode, string> 
{
  public override string Value
  {
    get
    {
      return base.Value;
    }
    protected set
    {
      if (value.Length != 5)
      {
        throw new ApplicationException("Zipcode must have length of 5.");
      }

      base.Value = value;
    }
  }
}

This:

Zipcode fail = Zipcode.SetValue("123");

will now throw an exception.

Security

By using the concept illustrated above, you can ensure that the underlying value is secured, whether encrypted, hashed, or if a credit card, the credit card digits are masked, etc.

Semantic Computing / Distributed Semantic Computing

And of course, if you want to go whole hog, semantic types are also very amenable to multithreaded and distributed computing, as I’ve written about here.

Conclusion

While it may seem strange to code semantically, you may find this a useful technique to provide even stronger type checking of parameters, especially when dealing with function calls or class properties that have many of the same native types in common.  The ability to also provide value checking and other behaviors during the setting of a semantic value is an added bonus.

Also check out Matt Perdeck’s excellent article Introducing Semantic Types in .NET.

What is Semantic Computing, and Why do we Need It?

What is Semantic Computing? In a nutshell:

  1. Semantic computing allows a program to perform computations based on type rather than relying on imperative methods operating on “meaningless” values.
  2. While the above is useful, the full power of semantic computing is revealed when used in conjunction with a publisher/subscriber mechanism and therefore, from my perspective, a pub/sub is an integral part of defining “what is” semantic computing.

I’ll use a simple refrigerator temperature sensor as an example (C# code).

A Non-Semantic Example

In a non-semantic implementation, we might have imperative code flow that looks like this:

public void SampleAndProcessTemperature()
{
  double fridgeTemp = TakeTemperature();
  UpdateTempDisplay(fridgeTemp);
  LogTemp(fridgeTemp);
  
  if (fridgeTemp > 45)
  {
    SendAlert(fridgeTemp);
  }
}

There are several problems here:

  1. This implementation is monolithic — it’s behavior is fixed by the programmer.
  2. The temperature type, a double, is meaningless to the machine.  Meaning exists only in the human-produced code, and even that is limited and error prone.
  3. Digging deeper, the concept of temperature unit (is it Fahrenheit, Celsius, Kelvin, etc) is completely missing.

Even though somewhat contrived, this is how I see 99% of coding done today (including my own) and, in my opinion, is the core reason for why programs have bugs and limited lifetimes requiring complete rewrites every three to seven years.

A Semantic Example

public class DegreesF
{
  public double Value {get; set;}
}

public class FridgeTemp
{
  public DegreesF {get; set;}
}

public void SampleAndProcessTemperature()
{
  FridgeTemp temp = new FridgeTemp() 
  {
    DegreesF = new DegreesF() 
      {
        Value = TakeTemperature()
      }
   };
  Publish(temp);
}

In the semantic example, the native type “double” is encapsulated in the semantic type DegreesF.  This allows the computer to meaningfully understand the native value and with little additional coding can provide conversions to other semantic types such a DegreesC or DegreesK. Also, the imperative function does one thing and one thing only — it publishes a semantic type.  Now, we can plug in subscribers of the semantic type:

sp1

These subscribers “listen” to specific semantic types and perform computations autonomously when that type is published.

A Costly Non-Semantic Failure

NASA lost a $125 million Mars orbiter because a Lockheed Martin engineering team used English units of measurement while the agency’s team used the more conventional metric system for a key spacecraft operation, according to a review finding released Thursday. If Lockheed Martin had used semantic types for the Mars probe software, the compiler would have reported the error in English vs. metric unit differences (source), or even better, would have provided automatic conversion.  Instead, $125 million of taxpayer money went, if not down the drain, then either into deep space or in the creation of crater on Mars.

Summary

Non-semantic programming is monolithic, non-extensible, and, lacking typeful meaning to both the program and the programmer, results in costly bugs and limited application lifetime. Semantic computing is dynamic, extensible, and conveys typeful meaning to the program and programmer.  applications have less bugs when using semantic types rather than native value types, semantic types have meaning to both program and programmer, and a semantic architecture is more resilient to product enhancements, technology changes, and so forth.

Distributed Semantic Computing

distcomp1

I’ve written an article on Distributed Semantic Computing:

Read the full article on Code Project: http://www.codeproject.com/Articles/1012650/Distributed-Semantic-Computing

Excerpts:

In this article, I will demonstrate distributed semantic computing using a Type-First Development (TFD) approach, a term first coined by Tomas Petricek in his blog entry “Why type-first development matters.”

In this article, I’ve re-written the HOPE engine to utilize “type declarative programming.”  This is a style of programming that relies heavily on generics to declaratively describe what should be done, not how.  It is the other side of the TFD coin — in addition to developing types first, we also implement processes that operate on generic types, particularly those that implement specific interfaces.  Similar to how events, delegates, callbacks, and so forth are used for an inversion of control with regards to program behavior, “type declarative programming” is an inversion of control for instantiating objects.  Unlike HOPE, where types are declared in XML and compiled at runtime, here we use types implemented in the code itself.  Because of .NET’s rich reflection and assembly loading capabilities, the difference is irrelevant to the overall goals of HOPE, but the difference to the developer is significant, especially with regards to the safety that a typed language gives you at runtime and the ease of programming (Intellisense and compile-time checking) in a typed language during development.

Type First Development (coined by Tomas Petricek) is applicable to imperative languages as well as functional languages.  We can use the C#’s type system to create rich types and declaratively establish the relationship between types and the methods that process those types.  We can also create containers (membranes) to create computational islands and control the flow of type instances between computational islands.  By using a semantic processor, the membranes, types and receptors that are declared in a “semantic system” becomes a expressive computational unit.  Specialized receptors, such as the distributed receptors illustrated in this article, demonstrate how easy it is to create a distributed semantic computing system.