Month: November 2017

HOPE Semantic Rendering of Microservice Data on the Browser

In a previous blog post, I described the high level architecture of microservices
running in a Docker container.  This blog post describes (again at a high level) the behind-the-scenes implementation of how a we render the output of a microservice back to the client’s browser. We’ll use the Weather Underground API for acquiring our current weather conditions.

First, let’s start with a semantic address:

public class ST_Address : ISemanticType
{
  public string Address1 { get; set; }
  public string Address2 { get; set; }
  public ST_City City { get; set; }
  public ST_State State { get; set; }
  public ST_Zip Zip { get; set; }

  public ST_Address()
  {
    City = new ST_City();
    State = new ST_State();
    Zip = new ST_Zip();
}

Note that there are other semantics involved here (ST_City, ST_State, and ST_Zip) which we won’t go into here.  We’ll populate the ST_Address with the two pieces that Weather Underground requires, city and state, and then publish this type to the docker container:

http://192.168.99.100:5001/publishSemanticType?
  semanticTypeName=ST_Address&instanceJson=
  {
    "City":
      {"City":"Hudson"},
    "State":
      {"State":"NY"}
  }

The microservice, in HOPE terminology, is a “receptor” that is interested in
the type ST_Address.  As implemented in C# (fragment):

public class WeatherForecast : IReceptor
{
  public void Process(ISemanticProcessor proc, IMembrane membrane, ST_Address address)
  {
    string json = Http.Get("http://api.wunderground.com/api/[yourapikey]/forecast/q/" + 
      address.State.State + "/" + 
      address.City.City + 
      ".json");

    var data = WeatherUnderground.FromJson(json);

    var statForecast = data.Forecast.Simpleforecast.Forecastday.Select(d =>
      new
        {
          Month = d.Date.Monthname,
          Day = d.Date.Day,
          Weekday = d.Date.Weekday,
          High = d.High.Fahrenheit,
          Low = d.Low.Fahrenheit,
          Conditions = d.Conditions,
          IconUrl = d.IconUrl,
          MaxWind = d.Maxwind.Mph,
        }).ToList();

    // ...
    // Building the day and night detail forecast is not shown.
    // ...

    Response result = new Response()
    {
      HopeResultWrapper = new WeatherResult()
      {
        StatForecast = statForecast,
        DayForecast = daySummaryForecast,
        NightForecast = nightSummaryForecast,
      }
    };

    // Call back to the host with the result.
    RestCall.Post("http://192.168.0.2/hope/result", result);
  }
}

The response is associated with an XML file that determines the rendering.
Here’s a fragment showing just the first to rows (the day and the icon) of
rendering definition:

hope-5

HOPE renders this into HTML and sends the HTML over an HTML5 websocket to the browser where the page defines this tag:

hope-6

and on receiving the message from host, executes this Javascript:

 ws.onmessage = function (wsevt)
 {
   var msg = wsevt.data;
   $("#appContent").empty();
   $("#appContent").append(msg);
 };

The result is rendered on the browser like this:

hope-2

By changing the rendering to a VerticalGrid we instead get:

hope-3

By changing the rendering XML definition again (not shown, it’s quite long) we can show the results from all three collections — stat, day, and night — to render a more detailed forecast.

hope-4

Of course, in real life, you’ll want to render the results of many microservices.  We’ll look at how that works soon!

Uninstalling Docker for Windows

docker-for-windows-x

Because of the performance issues with starting up a Docker for Windows container, I decided to uninstall it and go back to the Docker Toolbox and try out .NET Core 2.0.  I’m much more pleased with the experience — Docker is definitely a lot more usable with a Linux OS!  Furthermore, it turns out Hyper-V totally Elephant’ed my video drivers (there’s a lot of posts on the web about that), so NetFlix and “streaming” became a contradiction in terms.

Disabled Hyper-V (a much less painful process than enabling it) and uninstalled Docker. Yet, WTF, there is still 20GB of docker crap left in the ProgramData folder.

And it can’t be deleted — the typical “Administrator access is required” but my user account is an administrator BS in Windows. Find a post on the Docker forum that basically says “yeah, deleting ProgramData\Docker is really hard, we could do a better job of it, you should use docker rmi before uninstalling Docker.” Or something like that. Great, if there’s a command I can run manually, why can’t the uninstaller run it?

So more Googling, trying to take ownership of the file (fail) and I find this post that basically says:

  1. Boot into Safe mode. (Figuring out how to do that in W10 was yet another Google and 5 screens of restart options.)
  2. After reboot, do a “robocopy” of ProgramData\Docker to somewhere else.
  3. After the robocopy finishes, you can remove the directory with rd /S /Q (include subfolders, and don’t prompt for every freaking folder.)

What do you know, that worked. I’ve now reclaimed the 20GB that Docker for Windows failed to install.

Unfortunately, uninstalling Docker for Windows had a few pain points as well.  In my not-so-humble opinion, Microsoft has a ways to go for Docker for Windows to be a viable container development option.

 

 

My Docker for Windows Experience

docker-for-windows

Or “How I spent last weekend.”

Docker! More specifically, testing out the .NET Framework 4.7 image using Docker for Windows.

So first I had to enable Hyper-V. No problem, methinks. Riiight. After rebooting the computer, my multimonitor setup is totally trashed, thrashes, fubar’ed. As are my Fences icons. And there’s these extra monitors suddenly showing up. WTF?

Get that finally squared away, download the image (the instructions on the github repo are fubar’d) and launch VS2017.

Where is it? I get the VS2017 splash screen, then nothing. Elephant me, it looks like VS2017 & VS2015 are blown away. Uninstall both, reinstall both.

Same problem. Elephant.

Just by chance, I hit Alt-Tab and I see Visual Studio is one of the running applications. Where the Elephant is it??? So I think, maybe it’s on one of those weird extra monitors I saw. So I poke around, and there’s this “disconnect monitor” option in the display settings. So I try that for one of the “ghost” monitors. OMFG. Suddenly VS appears.

Next, I discover the BS that is Microsoft’s attempt at bringing .NET onto Docker. So there’s this image. OK. Docker tells me it’s downloading some 10 GIGA (yes, that’s GIGA) bytes of data for the image. OMG OMG OMG When all is said and done, my poor little 256GB SSD only has 1.4GB free (it was pretty stuffed already) but it turns out the whole Docker/Windows/.NET thing sucks up 17GB of disk space uncompressed. Well, at least the goofy demo Microsoft gives you compiled.

Oh, but I forgot to tell you, you need to switch to “Windows containers” in the Docker applet. Oh, and what I really forgot to tell you was the three install attempts for Docker for Windows because I had a previous “Docker Toolbox” installation (which I did uninstall before doing anything!!!) but that left various breadcrumbs – PATH entries (I had added those as they are necessary to work with the Docker VM), and a “.docker” folder in the Users\Marc folder. All of which caused Docker For Windows to be “No Docker For You”. Two separate SO posts resolved those issues.

So, we finally get to compiling. Fine. Then we get to running.

And We wait (at this point, it’s the royal WE)

And We Wait

await Marc;

And We Wait

And OMG it ran! There’s the output of the silly “bot” in the console window.

It takes something like 15-30 seconds on my 8 core 16GB fast machine to spin up the container, undoubtedly because I’m running a near full instance of Windows. I decide to try a few other things, all of which work, and finally landing on “let’s write a simple HttpListener” as a container. Great, 20 lines of code later and I have something that works in Windows. Let’s see if it works in the container.

Nope. Sesame Street time: One of these things (or most of these things) doesn’t belong: http://localhost, http://127.0.0.1, http://0.0.0.0, http://*

The last one, http://*:5001/ is what works. I go to my browser, to “localhost:5001” and…

nothing.

Hours later, I discover some interesting things. By now, after fussing with Dockerfile EXPORT and that http://*:5001, I discover that entering the container’s local IP into my browser works. But that’s container’s local IP, which changes every time the container is fired up, for each container that is fired up, all hail Mary to the container that never rules anything.

Various posts say you can’t access a container from the host. Some posts say you’re not using a container correctly if you’re doing this, you should use a VM instead (and indeed, I had a VM of this working with Python.) Some say other BS, but at the end of the day, they’re all saying BS, but I still can’t figure out the problem.

I make a post on the Docker forum. I refuse to give up. The oxygen tank is redlining, but I refuse to give up. And then I find it.

 Kymeric/DockerProxy: Proxy localhost for Docker Windows Containers

Indeed you can’t access the container through the host’s localhost unless you proxy localhost:port to [containerIP:port], and that’s what that little gem above does.

Sweet. Everest has been vanquished. No Elephants where harmed in this post.

At the end of the day though, Docker for Windows is an enormous footprint on your drive and painfully, actually unusable given how slow it is to fire up a container.  With round trip testing / debugging / code fixes, it’s not a viable solution, particularly when you’re figuring out the nuances of how to communicate bidirectionally and the code has to be tested in the container.

HOPE for the Web

hope-1

The above diagram is a high level view of what I’ve implemented with .NET Core 2.0 and Docker.  The idea is to take the work I’ve done with semantic types and allow users to create public or private repos of semantic types and microservices that operate on those types whenever they are “published” in the user’s environment.

With .NET Core 2.0, all the core pieces that I need (reflection, NewtonsoftJson, HttpListener, to name a few) are all fully usable with C# 7.  And while I’ll eventually return to supporting Python, it is a pleasure work with this concept in a strongly-typed language, as that’s a really key component to semantic processing (yes, of course it can be done in a dynamically-typed language with Python as well, as I’ve demonstrated in previous posts and an article.)

And the beauty of doing this with .NET Core is that it can run in a Linux container, which is so much faster than using Docker for Windows!

I’ll be posting more about the implementation details and writing some articles on Code Project on the subject.

Kademlia Peer-to-Peer Distributed Hash Table Implementation in C#

kademlia

Over the last several months I’ve been working on implementing the Kademlia P2P DHT in C#, as per the specification here.  This is a very complete implementation with a couple demos.  SyncFusion will be publishing an e-book that I wrote documenting the implementation, hopefully it’ll be out in December.

Now that the eBook has been published by SyncFusion, the source code can be found here.