Open code and you


Moving some of my projects to the wild, I mean to GitHub, has made me realize what others already knew, much of the software we use everyday is open source or has some roots there. Be honest, peeking to someone else’s software is great. As to my own experience I started learning programming as a child reading the source code of the Doom Editing Utilities (and that probably shifted and twisted my understanding of computers for my entire life). This concepts of seeing others work and learning from it is a hidden fact that affects our life in for example every gadget we use (read Android/iOS). But not only to learn but to understand what a piece of software does, its source code is the ultimate documentation. If you are using a third party library, ask for its source code and peek at it when needed. At Video Stream Networks we made a similar approach, we had our domain logic code written in C++ every time we had to tune it for every customer’s workflow it made us feel miserable. After we moved our domain logic into Casper, a Domain Specific Language for the media and broadcast world, our customer can not only peek at what our app does but understand and fine tune it themselves.

Thanks to GitHub and the Open Source Community, ShadingZen Engine is slowly maturing. The GitHub repo received a pull request with some great changes. The project now has Maven build, the structure is more clear with all the examples and documentation centralized in just one repository and the eclipse requirement has been removed (This last point allowed me to try out IntellIJ IDEA which so far I find as a really good java IDE).

Yes, open source level editors as well a robust designed and extensible engine was behind the success of DOOM, the very same that made some of us programmers.


Domain-Specific Languages


Half the work I’m currently doing here at Video Stream Networks is to design a DSL runtime engine as the agent which drives all the flow of data through our services. In fact, our current implementation is not in heart a true DSL as it donesn’t complete move you away from more mundane tasks like parsing data or having to solve some library/infrastructure matters (we are always trying to improve this point tho).

That was a response to our needs to provide the customers (and us)  a tool to deliver custom solutions, as often our customers are in need of an specific workflow which always requiered modifications to our “traditional” software. Building a server-side scripting system allows this and also lets thridparty enterprises join the game and develop custom tools that operate on top of our solution, further improving integration.

Our scripting language of choice is Boo, a dynamically typed programming language for the CLR, which has some interesting extensibility capabilities. By using macros, some code behind and time, one could develop some kind of DSL that may help the user concentrate on solving domain problems instead of wasting time solving technical problems related to the architecture of choice.

The source of inspiration was DSLs in Boo: Domain-Specific Languages in .Net, a not so structured book which guides the reader to each step involved on implementing a DSL based in Boo and RhinoDSL (his own dsl scripts factory library), it’s worth reading tho.

Our implementation has various key features:

  • Persitence through a well defined persistence points. Flow can be later resumed.
  • Wait for external events.
  • Not tied to any specific domain object.
  • Not tied to any specific domain object.
  • Parallel execution of scripts.
  • Service Oriented Architecture.

The script flow is moved from step to step (stages) which are the main code block within an script. These stages also define where the flow is persisted and subsequent runs won’t enter already executed stages.

The input of the script is open. It is defined by the script itself and its clients, which are responsible of feeding the correct data. To improve data handling we share a set of libraries to clients and server, where the input is defined for both parties.

Our runtime is capable of handling concurrent execution of scripts using the same (shared) underlying servicies (like persistence, tracking or any service that we want to publish to the script side).

We call this scripts “workflows”, being the script runtime one service of the new SOA platform we have built to deliver a product that can be easily modified to our customer needs.

Tracking the flow in the mesh of services

I’m currently worried on how our embrace of SOA and REST can take a big impact on our support team. In a world of closed entities where the boundary of the underlaying operation is well defined, a simple logging system is enough to track down the source of the problem. The failed operation is well confined inside an application scope, but on a service oriented architecture the message could have been propagated to other services. Tracking down the source of the problem definitively doesn’t seem to be trivial.

A central logging service is a good start. All message outcomes can be tracked into this service, which is a central shared point to look for answers. But what if your architecture is a large SOA implementation with a mesh of services? If a call to a service generates smaller calls to secondary services, the logging system is still insufficient. One could start reading log by log  in the hope to understand what was the outcome of each service call in order to get the whole picture, but the always-busy support team is not going to be happy with this.

I could think in a couple of simple solutions: one could be to propagate the id of the root message, the one that triggered everything, so that a query with this id to the logging system would return a list of all related message. Our we can scale this to not only track the root message but to propagate in each message (as embedded info into each message) the id of the previous message in the chain. A list of chained message can easily be looked up in our logging system as a list of log messages (of a list of services messages chain). This information should also be embedded into the event drive part of the system, thus all events should also track this chain of ancestor messages. With this approach we can easily visualize which message triggered a resource operation in another service, consequently allowing the technical support team to look and better comprehend into the flow of that certain execution path.

Our we can move to the next level: some sort of exception handling (in the SOA way)…

Workflows Foundation .Net 4

I have been making a prototype to evaluate the new incarnation of the Windows Workflow Foundation (WWF 4 beta 1), which will be part of the .Net 4 release. I’m glad of the new changes so far, asynchronous operations are now much easier, and this also applies to activities authoring too. I’m not so happy with the designer, it’s still a bit slow, although it is not an issue for me right now.

Workflow isolation has also been left behind. Running untrusted code can completely break the engine wich will also stop the other workflows running in parallel along with the faulted one. This is a big issue when running unmanaged code, where memory access violations or any other severe exception will crash the process. Writing the correct infrastructure to handle this situation is not trivial.

Many business problems are easily modeled using workflows, specially now that asynchronous transactions have been improved in this beta release. And WWF supports many (if not all) of the common workflow patterns. State machine workflows seems to be hard to implement with this release but I haven’t look into this yet.

You can find more info at the Windows Workflow Foundation for .Net 4 developer center:

More on workflow patterns here: