Open code and you

doom

Moving some of my projects to the wild, I mean to GitHub, has made me realize what others already knew, much of the software we use everyday is open source or has some roots there. Be honest, peeking to someone else’s software is great. As to my own experience I started learning programming as a child reading the source code of the Doom Editing Utilities (and that probably shifted and twisted my understanding of computers for my entire life). This concepts of seeing others work and learning from it is a hidden fact that affects our life in for example every gadget we use (read Android/iOS). But not only to learn but to understand what a piece of software does, its source code is the ultimate documentation. If you are using a third party library, ask for its source code and peek at it when needed. At Video Stream Networks we made a similar approach, we had our domain logic code written in C++ every time we had to tune it for every customer’s workflow it made us feel miserable. After we moved our domain logic into Casper, a Domain Specific Language for the media and broadcast world, our customer can not only peek at what our app does but understand and fine tune it themselves.

Thanks to GitHub and the Open Source Community, ShadingZen Engine is slowly maturing. The GitHub repo received a pull request with some great changes. The project now has Maven build, the structure is more clear with all the examples and documentation centralized in just one repository and the eclipse requirement has been removed (This last point allowed me to try out IntellIJ IDEA which so far I find as a really good java IDE).

Yes, open source level editors as well a robust designed and extensible engine was behind the success of DOOM, the very same that made some of us programmers.

Domain-Specific Languages

Boo

Half the work I’m currently doing here at Video Stream Networks is to design a DSL runtime engine as the agent which drives all the flow of data through our services. In fact, our current implementation is not in heart a true DSL as it donesn’t complete move you away from more mundane tasks like parsing data or having to solve some library/infrastructure matters (we are always trying to improve this point tho).

That was a response to our needs to provide the customers (and us)  a tool to deliver custom solutions, as often our customers are in need of an specific workflow which always requiered modifications to our “traditional” software. Building a server-side scripting system allows this and also lets thridparty enterprises join the game and develop custom tools that operate on top of our solution, further improving integration.

Our scripting language of choice is Boo, a dynamically typed programming language for the CLR, which has some interesting extensibility capabilities. By using macros, some code behind and time, one could develop some kind of DSL that may help the user concentrate on solving domain problems instead of wasting time solving technical problems related to the architecture of choice.

The source of inspiration was DSLs in Boo: Domain-Specific Languages in .Net, a not so structured book which guides the reader to each step involved on implementing a DSL based in Boo and RhinoDSL (his own dsl scripts factory library), it’s worth reading tho.

Our implementation has various key features:

  • Persitence through a well defined persistence points. Flow can be later resumed.
  • Wait for external events.
  • Not tied to any specific domain object.
  • Not tied to any specific domain object.
  • Parallel execution of scripts.
  • Service Oriented Architecture.

The script flow is moved from step to step (stages) which are the main code block within an script. These stages also define where the flow is persisted and subsequent runs won’t enter already executed stages.

The input of the script is open. It is defined by the script itself and its clients, which are responsible of feeding the correct data. To improve data handling we share a set of libraries to clients and server, where the input is defined for both parties.

Our runtime is capable of handling concurrent execution of scripts using the same (shared) underlying servicies (like persistence, tracking or any service that we want to publish to the script side).

We call this scripts “workflows”, being the script runtime one service of the new SOA platform we have built to deliver a product that can be easily modified to our customer needs.

Rest back-end and UIs

We have been spending the last four months moving our Win32 stand-alone applications to a more service oriented architecture. This new back-end  can be scaled to handle a wide variety of  TV Broadcasting enterprises. It has been a major movement from us, as many domain logic that was previously hardcoded into the desktop applications is now implemented into this back-end and shared among all clients. The logic is customizable per client basis through an scripting system.

We have targeted two desktop operating systems, MacOSX 10.6 and WindowsXP+ (in this order). These led to a new problem: developing a front-end that would offer the most user-friendly experience to the end user. Cocoa and WPF were the best choices and Monobjc provided a nice bridge to handle the Cocoa interface through Net/Mono. Silverlight was the third choice to those interfaces that could be also used from the web. The RESTful back-end allows us to easily develop and deploy any user interface while maintaining full compatibility within the installation with other clients.

Compared to AJAX and HTML5, Silverlight was the most cost-effective technology for us as we are used to write .Net code and we could use a large .net library we have already developed. So we have software (desktop applications for video editing, video review and cataloging) that share a common back-end and offer a native interface through Cocoa and WPF and some Web applications using Silverlight that also target MacOSX and Win32 clients but are more suited for the Web.

This huge project has been fun and having the chance to take part in almost any part of the new architecture (I have designed and implemented  part of our back-end and also being in charge of evaluating and teaching  Cocoa/Monobjc).

Tracking the flow in the mesh of services

I’m currently worried on how our embrace of SOA and REST can take a big impact on our support team. In a world of closed entities where the boundary of the underlaying operation is well defined, a simple logging system is enough to track down the source of the problem. The failed operation is well confined inside an application scope, but on a service oriented architecture the message could have been propagated to other services. Tracking down the source of the problem definitively doesn’t seem to be trivial.

A central logging service is a good start. All message outcomes can be tracked into this service, which is a central shared point to look for answers. But what if your architecture is a large SOA implementation with a mesh of services? If a call to a service generates smaller calls to secondary services, the logging system is still insufficient. One could start reading log by log  in the hope to understand what was the outcome of each service call in order to get the whole picture, but the always-busy support team is not going to be happy with this.

I could think in a couple of simple solutions: one could be to propagate the id of the root message, the one that triggered everything, so that a query with this id to the logging system would return a list of all related message. Our we can scale this to not only track the root message but to propagate in each message (as embedded info into each message) the id of the previous message in the chain. A list of chained message can easily be looked up in our logging system as a list of log messages (of a list of services messages chain). This information should also be embedded into the event drive part of the system, thus all events should also track this chain of ancestor messages. With this approach we can easily visualize which message triggered a resource operation in another service, consequently allowing the technical support team to look and better comprehend into the flow of that certain execution path.

Our we can move to the next level: some sort of exception handling (in the SOA way)…

Harmful dependency cycles

Rushing things is bad.  You are in a hurry to get to the supermarket just to find, once you are there, that you forgot the shopping list at home. When developing applications, rushing things without a proper background design leads to spaghetti code. You module A depends on B, B depends on C, and C depends on A. There is some hidden master components, the sums of all of them, which is hard to notice, maintain and reuse.

 

Dependecy structure matrix example

 

Dependency structure matrix can help you find those harmful dependencies. The key on this tool is the way the important data is presented to the user (you). Matrix organization let’s you easily spot dependencies, where graphs tend to be hard to visualize.

Dependency diagram

This is one of the tools we are currently using to deliver high quality software to our clients. You can find more info in the following links:

http://sdg.csail.mit.edu/pubs/2005/oopsla05-dsm.pdf

http://www.theserverside.net/tt/articles/showarticle.tss?id=ControllingDependencies

Reservation pattern

I was reading the SOA pattern – Reservations from Arnon Rotem (blog) I found it quite interesting. You can see it as a mix of optimistic resource locking with expiration time, where the service itself can revoke the lock in the first place due to some internal reason.

#Reservation patter. Image taken from Arnon Rotem blog.

Although this pattern may not be applicable in all scenarios due to the fact that the reservation may not be effective when the transaction is committed, making this assumption more restrictive (resources are guaranteed until the expiration time) makes this pattern more widely usable.

By providing several levels of guarantee the service can offer its resources to more consumers if some of them doesn’t require a high level of guarantee because they can compensate for ineffective reservations. This also adds another order of complexity in logic though.

PostSharp

You may already know Aspect-oriented programming and how further decouple non business logic from the model. .Net has been using this paradigm across various frameworks like WCF by using attributes. Reflection has been for me the only reasonable way to exploit these attributes so far, which means the behavior is changed in runtime (which also added complexity). You also had to write down how this attributes are handled which didn’t help concentrating code.

PostSharp is a true AOP helper here, letting you completely describe the business and adorn the fields, methods and classes with .Net attributes. During a post-build step, the assembly is injected with some code based on those attributes. The aspects are encapsulated as attributes freeing the team from writing tons of repetitive lines. Decoupling aspects from the main logic also encourages code reutilization and increases maintainability by unifying scattered code in just one place. Although sometimes finding current crosscutting concerns in the code, or taking the effort to re-implement them as aspects can be overwhelming at first.

PostSharp 2.0 has gone even further allowing dependency solving which is something very promising (although its license may change).