August 16, 2018

Posted by Fabrizio Montesi

Microservice-aware Web Application (MAWA)

Chuck kicks

Introduction (and TL;DR)

You want to build a website based on microservices.

Leonardo makes it easy to build the services and handle separation of concerns by routing messages to (sub)services exposed to the web browser. Jo exposes these services as JavaScript objects to the webpage (an idea I’ve been toying with recently). I call the result a Microservice-aware Web Application (MAWA), because the webpage accesses explicitly (is aware of) the different (micro)services made available by the server.

The purpose of this document is to show you how. The only software you need installed in your computer to try out what I show here is Docker.

I’m going to use some Jolie language, because it makes things very simple for the purposes of this tutorial. You do not need to know Jolie to read what follows: the code is simple enough to be explained on the fly.

References

You can run the finished application with the following Docker commands.

docker pull fmontesi/jo-demo-chuck
docker run -it --rm -p 8080:8080 fmontesi/jo-demo-chuck

Just browse to http://localhost:8080/ once the container is running.

You can look at the finished source code here: https://github.com/fmontesi/jo-demo-chuck.

Get started

Create the following directory structure.

app/
	internal/
web/

We will use app for our services, and web for our web frontend files.

Install the web server

Go to app/internal and download the Leonardo web server in the subdirectory leonardo:

curl -L https://github.com/jolie/leonardo/archive/0.2.tar.gz | tar xvz --transform 's/leonardo-0.2/leonardo/'

We will use Leonardo to serve static content and to route service invocations from webpages to our services.

Write the main application

Now we write our main Jolie application, in file app/main.ol.

// This includes the LeonardoAdmin output port, which we'll use to configure Leonardo
include "internal/leonardo/ports/LeonardoAdmin.iol"

// Embedding is Jolie for running a service inside of the same VM
embedded {
Jolie:
	/*
		Run Leonardo and link it to the output port LeonardoAdmin
		Communications over embedding use local memory (they're fast)
		The Standalone=false parameter means that we'll have to configure Leonardo
	*/
	"-C Standalone=false internal/leonardo/cmd/leonardo/main.ol" in LeonardoAdmin
}

main
{
	with( config ) {
		.wwwDir = "../web" // The web content directory
	};
	config@LeonardoAdmin( config )(); // Send the configuration to Leonardo
	linkIn( Shutdown ) // Wait until somebody terminates us
}

Make a run script

First, pull Jolie.

docker pull jolielang/jolie

Now we make a convenience script to run our website. Write this in run.sh in the root directory of your project.

#!/bin/sh

docker run -it --rm \
	-v "$(pwd)"/web:/web \
	-v "$(pwd)"/app:/app \
	-w /app \
	-p 8080:8080 \
	jolielang/jolie \
	jolie main.ol

The script mounts the app and web directories as volumes, goes in app, and runs our program in main.ol. We expose the TCP port 8080.

Make the script executable:

chmod +x run.sh

Whenever I say “run the application” in the remainder, I mean: go to the root directory of the project and run ./run.sh (sh run.sh also works, if you did not make run.sh executable).

Test if your setup works

Create a file web/index.html with this content:

<html><body>Hello, World!</body></html>

Run the application (reminder: this means executing run.sh). You should see the output:

Leonardo started
	Location: socket://localhost:8080/
	Web directory: /web/

Go to http://localhost:8080/ with your web browser. You should see the “Hello, World!” message we wrote in web/index.html.

Hello, World!

Adding services

Let’s add some dynamic fun. We can use Leonardo to proxy requests to both external and internal services. External services are services run outside of our application. Internal services are provided by our application. Other than that, their configuration in Leonardo is the same.

External services

By “dynamic fun”, I meant Chuck Norris jokes. There’s an API for that: https://api.chucknorris.io/. (If you didn’t know about this, reading this tutorial might be worth it just for getting that link now…) For example, to search for a joke containing the word “computer”, we just invoke https://api.chucknorris.io/jokes/search?query=computer.

To get Chuck Norris in our application, we need to update app/main.ol as follows. I comment the new lines (the rest is as before).

include "internal/leonardo/ports/LeonardoAdmin.iol"

// Output port to contact api.chucknorris.io
outputPort Chuck {
Location: "socket://api.chucknorris.io:443/" // Use TCP/IP, port 443
Protocol: https {
	.osc.search.method = "get"; // HTTP method for operation search
	.osc.search.alias = "jokes/search?query=%{query}" // URI template for operation search
}
RequestResponse: search // Request-response operation declaration (search)
}

embedded {
Jolie:
	"-C Standalone=false internal/leonardo/cmd/leonardo/main.ol" in LeonardoAdmin
}

main
{
	with( config ) {
		.wwwDir = "../web";

		// Expose a service to the web application
		with( .redirection[0] ) {
			.name = "ChuckNorris"; // Name of the exposed service
			.binding -> Chuck // Binding information (from output port Chuck)
		}
	};
	config@LeonardoAdmin( config )();
	linkIn( Shutdown )
}

Our web server now exposes a ChuckNorris service that supports operation search. Next, we update the web/index.html page to invoke this service. We use Jo (a library to interact with Jolie web servers) to call the web server, and jQuery to interact with the DOM (but you can use any other framework you like).

<html>
<head>
	<script type="text/javascript"
		src="https://raw.githubusercontent.com/fmontesi/jo/master/lib/jo.js"></script>
	<script
		src="https://code.jquery.com/jquery-3.3.1.slim.min.js"
		integrity="sha256-3edrmyuQ0w65f8gfBsqowzjJe2iM6n0nKciPUp8y+7E="
		crossorigin="anonymous"></script>
</head>
<body>
	<p>
		<input type="text" id="jokeQuery" value="Computer" />
		<button id="getJokeBtn" type="button">Get me a Chuck joke!</button>
	</p>
	<p id="display"></p>

	<script type="text/javascript">
		$(document).ready( () => {
			$("#getJokeBtn").click( () => {
				$("#display").html( "Work in progress..." );

				// Call operation search on the exposed service ChuckNorris
				Jo("ChuckNorris").search( { query: $("#jokeQuery").val() } )
					.then( response => {
						// Pick a random joke
						// api.chucknorris.io returns jokes in a "result" array subelement
						$("#display").html(
							( Array.isArray( response.result ) )
							? response.result[ Math.floor(Math.random()*response.result.length) ].value
							: "This is not a joke"
						);
					} )
					.catch( JoHelp.parseError ).catch( alert );
			} );
		} );
	</script>
</body>
</html>

Clicking the button “Get me a Chuck joke!” will fetch a Chuck Norris joke from https://api.chucknorris.io/ for us. I got this joke:

Chuck Norris is no joke

Internal services

What good is a joke if you can’t share it? We write a Jolie service to post our jokes to https://telegra.ph/. I’m going to put all the code in our app/main.ol. (In a real project, you’d probably want to start creating separate files at this point and embed them, just like we did with Leonardo.)

Here’s the new app/main.ol.

include "internal/leonardo/ports/LeonardoAdmin.iol"

outputPort Chuck {
Location: "socket://api.chucknorris.io:443/"
Protocol: https {
	.osc.search.method = "get";
	.osc.search.alias = "jokes/search?query=%{query}"
}
RequestResponse: search
}

embedded {
Jolie:
	"-C Standalone=false internal/leonardo/cmd/leonardo/main.ol" in LeonardoAdmin
}

include "string_utils.iol"

// Output port to Telegraph. We're going to need it to implement our internal service.
outputPort Telegraph {
Location: "socket://api.telegra.ph:443/"
Protocol: https {
	.format = "json";
	// Yeah, I'm using querystrings... that's the official example in the Telegraph API documentation
	.osc.createAccount.alias =
		"createAccount?short_name=%{short_name}";
	.osc.createPage.alias =
		"createPage?access_token=%{access_token}&title=%{title}&content=[\"%{content}\"]"
}
RequestResponse: createAccount, createPage
}

type CreatePageRequest:void { .title:string .content:string }

interface TelegraphPosterIface {
RequestResponse: createPage(CreatePageRequest)(string) throws TelegraphError(string)
}

// Here's the internal service that we're going to expose to the webpage
service TelegraphPoster {
Interfaces: TelegraphPosterIface
main {
	createPage( postRequest )( postUrl ) {
		// Create a fresh account on Telegraph
		short_name = new;
		replaceAll@StringUtils( short_name { .regex = "-", .replacement = "" } )( short_name );
		createAccount@Telegraph( { .short_name = short_name } )( response );
		if ( !response.ok ) throw( TelegraphError, response.error );

		// Post the joke with the account we've just created
		access_token = response.result.access_token;
		createPage@Telegraph( {
			.access_token = access_token,
			.title = postRequest.title,
			.content = postRequest.content
		} )( response );
		if ( !response.ok ) throw( TelegraphError, response.error );
		postUrl = response.result.url // Return the post's url to the webpage
	}
}
}

main
{
	with( config ) {
		.wwwDir = "../web";
		with( .redirection[0] ) {
			.name = "ChuckNorris";
			.binding -> Chuck
		};
		with( .redirection[1] ) { // We add the redirection for TelegraphPoster
			.name = "TelegraphPoster";
			.binding.location = TelegraphPoster.location
		}
	};
	config@LeonardoAdmin( config )();
	linkIn( Shutdown )
}

So now we have a new service TelegraphPoster that the webpage can invoke to post content on Telegraph. Let’s use it to share our jokes.

<html>
<head>
	<script type="text/javascript"
		src="https://raw.githubusercontent.com/fmontesi/jo/master/lib/jo.js"></script>
	<script
		src="https://code.jquery.com/jquery-3.3.1.slim.min.js"
		integrity="sha256-3edrmyuQ0w65f8gfBsqowzjJe2iM6n0nKciPUp8y+7E="
		crossorigin="anonymous"></script>
</head>
<body>
	<p>
		<input type="text" id="jokeQuery" value="Computer" />
		<button id="getJokeBtn" type="button">Get me a Chuck joke!</button>
	</p>
	<p id="display"></p>
	<p>
		<button id="postJokeBtn" type="button">Share this wisdom</button>
	</p>
	<p id="telegraphDisplay"></p>

	<script type="text/javascript">
		$(document).ready( () => {
			$("#getJokeBtn").click( () => {
				$("#display").html( "Work in progress..." );
				Jo("ChuckNorris").search( { query: $("#jokeQuery").val() } )
					.then( response => {
						// Pick a random joke
						$("#display").html(
							( Array.isArray( response.result ) )
							? response.result[ Math.floor(Math.random()*response.result.length) ].value
							: "This is not a joke"
						);
					} ).catch( JoHelp.parseError ).catch( alert );
			} );

			// Should be self-explanatory by now
			// When the button is clicked, invoke createPage at TelegraphPoster
			$("#postJokeBtn").click( () => {
				$("#telegraphDisplay").html( "Work in progress..." );
				Jo("TelegraphPoster").createPage( {
					title: "Chuck Norris Wisdom",
					content: $("#display").html()
				} ).then( response => {
					$("#telegraphDisplay").html(
						`Wisdom shared at <a href=\"${response.$}\" target=\"_new\">${response.$}</a>`
					);
				} ).catch( JoHelp.parseError ).catch( alert );
			} );
		} );
	</script>
</body>
</html>

Here’s a screenshot of what the final page looks like.

Now everybody knows

Done!

It’s done!

If you’d like to containerise your application, you just need a simple Dockerfile, like the following.

FROM jolielang/jolie

COPY app /app
COPY web /web

WORKDIR /app
EXPOSE 8080

CMD ["jolie","main.ol"]

References

  • Fabrizio Montesi. (2016). Process-aware web programming with Jolie. Sci. Comput. Program., 130, 69–96. doi arxiv
  • Fabrizio Montesi, Claudio Guidi, & Gianluigi Zavattaro. (2014). Service-Oriented Programming with Jolie. In A. Bouguettaya, Q. Z. Sheng, & F. Daniel (Eds.), Web Services Foundations (pp. 81–107). Springer. doi pdf

August 14, 2018

Posted by Fabrizio Montesi

A script to serve webpages quickly with Leonardo and Docker

What

You want to serve some web pages, quickly!

Installation

You’ll need Docker. Pull the Docker image for Leonardo if you don’t have it already.

docker pull jolielang/leonardo

Download the serve script and set it as executable.

wget https://rawgit.com/jolie/leonardo/master/cmd/serve/serve && chmod +x serve

Use

Syntax: serve [path] [-p TCP_PORT] where the default path is . and the default TCP_PORT is 8080.

Examples

Serve the current directory: serve

Serve directory web: serve web

Serve directory web on port 8000: serve web 8000

January 09, 2018

Posted by Saverio Giallorenzo

A Microservice Implementation of the Raft Consensus Protocol

Raft is a distributed consensus algorithm: it offers a generic way to distribute a state machine across a cluster of computing systems, ensuring that each node in the cluster agrees upon the same series of state transitions.

The main motivation that drove Ongaro and Ousterhout in the design of Raft was to make it easy to understand and reason about.

Last year (2017) I empirically tested Ongaro and Ousterhout’s thesis by giving as laboratory assignment for the second-year bachelor course on Operating Systems (Information Science for Management, DISI, UniBo) the implementation of an online shop, backed by a cluster of servers running the Raft protocol.

For the delivery, I strongly advised the students to use Jolie as programming language.

Why Jolie?

Previous successful experiences. For one, with Jolie students implicitly become familiar with the popular architectural style of microservices. Second, by developing with Jolie, neophyte students can rapidly grasp the essential concepts behind distributed applications and create complex systems. Third, Jolie lifts a lot of the complexity of setting up and running a distributed architecture, which lets programmers focus on getting their distributed logic right, rather than spending time and time on network bindings, data marshalling and transmission, etc.

Among the many excellent implementations that I received, the one delivered by Matteo Berti, Arnaldo Cesco, and Salvatore Fiorilla (group 01011001) strikes for its clarity and ease of use. In particular, they walked the proverbial “extra mile” to include additional features like easy-to-use CLIs and Web GUIs.

After the examination, they agreed to further polish their project and publish it in a github repository so that everyone interested in using Raft in their Jolie (and Java, and JavaScript) projects can take it as a reference implementation. Technically, since in the project Raft is developed as a standalone set of microservices — composed with the embedding mechanism — other programmers can easily extract and integrate them in their projects.

In addition, they also wrote a thorough description of their project for an audience that has little to no knowledge of the Jolie language. In the document they both illustrate their design decisions and which features of the Jolie language they used in the implementation.

A genuine thank to Matteo, Arnaldo, and Salvatore which provided a significant contribution to the Jolie community. Keep up the good work!

December 01, 2017

Posted by Fabrizio Montesi

Instant web search results from DuckDuckGo in Jolie

DuckDuckGo has an API for instant search results. It’s really easy to invoke and it can reply with JSON or XML data (here I use XML).

Here’s a snippet that uses it.

include "console.iol"

outputPort DuckDuckGo {
Location: "socket://api.duckduckgo.com:443/"
Protocol: https {
  .osc.search.alias = "?q=%{q}&format=xml&t=jolie_example";
  .osc.search.method = "get"
}
RequestResponse: search // I feel lazy, so I'm using dynamic typing
}

main
{
  // Get the results
  search@DuckDuckGo( { .q = args[0] } )( response );

  // Print results
  println@Console( "# Found " + #response.Results.Result + " result(s)" )();
  for( result in response.Results.Result ) {
    println@Console( "\n" + result.Text + "\n" + result.FirstURL )()
  };
  println@Console()();

  // Print related topics
  println@Console( "# Found " + #response.RelatedTopics.RelatedTopic + " related topic(s)" )();
  for( topic in response.RelatedTopics.RelatedTopic ) {
    println@Console( "\n" + topic.Text + "\n" + topic.FirstURL )()
  }
}

Just save this as a file, say search.ol, and then you can use it to fetch some results. For example, try jolie search.ol DuckDuckGo or jolie search.ol "Jolie Programming Language".

A note about the rules from DuckDuckGo at the time of this writing. If you use this in some project, remember to update the t parameter in the alias of operation search from jolie_example to something that describes your own project.

November 28, 2017

Posted by Fabrizio Montesi

Invoking Google Analytics from Jolie

Do you need to send events to Google Analytics from your Jolie service? Here is a self-explanatory snippet.

interface GoogleAnalyticsIface {
RequestResponse: collect
}

outputPort GoogleAnalytics {
Location: "socket://www.google-analytics.com:80/"
Protocol: http { .format = "x-www-form-urlencoded" }
Interfaces: GoogleAnalyticsIface
}

main
{
	collect@GoogleAnalytics( {
    .v = 1,
    .tid = "UA-XXXXXXXX-Y", // put your own tracking id here
    .cid = 555,
    .t = "event",
    .ec = "Publications",
    .ea = "Download",
    .el = "mypaper.pdf"
  } )()
}

To know what the fields mean, have a look at the official documentation from Google: https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters.

I myself find it useful to send events from web servers or other microservices. That snippet is from my own website written in Jolie, to track downloads of my research papers. (Sending these events from the server has the nice consequence that you do not track extra information from the web browser.)

Enjoy!

July 14, 2017

Posted by Claudio Guidi

Automating container environments for testing microservices

Testing microservices is a fundamental task in a microservice oriented system and containerization offers a great opportunity for automating it. Containers can be created and connected on demand, thus they provide the perfect environment where performing tests because it can be created and then destroyed when the tests are terminated. Moreover, we could create a testing environment system which is the exact copy of the production one.

In my previous post I showed Jocker, a Jolie component able to interact with Docker by offering a subset of its functionalities as Jolie operations instead of REST ones. In this post I am going to exploit Jocker for automatizing a test on a simple Jolie microservice by orchestrating it from another jolie orchestrator. In order to do that I will use a simple example you can find in the jocker git repo under the folder: ExampleOrchestrator/TestingDBSystem

The system under test
The system under test is very simple and it is just represented by a microservice connected with a PostgreSQL database.



You can find the code of this simple Jolie microservice here. As you can see, this microservice has only one RequestResponse operation called getContent which is in charge to select the field of a row (field2) from table testTable of the DB depending on the value of column field1. Very simple.

Orchestrating the Test
Now, I'll show you how to test the microservice by checking if it properly returns the correct responses when some request messages are sent. In order to do so I use this simple orchestrator for interacting with Jocker which executes all the actions I need. In particular, the orchestrator performs three main activities:
  • Preparation of the testing environment
  • Test execution
  • Removal of the testing environment
Preparation of the testing environment
The main idea is to prepare the testing environment by creating a container for each basic component of the system. Here we have two basic components: the PostgreSQL database and the Jolie microservice.

The PostgreSQL Database container is obtained in the following way:
  1. pulling down the postgresql image from docker hub
  2. creating the corresponding container
  3. starting the container
  4. initializing the database we need by creating it from scratch
  5. initializing the required known data into the database
Steps 4 and 5 could be skipped if we consider to already have a postgresql test image with a pre-installed database initialized with the required data.

On the other hand, the Jolie microservice image can be built by following the same steps explained here. In particular:

  1. a temporary directory is created and all the content of the ServiceToTest folder is copied in. 
  2. a Dockerfile is dynamically created and added to the temporary folder
  3. a tar file of the temporary folder is created 
  4. a docker image of the microservice is created invoking Jocker
  5. a container is created starting from that image
  6. the container is started
The final environment is a system like the following one where the two basic components are now encapsulated within two different containers:




Test execution
The test execution is a very simple phase because the orchestrator just sends all the requests I want to test to the microservice inputPort and checks if the results are like expected. For the sake of this example there is only one request to test but, clearly, they can be hundreds depending on the operations to test and the variety of data to be considered.

Removal of the testing environment
When the test is done, I don't need the test environment any more and I can destroy it. In particular:
  1. I stop the two containers
  2. I remove the two containers 
  3. I remove the two images (in reality, as you can see in the code, I do not remove the postgresql image just because it takes time for pulling it down from the docker hub, but it is up to you).

Some interesting notes
  • In this example we do not exploit the possibility to create links among different docker containers but we directly call the container on the ports they provide. In order to do this, once a container is created we also query Jocker in order to get their info details for extracting the local IP assigned by Docker to them. We will use these IPs as reference hosts for connecting the microservice with the PostgreSQL database and the tests to the microservice.
  • In order to run the example it is sufficient to run the jocker container as described here and the run the orchestrator placed in the folder  ExampleOrchestrator/TestingDBSystem by using the following command

    jolie orchestrator.ol

Conclusions
I hope this post could be of inspirations for all those software engineers who are addressing testing issues with microservices and containers. Let me know if you have questions or doubts.













July 06, 2017

Posted by Claudio Guidi

Jocker: orchestrating Docker containers with Jolie

recently we spent time in integrating Jolie and Docker. Our idea was very simple: since Jolie is a very good language for orchestrating microservices in general, why not use it also for orchestrating docker containers??

Thanks to Andrea Junior Berselli who started to work on this topic during his University degree at the University of Bologna, we can now say that a first component able to integrate Docker with Jolie exists! Its name is Jocker [github project]!

How does Jocker work?
Jocker is a jolie microservice which is able to call the REST API of Docker (we implemented only a subset so far) and it offers them as simple jolie operations thus avoiding to deal with all the details related to rest json calls. Here you can see the jolie interface of Jocker. The architecture is very simple:


Jocker must be executed in the same machine where docker server is running. It is communicating by exploiting localsocket //var/run/docker.sock and it will supply jolie operations in the default location localhost:8008 with protocol sodep.

Jocker container
The easy way for running jocker is to pulling down its container and then starting it. The Jocker image is available at jolielang section on Docker Hub and it can easily pulled down by using the followingcommand:

docker pull jolielang/jocker

When pulled down run the following command for executing it:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 8008:8008 jolielang/jocker

Jocker from sources
If you want to run Jocker from sources, you need some extra steps before continuining:
  • you need to install Jolie in your machine 
  • you need to install libmatthew Java libraries in order to enable localsockets.
Running Jocker is very simple, just go into the jocker folder and then type the followin command:

jolie dockerAPI.ol

Jocker listening location can be changed by editing file config.ini.

Jocker Clients
It is very easy to interact with Jocker, just create the following outputPort in your Jolie microservice and use it as usual:

outputPort DockerIn {
    Location: "socket://localhost:8008"
    Protocol: sodep
    Interfaces: InterfaceAPI
}


where InterfaceAPI can be downloaded from here. As an example you can request for the list of all the containers with the following client:

include "console.iol"
include "string_utils.iol"
include "InterfaceAPI.iol"

outputPort DockerIn {
    Location: "socket://localhost:8008"
    Protocol: sodep
    Interfaces: InterfaceAPI
}

main {
    rq.all = true;
    containers@DockerIn( rq )( response );
    valueToPrettyString@StringUtils( response )( s );
    println@Console( s )()
}


In the github repository of the project there are some sample clients you can use for testing Jocker.

Enjoy Jocker and, please, let us know comments and suggestions for improving it.

June 29, 2017

Posted by Balint Maschio

MongoDB Jolie Connector Part 3 : ObjectID creation and cross collections handling

In the last year I have been working on the development of  MongoDB connector for Jolie ( First Language for microservices ). In my previous two posts I have been looking into the basic CRUD operations and how to program them with Jolie.
In this post I am going to look at the handling of ObjectID and cross collections relationships. The two topics are closely related as MongoDB used ObjectID to cross reference documents between collections.

Assigning ObjectID to a document and adding ObjectID reference to a document.

ObjectID is a MongoDB specific data type that is not native to Jolie , therefore like in previous cases where there was not direct correspondence between MongoDB data type and Jolie data type a  the following semantic solution was adopted  


The node "@type" signal to the connector that it needs to handle the value of the node "_id" as a ObjectID. The node "_id" it is a specific node name that will trigger the the assignment of a unique ObjectID to the document as shown in the following example 


The response from the insert operation will return an structure identical to that used to insert the ObjectID


Of course one should really consider the opportunity of setting an objectID with a custom value, considering the all the drawbacks connected with handling the uniqueness of the value.
If the "_id" node is not defined the insert operation will return the auto generated value; that can be used to create documents cross referencing.
In a similar manner we can set one of the document fields as ObjectID to reference an other document as shown in the code under .


The resulting document looks like


We can see that the resulting document contains the auto generated "_id" and the reference to another object that is desired result.It is also clear that any time we can update the document adding further documents reference as shown in the following code.



The document now looks like


Cardinality of the relationship

When approaching MongoDB from a SQL prospective the first question came to my mind was "what about schema". In the excellent post "6 rules of thumb for MongoDB schema design" the author goes back to basic identifying three types of possible relationships:

  1. One to Few
  2. One to Many
  3. One to Squillions
The first  is not really strictly pertinent to the topic of this post but it comes without saying that One to Few can be easily model with an embedded document.
The renaming two are more pertinent: and I will try to demonstrate to code these relationships with Jolie
In my example  are present 3 collections that model a small part of a e-commerce solution:
  • CustomerData
  • CartData
  • SearchData
The CustomerData document ( see below ) will contain an array field cartHistory containing the reference to CartData document
   


The document for CartData( see below ) also contains a reference to a forth collection ProductData that will be not specified in this post.


In both cases the documents are modeling a One to Many relationship The SearchData document ( see below ) acts as log of all the search activity logging the visited object by a the customer


The SearchData document is design to model One to Squllion or better Squillion to one

One to Many


We can imagine a situation where by on the end of the purchase the CustomerData will be updated adding the concluded CartID the code below shows how this would be done in Jolie


Of course this code is simplified and it does not consider any exceptions compensation or business logic consideration, but it aims to show how is possible to manipulate MongoDB ObjectID across collections using Jolie


One to Squillions

Although the used data model is probably not the most correct to keep track of the navigation history it is a simple example to give the idea how handle this problem in Jolie 


Of course one should consider now how to search the data of this potentially enormous collection and how efficient this search may be but it is not topic of this post.

Posted by Balint Maschio

MongoDB Jolie Connector Part 2: Application Example

In the previous post I looked at the basic use of Jolie's MongoDB connector with an overview of its characteristic and it basic uses. In this post we are looking at a real use of the connector, the obvious choice would have been the implementation of Sales and Distribution application; yet in the company where I work such a need has not arisen. The opportunity to use MongoDB and jolie have arisen when approving the development of a Plant Management software. Production plants are often very complex structures with hundreds of  components. The best way to represent such complex structures is by using a tree representation  where the hierarchical nature of plant and its sub component can be best expressed. The tree data structure also fits perfectly with jolie data structures and with MongoDB document approach.
Therefore the first step in the development process was to define a "document" or better a tree data structure that represents my company production plants.  
     
 type ProductionSiteType {  
.productionSite:void{
.productionSiteName:string
.productionSiteErpCode:string
.mainLine*:void{
.mainLineName:string
.mainLineErpCode:string
.subLine*:void{
.subLineName:string
.subLineErpCode:string
.equip*:void{
.equipName:string
.equipErpCode:string
}
}
}
}
}
}

The second issue was to valuate how flexible this data structure is? can it adapt to changes required by the business , with the insertion of  new information. Let us suppose that the business requires to specify a mainLineManager (under the mainLine node ) with his/her contact details, and also it is required to supplierMaterialCode. The compulsiveness of the information can be now handled using the jolie  node's cardinality.

       .mainLineManager:void{  
.managerName:string
.email:string
.phone:string
}

The insertion of this node node will require the data within the collection to be  processed again

        .equip*:void{  
.supplierMaterialCode?:string
.equipName:string
.equipSapCode:string
}

Where in this case there will be no need for the collection to be processed again
So far the reader may think that the writer has only shown  how to represent a possible plant structure using jolie.
The advantage of the document approach will become evident when looking at the interface and the operation

interface MaintanaceServiceInterface{
RequestResponse:
addProductionSite( ProductionSiteType )(AddProductionSiteResponse)
}

It can be seen that any change in the document is propagated at the addProductionSite operation, and it is also evident looking at the implementation,

   [addProductionSite(request)(response){
scope (addProductionSiteScope){
install (default => valueToPrettyString@StringUtils(addProductionSiteScope)(s);
println@Console(s)());
q.collection = "SiteStructure";
q.document << request;
insert@MongoDB(q)(responseq)
}
}]{nullProcess}

The same can be said for the reading of plants structures


type GetMainLineRequest: void{
.productionSiteErpCode?:string
}
type GetMainLineResponse:void{
.document*:productionSiteType
}

interface MaintanaceServiceInterface{
RequestResponse:
addProductionSite( ProductionSiteType )(AddProductionSiteResponse),
getProductionSite(GetProductionSiteRequest)(GetProductionSiteResponse)
}


With the following implementation


[getProductionSite(request)(response){
scope (getProductionSiteScope){
install (default => valueToPrettyString@StringUtils(getProductionSiteScope)(s);
println@Console(s)());
q.collection = "SiteStructure";
if ( is_defined (request.productionSiteErpCode)){
q.filter << request;
q.filter = "{'productionSite.productionSiteErpCode': '$productionSiteErpCode'}"
};
if (DEBUG_MODE){
valueToPrettyString@StringUtils(q)(s);
println@Console(s)()
};
query@MongoDB(q)(responseq);
if (DEBUG_MODE){
valueToPrettyString@StringUtils(responseq)(s);
println@Console(s)()
};
response << responseq
}
}]{nullProcess}


Once again it can be noticed that the filter is expressed as MongoDB standard query language with the injection of the data directly from the jolie type.
It has been also necessary to refactor some of the subNode such as the equip defining a not inline type



type equipType:void{
.equipName:string
.equipSapCode:string
}

type AddEquipmentRequest {
.productionSiteErpCode:string
.mainLineErpCode:string
.subLineErpCode:string
.equip: equipType
}
interface MaintanaceServiceInterface{
RequestResponse:
addProductionSite( ProductionSiteType )(AddProductionSiteResponse),
getProductionSite(GetProductionSiteRequest)(GetProductionSiteResponse),
addEquipment( AddEquipmentRequest )(AddProductionSiteResponse)
}


By defining a new jolie complex type it has been possible centralizing the definition of the equipment that can be used both as type for further operations or directly as new document for a new collection
An other example of the usefulness of the type/document  driven design is the access to subNodes content as individual nodes


type GetMainLineRequest: void{
.productionSiteErpCode?:string
}
type GetMainLineResponse:void{
.document*:productionSiteType
}
interface MaintanaceServiceInterface{
RequestResponse:
addProductionSite( ProductionSiteType )(AddProductionSiteResponse),
getProductionSite(GetProductionSiteRequest)(GetProductionSiteResponse),
addEquipment( AddEquipmentRequest )(AddProductionSiteResponse)
}


[getMainLine(request)(response){
scope (getMainLineScope){
install (default => valueToPrettyString@StringUtils(getMainLineScope)(s);
println@Console(s)());
q.collection = "SiteStructure";
q.filter[0] = "{ $match : { 'productionSite.productionSiteErpCode' :'$productionSiteErpCode' } }";
q.filter[0].productionSiteErpCode = request.productionSiteErpCode;
q.filter[1] = "{$unwind : '$productionSite'}";
q.filter[2] = "{$unwind : '$productionSite.mainLine'}";
if (DEBUG_MODE){
valueToPrettyString@StringUtils(q)(s);
println@Console(s)()
};
aggregate@MongoDB(q)(responseq);
if(DEBUG_MODE){
valueToPrettyString@StringUtils(responseq)(s);
println@Console(s)()
};
response << responseq


}
}]{nullProcess}


By using the aggregation operation it has been possible to extract the content of the subNode mainLine for a specific plant and creating the correlation one document for each mainLine.

With in the limit of blog post I hope the reader will appreciate how jolie and jolie's MongoDB connector are a valid  technology to approach microservices  based data intensive applications.

May 03, 2017

Posted by Balint Maschio

Implementing IT solution for customer loyalty program using a microservices approach

General view

We are all familiar with the concept of  loyalty program, and probably we all are members of at least of one or two of these programs. Designing and implementing a reliable and flexible IT solution, that handles such program, may result more complex that expected.
In the recent few months I had to coordinate a small team of developers in the design and development of  such a solution for the company I work for .
The company that I work for operates in the media industry with and in focus on newspapers both classical and online.  
From the start it was evident that the implementation of this project presented the following constrains :
  • It needed to operate in well establish IT ecosystem 
  • It needed to be capable of adapting itself to the new marketing demands
  • It needed to be able to communicate and exchange data with third party software
It is my opinion that to match all three contains the best approach is  the microservices one , with the following  microservices based architecture is described by the diagram  :  




External API Service

This service provides access to data regarding the loyalty program and marketing promotion and other customer management operation. The API have been published as REST API , being REST a simpler meteorology of interaction with third parties software.

WEB Server

The solution provides the user with a website where to check is loyalty program status and interact with the marketing department The instance of  the WEB server was generated  from the  Jolie Leonardo WEB Server template and evolved to fit with the project requirements

Integration with third part Web CMS

One of the project requirement was the integration with the the Applix Polopoly WEB CMS , the customer web portal had to be embedded in standard web article of our on-line newspaper
The solution adopted was to spiting the technology stack in two all the presentational  files (HTML and CSS ) were deployed on Polopoly and the JavaScript provided by the Jolie WEB Server.
It was also necessary modify the HTTP message response header to handle access control issues adding in the HTTP port configuration  parameters as shown  in following code


Protocol: http{

.osc.OperationName.response.headers.("Access-Control-Allow-Origin") -> SiteOrigin;
.osc.OperationName.response.headers.("Access-Control-Allow-Headers") = "Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With";
.osc.OperationName.response.headers.("Access-Control-Allow-Credentials") = "true"

}

SiteOrigin is variable that should  be set to the location of the caller

Integration with an Authentication Service Provider 

In order to provide a secure and reliable process of authentication  Auth0 has been identified as our authentication  provider. Auth0 provides social and Gmail integration and several others interesting features.
Auth0  requires the client WEB Server to publish a REST callback operation to complete the login or registration process.
As in the case of the WEB CMS integration it was required to define specific HTTP message response hearder value to handle HTTP handle access control issues and  cookie policies adding in the HTTP port configuration  parameters as shown  in following code

Protocol: http{
   .osc.callback.response.headers.("Set-Cookie")->sid;
   .osc.callback.response.headers.("Access-Control-Allow-Origin") -> Auth0Site;
}


The variable sid is passed via the response Header and it allows to set a value for the cookie that will be used by the WEB Server to  provide user dependent data.
The cookies is recovered and piped in in the operation request value using the following code

Protocol: http{
 .osc.OperationName.cookies.AuthCode = "sid"
}

This will free the frontend developer from having to pass as value the AuthCode in the calls. and also it allows the backend developer to manage at port level and operation level the use of cookie

Technology stack 

The technology stack used in the web solution is the following

  • HTML5 /CSS
  • jQuery 
  • Jolie (server side)
The interaction between the HTML page and the backend operation is achieved via AJAX call. The implementation of all the JQuery function and it the operations request response types is generated automatically by a tool
Handling the asynchronicity of the AJAX is achieved using a event driven model. the tools includes in the generated code an specific event for each operation. 
   
function operationName(value) {
    $.ajax({
        url: '/operationName',
        dataType: 'json',
        type: 'POST',
        contentType: 'application/json;charset=UTF-8',

        success: function(response) {
            if (typeof(response.error) == 'undefined') {

                responseObj = new OperationNameResponse()
                responseObj.parseJson(response)
                var event = new CustomEvent('EventOperationNameResponse')
                event.data = responseObj
                document.dispatchEvent(event)
            } else {

                var eventError = new CustomEvent('EventGeneralError')
                eventError.data = response
                document.dispatchEvent(eventError)
            }

        },

        error: function() {
            var eventErrorHttp = new CustomEvent('EventGeneralHttpError')
            document.dispatchEvent(eventErrorHttp)
        },
        data: JSON.stringify(value.toJson())
    })
}


In this way the frontend programmer will be able to work with the the operation published by the WEB Server by calling operation or methods of classes in js


handler_EventModifyAnagraphicResponse(e){
    operationNameResponse = e.originalEvent.data
   /*   insert your code here */
}

$(document).ready(function() {
              $(document).on('EventGeneralError', handler_EventGeneralError);
              $(document).on('EventOperationNameResponse', handler_EventOperationNameResponse);
 }

also it will be able to handle the data from the call in an event handler function
The use of this tool has allowed the team to increase the efficiency of the Web Application development achieving a good separation between the different roles in the developing team  .

Business Logic Service  

This service is the fulcrum of the solution it handles the following aspects 
  •  Logic for the loyalty program
  •  Logic for the creation/update and recovery of customer data
  •  Logic for the creation/update and recovery of distribution data
Each of this aspects has generated a separated interface definition, to make make simple the separation of the implementation when required 
The business logic service acts as an coordinator  of internal and external data to achieve the desired business objective. The internal data are masked behind specific CRUD services to increase the decoupling between data and logic.

CRUD Services  

The CRUD Services mask data from existent database. The choice of developing specific services to publish internal data has been driven by the desire to make such data available to other future application.

Third Party Connectors 

Some of the data required by the application are provided by external service of the following list 
All three implementations are center of the use of a HTTP Jolie output port, such a port acts as a HTTP client .
The process of implementing a integration to a Third Party REST Service requires the following steps

  1. Definition of the interface and its operation ( it can be also a subset of the provided operation)
  2. Definition of the request and response type 
  3. Definition of the port parameters such as  Method Aliases ecc ecc
Using this simple methodology it has been possible developing the connector that now can be used for any other application it  may be needed for. 

Google Geocoding API

It was necessary to implement a geo localization process to populate the distributor and shop data with the their latitude ad longitude coordinate  In this way the marketing team will be able to add a new shop or distributor without worrying to provide the localization data.
These data will then available via web application to the customers searching for shop or distributor in his/her area.

Auth0

In the WebServer section it was mentioned the Auth0 as authentication  provider.  It was also necessary to implement the integration serverside with some of the API provided by Auth0. This has made possible recovering user and token info.

Prestashop 

It was also necessary to implement the integration with our instance of Prestashop, The business requirement was to provide  to our customer a feedback on the use of their loyalty points via our e-commence allowing the customers to see customer on run-time the status of its discount voucher and the history of his/her purchases

April 18, 2017

Posted by Claudio Guidi

What is a microservice? (from a linguistic point of view)


As we described here, with Jolie we are pioneering the linguistic approach for dealing with microservices. Our idea is that microservices are introducing a new programming paradigm which can be crystallized within a programming language. The focus of this post is about the definition of microservice starting from our point of view: a linguistic point of view.

A service is the single unit of programmable software
In the last years the concept of service has been investigated in the area of Service Oriented Computing and several definitions have been provided for defining service contracts, service providers, service discovery and so on. All these definitions are quite abstract because services have been conceived to be technology agnostic both in the case of SOA and microservices. Such a fact means that it is possible to develop a service in any given technology. They say that services are technology agnostic.

Technology agnosticism is a very important feature which allows us to engineer a software system independently from any technology lock-in. But our purpose here is to give the definition of a service as a single unit of programmable software which cannot be fragmented in sub-parts. For this reason, you will find that all the definitions I am going to provide here are strongly related to a specific technology: Jolie. If you like to see how we chose to model the service oriented programming paradigm in a single language you can continue to read, otherwise you can skip this post. If we are wrong, or we are missing some points, or you know other technologies which match the definitions please write us your feedbacks. We are very exited to share ideas on this open topic.

As a starting point, let me explain the first assumption we made in the linguistic paradigm: the service is the single unit of programmable software. Usually, a service is always obtained by programming a server (it is not important if it is simple or not) joint with some business logic which represent the functionalities to serve:

                             SERVER + BUSINESS LOGIC = SERVICE

In a linguistic paradigm such an equation is not more valid just because servers do not exist. Only services exist. It is not possible to program a server because you can program only services. So, forget servers (do not confuse with serverless, it is a different approach). So, if there are no servers but only services, what is a service? As it happens for Object Orientation where classes are logical definitions and objects are the instances of classes in a running environment, let me call the logical definition of services with the term service and its running instance with the term microservice.

                                     SERVICE --> MICROSERVICE

For each service there could be more than one microservices, but each microservice is just the running instance of a service. The service is the single unit of programmable software. In the following I am going to build the definition of a service by giving some qualities it has to provide. At the end of this post I'll give the definition of service.

Services exchange messages
The only way for exchanging data among services are messages. There are no other way. A message is just a limited portion of data transmitted in a limited portion of time. A service can both receive messages and send messages. In a SOA a service which receives and sends messages is usually called orchestrator. Such a difference in the linguistic paradigm does not exist, an orchestrator is just a service. In particular, in Jolie message exchange is enabled by means of ports. Messages are received by means of inputPorts and they are sent by means of outputPorts. Similar constructs are used in WS-BPEL, they are called partnerLinkTypes.

Services can have a behaviour
The behaviour defines the business logic to be executed when a message is received.  In the behaviour it is possible to compute the received data and/or send messages to other services. In Jolie the behaviour is expressed in scope main. A behaviour can define different operations. Different business logics can be joint to different operations. In Jolie multiple operations can be expressed in the same behaviour by using the non deterministic operator:

main {
[ op1( req1 )( res1 ) {
    businessLogic1
}]  

[ op2( req2 )( res2 ) {
    businessLogic2
}]  

...

[ opn( reqn )( resn ) {
    businessLogicn
}]  
}

An operation must always express a finite computation in time. In other words, when triggered an operation must always reach an end state. I say that an operation is divergent if its behaviour defines an infinite computation. Jolie allows for the definition of divergent operations by defining infinite loops:

divergentOperation( request )( response ) {
    while( true ) {
           nullProcess 
    }
}

Divergent operations are deprecated in Jolie.

Services declare interfaces
The operations of the behaviour must be declared in a machine readable interface. The interface defines all the available operations of a given service. Interfaces are used as a concrete contract for interacting with a service. In Jolie interfaces are also equipped with message type declarations.

type MyRequestType: void {
    .msg: string
}

type MyResponseType: void {
    .resp_msg: string
}

interface myInterface {
RequestResponse:
     myOperation( MyRequestType )( MyResponseType )
}

Services execute sessions
Services execute sessions for serving requests. A session is a running instance of a service operation. Sessions are independently executed and their data are independently scoped. If we suppose to send three messages to a microservice which implements the following service we will receive three different messages for each request message.

main {
   test( request )( response ) {
      response = request
   }
}

if we concurrently send three request messages with content "1", "2", "3", we will receive three related reply messages with the same content.


A definition of service

Here I try to summarize a definition of service starting from the basic concepts highlighted above. Jolie provides more linguistic elements w.r.t. those highlighted here. Maybe more basic concepts could be added to the definition and other should be removed. This is just a starting point for trying to investigate microservices from a linguistic point of view. All the contributions are encouraged!

A service is a unit of programmable software able to exchange messages with other services whose behaviour is triggered by incoming messages and it is defined by a set of finite computation logics called operations which are declared within a machine readable interface. A running instance of a service is called microservice whose inner instances of triggered operations are called sessions. Sessions are executed independently and their variables are independently scoped.









November 12, 2016

Posted by Fabrizio Montesi

Service Decorators

TL;DR: decorators for services are pretty easy to write in Jolie, thanks to a native feature called aggregation. Be careful in the deployment phase for managing performance hits!

Service Decorators

Just read this article about using decorators over inheritance in object-oriented programs. I assume that you know decorators in the following. If you don’t, get informed (e.g., by reading that article) before proceeding.

The Decorator design pattern offers a way to cleanly modify the behaviour of an object by using composition instead of inheritance. I won’t enter in the merits of composition against inheritance in object-orientation (both have their own, depending on the other features of the language), because I’m interested in microservices here. In microservices, composition is typically the only option you have. That’s because your microservices may be written in different languages, or even paradigms. And even if their codebases were somehow compatible, inheritance would still be out of place: your microservices can only depend on the communication APIs of other services, not on their implementation details (e.g., by looking at their source code).

Since decorator acts through composition, this pattern is potentially interesting for service programming. But as the author of the article linked above mentions, using it can be frustrating since it requires a lot of boilerplate code. This makes it also error-prone. As a running example, let’s look at the e-mail service interface proposed in that article, rewritten in Jolie:

interface EmailServiceIface {
RequestResponse:
  send(Email)(void),
  listEmails(Range)(EmailInfoList),
  downloadEmail(EmailInfo)(Email)
}

Decorating an operation

Now, suppose that this service is available to us as EmailService. Assume also that we want to write a new service that decorates EmailService with the following logic: whenever send is called, we check if we have sent an “important” e-mail (e.g., the subject contains a specific keyword telling us that the e-mail is important, or the addressee is in a special list); if an important e-mail has been sent successfully, we backup its content by calling another service, for indexing and safe-keeping.

A naive implementation is to write the decorator by re-implementing interface EmailService, like this:

inputPort EmailServiceImportant {
Location: MyLocation Protocol: MyProtocol
Interfaces: EmailServiceIface
}

main
{
  [ send( request )( response ) {
    send@EmailService( request )();
    check@Important( { .subject = request.subject, .to = request.to } )( important );
    if ( important ) {
      backup@BackupService( request )()
    }
  } ]

  [ listEmails( request )( response ) {
    listEmails@EmailService( request )( response )
  } ]

  [ downloadEmail( request )( response ) {
    downloadEmail@EmailService( request )( response )
  } ]
}

The code for listEmails and downloadEmail is boilerplate, since we’re just forwarding requests and responses. The author of the article suggests that it would be nice if languages supported a native feature that makes writing this code unnecessary. Luckily, we have it in this case!

Forwarding is a native feature in Jolie, since building proxies is the bread and butter of service composition (think of load balancers, caches, etc.). We can rewrite our decorator like this:

inputPort EmailServiceImportant {
Location: MyLocation Protocol: MyProtocol
Aggregates: EmailService // Aggregates instead of Interfaces!
}

main
{
  [ send( request )( response ) {
    send@EmailService( request )();
    check@Important( { .subject = request.subject, .to = request.to } )( important );
    if ( important ) {
      backup@BackupService( request )()
    }
  } ]
}

No boilerplate, same behaviour as our previous decorator!

Interface Decoration

What if you want to change the behaviours of many operations, not just one? What if this behaviour change is always the same? For example, suppose that we want to write a decorator that keeps track of all events: whenever an operation is called, we write this in an external log.

Here’s a naive implementation:

inputPort EmailServiceLogger {
Location: MyLocation Protocol: MyProtocol
Interfaces: EmailServiceIface
}

main
{
  [ send( request )( response ) {
    send@EmailService( request )();
    log@Logger( request )()
  } ]

  [ listEmails( request )( response ) {
    listEmails@EmailService( request )( response );
    log@Logger( request )()
  } ]

  [ downloadEmail( request )( response ) {
    downloadEmail@EmailService( request )( response );
    log@Logger( request )()
  } ]
}

Ouch, boilerplate again! What we need is the capability of writing that logging code just once, and applying it to the entire API of EmailService. That’s what courier processes in Jolie are for. Here’s the improved code, using courier:

inputPort EmailServiceLogger {
Location: MyLocation Protocol: MyProtocol
Aggregates: EmailService // Aggregates again
}

courier EmailServiceLogger // courier enables primitives for whole-interface behaviour
{
  [ interface EmailServiceIface( request )( response ) ] { // Apply to all operations in the interface
    forward( request )( response ); // forward is a primitive: it forwards the message to the aggreated (in this case, decorated) service
    log@Logger( request )()
  }
}

No boilerplate again!

Circuit breakers

What’s a cool example of a decorator? Circuit breaker! Yup. A sketch in Jolie using aggregation can be found in this paper.

Conclusions

As cool as decorators are, don’t forget that you are adding a layer of indirection. Many times, you don’t have a choice and your code benefits so much that you should pay the price. But in microservices, be very careful about how efficient your extra layer will be. If you stack too many decorators and they are all communicating remotely via sockets, you’ll soon have a performance problem. This is a deployment challenge: in Jolie, using a different communication medium doesn’t alter your behavioural code. So choose your communication media wisely when you deploy! If you have control over your stack of decorators, consider whether it would be better to put them all in one place and have them communicate using shared-memory (local location in Jolie).

October 12, 2016

Posted by Claudio Guidi

Fast Jolie microservices deployment with Docker

Jolie is now available on Docker and now it is possible to develop and run a microservice inside a container.

But what about the deployment of a microservice in Docker? How can we build a deployable docker container which includes the microservice we are working on?

Actually, it is a very easy task. It is sufficient to develop the microservice by following some simple rules and your docker image will be ready in few seconds!

Rule 1 : you need a Dockerfile for building an image of your microservice
First of all, create a file named Dockerfile in your working directory and write the following lines:

FROM jolielang/jolie-docker-deployer
MAINTAINER SURNAME NAME


where SURNAME, NAME and EMAIL must be replaced with the maintainer's surname, name and email respectively. The Dockerfile will be used by Docker for building the image related to your microservice. As you can see, the image you are creating is layered upon a previously created image called jolielang/jolie-docker-deployer.
You can find this image in the docker hub of jolielang here. Such a docker image, is prepared for facilitating the deployment of a jolie microservice as a docker image. In order to use it in the right way, just follow the next rules. As an example, let me suppose to deploy the following microservice saved in file helloservice.ol:

interface HelloInterface {
RequestResponse:
     hello( string )( string )
}

execution{ concurrent }

inputPort Hello {
Location: "socket://localhost:8000"
Protocol: sodep
Interfaces: HelloInterface
}

main {
  hello( request )( response ) {
        response = request
  }
}


Rule 2 : EXPOSE inputPorts ports
Remember that all the inputPorts of your microservice must be reached from outside the container, thus you need to expose them in the Dockerfile.



In the example the inputPort is located at localhost:8000, thus we need to add EXPOSE 8000 in the Dockerfile. So, your Dockerfile now becomes like this:

FROM jolielang/jolie-docker-deployer
MAINTAINER SURNAME NAME

EXPOSE 8000

Rule 3 : COPY the files of your project and define the main.ol
Now, everything is quite done for preparing the image, we just need to copy the files of the project in the docker image. When doing this, pay attention to rename the file which must be run with the name main.ol.

FROM jolielang/jolie-docker-deployer
MAINTAINER SURNAME NAME

EXPOSE 8000
COPY helloservice.ol main.ol

Building your image

When the Dockerfile is ready we can build the docker image of the microservice. In order to do this you just need to run the following command within your working directory which also contains the Dockerfile.

docker build -t hello .

where hello is the name we give to the image. Once it is finished, you can easily check the presence of the image in the local registry by running the following command:

docker images

Running a container
Now, starting from the image, you can run all the containers you want. A container can be run by launching the following command:

docker run --name hello-cnt -p 8000:8000 hello

where hello-cnt is the name we give to the container. Note that the parameter -p allows you to map the microservice port (8000) to the port 8000 of your localhost. You can check that the container is running just launching the following command which lists all the running containers:

docker ps

Your microservice is now deployed and it is listening for requests at port 8000. You can just try to invoke it with a client like the following one. Remember to launch the client in a separate shell of your localhost!



include "console.iol"

interface HelloInterface {
RequestResponse:
     hello( string )( string )
}


outputPort Hello {
Location: "socket://localhost:8000"
Protocol: sodep
Interfaces: HelloInterface
}

main {
  hello@Hello( "hello" )( response );
  println@Console( response )() 
}



Advanced settings
So far, we have deployed a very simple service but, usually we deal with microservices that are more complicated than the hello service presented before. In particular, it is very common the case where some constants or outputPort locations must be defined at deploying time. In order to show this point, let me now consider the following service:

interface HelloPlusInterface {
RequestResponse:
     helloPlus( string )( string )
}

interface HelloInterface {
RequestResponse:
     hello( string )( string )
}

execution{ concurrent } 

constants {
   CUSTOM_MESSAGE = " :plus!"


outputPort Hello {
Location: "socket://localhost:8000"
Protocol: sodep
Interfaces: HelloInterface
}

inputPort HelloPlus {
Location: "socket://localhost:8001"
Protocol: sodep
Interfaces: HelloPlusInterface 
}

main {
  helloPlus( request )( response ) {
        hello@Hello( request )( response );
        response = response + CUSTOM_MESSAGE
  }


This is a very simple microservice which has a dependency on the previous one. Indeed, in order to implement its operation helloPlus, it requires to invoke the operation hello of the previously deployed microservice. Moreover, it uses a constants CUSTOM_MESSAGE for defining a string to be added to the response string.




Usually, we would like that some of these parameters can be defined at deploying time because they directly deal with the architectural context where the microservice will run. Thus, we would like to create an image which is configurable when it is run as a container. How can we achieve this?

Rule 4 : Prepare constants to be defined at deploying time
The image jolielang/jolie-docker-deployer we prepared for deploying microservice has been built with some specific scripts which are executed before running the main.ol. These scripts just read the environment variables passed to the docker container and transform them in a file of constants which must be read by your microservice. The most important facts we need to know here are:
  1. Only the environment variables prefixed with JDEP_ will be processed
  2. The processed environment variables will be collected in a file of constants  named dependencies.iol. If it exists it will be overwritten.
From these two points we change the microservice as it follows:
 

interface HelloPlusInterface {
RequestResponse:
     helloPlus( string )( string )
}

interface HelloInterface {
RequestResponse:
     hello( string )( string )
}

include "dependencies.iol"

execution{ concurrent } 

outputPort Hello {
Location: JDEP_HELLO_LOCATION
Protocol: sodep
Interfaces: HelloInterface
}

inputPort HelloPlus {
Location: "socket://localhost:8001"
Protocol: sodep
Interfaces: HelloPlusInterface 
}

main {
  helloPlus( request )( response ) {
        hello@Hello( request )( response );
        response = response + JDEP_CUSTOM_MESSAGE
  }
}

As you can notice here there are two constants JDEP_HELLO_LOCATION and JDEP_CUSTOM_MESSAGE which require to be defined at the start of the microservice. They must be defined in the file dependencies.iol which MUST BE included in your microservice. This file just contains the declaration of the two constants.

constants {
JDEP_HELLO_LOCATION = "socket://localhost:8000",
JDEP_CUSTOM_MESSAGE = " :plus!"
}

During the development keep this file in your project and collect here all the constants you want to define at deploying time. When the service will be run in the container this file will be overwritten thus, you don't need to copy it into the docker image.

The Dockerfile of the helloPlus service is very similar to the previous one:

FROM jolielang/jolie-docker-deployer
MAINTAINER SURNAME NAME

EXPOSE 8001
COPY helloservicePlus.ol main.ol
We can create the image with the same command used before, but where the name is hello-plus.

 docker build -t hello_plus .

Configuring the container
Now, we just need to know how to pass the constants to the running container and everything is done. Docker allows to pass environment variables with the parameter -e available for command run. Thus the command is:

docker run --name hello-plus-cnt -p 8001:8001 -e JDEP_HELLO_LOCATION="socket://172.17.0.4:8000" -e JDEP_CUSTOM_MESSAGE=" :plus!" hello_plus

where hello-plus-cnt is the name we give to the container. Note that the constant JDEP_HELLO_LOCATION is set to  "socket://172.17.0.4:8000" where the IP is set to 172.17.0.4. It is just an example, here you need to specify the IP that Docker assigned to the container hello-cnt which is executing the service hello.ol. You can retrieve it just launching the following command:

docker inspect --format '{{ .NetworkSettings.IPAddress }}' hello-cnt

Once the hello-plus-cnt container is running, you can simply invoke it with the following client:

include "console.iol"

interface HelloPlusInterface {
RequestResponse:
     helloPlus( string )( string )
}

outputPort HelloPlus {
Location: "socket://localhost:8001"
Protocol: sodep
Interfaces: HelloPlusInterface
}

main {
  helloPlus@HelloPlus( "hello" )( response );
  println@Console( response )()
}


Conclusion
In this post I show how it is possible to deploy a microservice developed with Jolie as a docker container. The procedure is very easy, just pay attention to inputPorts and the constants you want to configure at deploying time. For all the other things you can just rely on the Jolie language. Don't forget that you can also exploit embedding for packaging more microservices into one, thus deploying all of them inside the same container if necessary.

Enjoy!

September 05, 2016

Posted by Claudio Guidi

An easy way to build microRESTservices with Jolie

This post is about a couple of tools [https://github.com/jolie/jester] we developed for facilitating the programming of REST microservices with Jolie. Personally I am not a big supporter of REST services, but I think that a technology which aims at being a reference in the area of microservices like Jolie must have some tools for supporting REST services programming. Why? Because REST services are widely adopted and we cannot ignore such a big evidence.

Ideally, Jolie as a language is already well equipped for supporting API programming also using http, but REST approach is so deep coupled with the usage of the HTTP protocol that it introduces some strong limitations in the service programming paradigm. Which ones? The most evident one is that a REST service only exploits four basic operations: GET, POST, PUT and DELETE. The consequence of such a strong limitation on the possible actions is that the resulting programming style must provide expressiveness on data. This is why the idea of resources has been introduced in REST! Since we cannot programming actions we can only program resources.

Ok,  let's go with REST services!

...But,
...here we have a language, Jolie, that is more expressive than REST because the programmer is free to develop all the operations she wants. From a theoretical point of view, in Jolie she can program infinite actions instead of only four!



- Houston we have a problem! We need to put infinite operations inside just four!!!

- Why only four when we can have infinite?

- This does not matter now Houston, we have to do it!



Ok no problem! Follow our instructions!
First of all, take note that here we want to preserve all the benefits of using Jolie, thus the possibility to develop all the operations we want, but finally publish the microservice as a REST service. We achieve such a result by exploiting a specific microservice architetcure which is a composition of a router and the target microservice we want to publish as a REST one. The router, as described in Fabrizio's paper [https://arxiv.org/abs/1410.3712], is in charge to transform REST calls into standard Jolie calls.



Ok, in order to exaplain how to proceed, let us consider as a target microservice the demo one reported in the jester project [https://github.com/jolie/jester/tree/master/src/jolie/tools/demo]. It is a very simple service which emulates a manager of orders which supplies four operations: getOrders, getOrdersByItem, putOrder, deleteOrders.

Ok, there are four operations but it is just an example, we could have more than four operations :-). The main question now is:

What we have to do for transforming these operations in REST operations?
We need to use the jolie2rest.ol tool you can find the jester project. Such a tool analyzes the interface of the demo service and it extracts a descriptor for the router which enables it to publish the demo service as a REST service.

Very simple! But before running the tool we need to know something more. We need to define how to convert the single operations of the demo microservice will be tranformed into the REST one. In particular, fro each target operations we need to specify if we want a GET operation, a POST operation, a PUT operation or a DELETE operation. It is possible to provide these instructions, directly into the interface of the demo service by exploting the annotation @Rest into the documentation comments. As an example let us consider the operation getOrders:

/**! @Rest: method=get, template=/orders/{userId}?maxItems={maxItems}; */
getOrders( GetOrdersRequest )( GetOrdersResponse )


The annotation defines that the operation getOrders it must be transformed into a GET http method and the URL template that must be adopted is /orders/{userId}?maxItems={maxItems}. What is the template URL?
Since the REST services deal with resources, the URL is used as a means for expressing the resource we want to access. Here we use the template as the means for transforming a call to an operation of the target service into a resource. In particular, the parameters between curly brackets will be directly filled by using the corresponding values of the request message node which is defined in the GetOrdersRequest type:

type GetOrdersRequest: void {
    .userId: string
    .maxItems: int
}


Now, we can proceed by running the following command:

jolie jolie2rest.ol localhost:8080 swagger_enable=true

where localhost:8080 is the location where the router is deployed and the parameter swagger_enable specifies if we want to enable the creation of the swagger json descriptor file.
Note: the file service_list.txt contains the list of the target microservices to be transformed and the related inputPorts to be converted (more instructions can be found here: ref).

As a result the tool will generate two files:
  • router_import.ol
  • swagger_DEMO.json
The former file must be copied in the router folder and it contains the script which enables the router to transform the REST calls into the operations call of the target service. The latter is just the swagger descriptor to be provided in input into a SwaggerUI. You don't have a local SwaggerUI available? Follow these instructions to have it locally, otherwise go to point 6:



  1. Prepare the web server of the SwaggerUI application by downloading Leonardo from here [https://github.com/jolie/leonardo]
  2. Go to this SwaggerUI URL [https://github.com/swagger-api/swagger-ui/archive/v2.2.3.zip] and download the related web application project. 
  3. Copy the content of the folder dist of the SwaggerUI project inside the folder www of Leonardo.
  4. Open a shell and run jolie leonardo.ol
  5. Open a browser at the url http://localhost:8000, the SwaggerUI web application should appear.
  6. Copy the swagger_DEMO.json file into the folder where it is reachable from the SwaggerUI, in the Leonardo scenario put it inside the www folder.
  7. Write in the explorer bar of the SwaggerUI the URL for reaching the swagger descriptor, in the Leonardo case write: http://localhost:8000/swagger_DEMO.json
After this step, you should see the Swagger definition of your target demo service transformed into a REST one. In particular, you should see something like this:







As you can see the four operations of the target service have been transformed in the four different types of REST operations. Have a look to the interface annotations of the demo service for finding the matches with the Swagger interface!

Nice, but what happen if I want to transform more than one Jolie microservices instead of a single one?
This question is dealing with an architecture like the following one where the router is connected with several microservices:



There are no particular problems in achieving such a result. It is sufficient to list all the target microservices with the related input ports in the file service_list.txt and re-launch the jolie2rest tool!

Ok Houston! Here we are! But is it possible to avoid the usage of the router?
Yes, it is possible. But only if you accept to not completely adhere to the REST approach. A jolie microservice can be directly published by using a http inputPort with message format set to json [http://docs.jolie-lang.org/#!documentation/protocols/http.html#http-parameters]. In this case the microservice will be able to serve the http requests without requiring any router or proxy in the middle. If you want this, change the inputPort protocol of the demo service to a http one and then use the jolie2rest tool with the parameter easy_interface set to true.

jolie jolie2rest.ol localhost:8080 swagger_enable=true easy_interface=true

In such a case the router_import.ol file is not generated but only the swagger_DEMO.json one. The operations are all converted in POST methods and the URLs do not follow the templates but the request json messages must be entirely defined in the body of the message. Try to replace the file swagger_DEMO.json in the SwaggerUI and perform some calls.

Generating client stubs from an existing Swagger definition:
A last tool which can be very useful when integrating existing REST services in a Jolie architecture, is jolie_stubs_from_swagger.ol. Such a tool takes in input an existing Swagger definition and generates a Jolie client for each published api.

As an example you could try it by generating the clients for the petstore example supplied by the Swagger project [http://petstore.swagger.io/v2/swagger.json]. In order to do it, create a target folder where you want to store all the generated clients (for example petstoreFolder) and then run the following command:

jolie jolie_stubs_from_swagger.ol http://petstore.swagger.io/v2/swagger.json petstoreService petstoreFolder

where petstoreService is the token that will be used for generating the Jolie outputPort name of the petstore service inside the clients. As a result, in folder petstoreFolder you will find a list of Jolie clients. In particular, you will have a client for each api defined in the petstore swagger definition.

If you want to try to send a request just open one of them and create the request message. For example open the getOrderById.ol and prepare the request message by adding the following jolie code:

with( request ) {
.orderId = 8
};

then run the client as a usual jolie script:

jolie getOrderById.ol


the result should be printed out on the console!

You can exploit these clients inside your existing jolie microservices. Just note that the generated file outputPort.iol defines all the information necessary to declare the outputPort to be used. Thus just include it in your microservice project and then make the calls when is more useful for you!

Conclusion
Houston, everything is clear now! :-) With the REST tools described in this article we want to improve the Jolie language providing the possibility to publish Jolie microservices as REST services and giving an easy way for generating clients from existing REST services. We hope this could be useful for your  projects and, please, do not forget to send us your feedbacks and improvements!





July 04, 2016

Posted by Balint Maschio

MongoDB Jolie Connector Part 1

One of the buzz words of the IT community it is definitely "Big Data", yet like it often happens the actual definition is somewhat vague, is "Big Data" a question of volume of data or also a question of performances.The answer can be found ,in my opinion, in the definition provided by Gartner: "Big Data:is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation"
Does it means that to do "Big Data" we need to have a new DB technology: well  the short answer is yes we do, traditional DB technologies poorly fit  with some of the "Big Data" requirements. This new challenge has created a proliferation of new DB technologies and vendors with different approaches and solution. A good review on the several option has been provided by the following site Database technology comparison-matrix
MongoDB it is one of the possible technology choices, it is my opinion that with its document approach makes data modeling a much more linear approach easing also the not always comunication between IT and not IT members of a company.
The next question  is: "Can microservices play a role in a Big Data scenario?". It is my opinion  that microservices  can play an important role in developing data handling in a "Big Data" landscape. Expanding on the micro services orchestration concept, it can imagined a scenario where a poll of microservices  can be used as data orchestration where new "aggregated data" are created by using individual function exposed by the microservices. This will certainly cover two of characteristic of Gartner definition:
  1. high-variety information:  The orchestration of original data into new aggregated data will avoid the creation of new  database bounded table or collection 
  2.  cost-effective: microservices are effective as much we are able to decompose correctly the problem on hand
On the purpose of  bringing Jolie within the Big Data discourse a MongoDB connector has been developed: the choice of MongoDB as first Big Data technology approached  by the Jolie's team has not been accidental but driven by the tree like data representation used by MongoDB that extremity similar if not identical to the way Jolie represents complex data in its  complex type definition.
The connector has been developed externally the normal Jolie Language development but it is developed by the Jolie team members and it is also responds to a specific demands of a growing Jolie programming community.
The developing team has adopted a top down approach when developing the connector starting from an analysis of a set of requirement passing through the interface definition to arrive to some pilot project ( which  result will be presented further on ).
The identified requirement were  the following
  1.  Preserve the not SQL nature of MongoDB
  2. Give to the connector user a simple and instinctive interface to the MongoDB world
  3. Preserve some of the native aggregation and correlation characteristic of MongoDB
From this three requirements the following considerations have been thought 
  1. The connector will have to have all the CRUD operation 
  2. The connector will use the native query representational (JSON) 
  3. The connector will provide access to MongoDB aggregation capability
  4. The naming convention of  service operation types replicate the terms used by MongoDB,
and resulted in the following interface:


interface MongoDBInterface {
  RequestResponse:
  connect (ConnectRequest)(ConnectResponse) throws MongoException ,
  query   (QueryRequest)(QueryResponse)   throws MongoException JsonParseException ,
  insert  (InsertRequest)(InsertResponse)   throws MongoException JsonParseException ,
  update  (UpdateRequest)(UpdateResponse)   throws MongoException JsonParseException ,
  delete  (DeleteRequest)(DeleteResponse)   throws MongoException JsonParseException ,
  aggregate (AggregateRequest)(AggregateResponse)   throws MongoException JsonParseException
}



Like mentioned before the connector aims to propose a simple to use interface:
Starting from insert operation

q.collection = "CustomerSales";
    with (q.document){
          .name    = "Lars";
          .surname = "Larsesen";
          .code = "LALA01";
          .age = 28;
           with (.purchase){
             .ammount = 30.12;
             .date.("@type")="Date";
             .date= currentTime;
             .location.street= "Mongo road";
             .location.number= 2
          }
     };
As can be seen the document can be inserted int the "CustomerSales" collection by simply defining the document to be insert as Jolie structured variable.

The query operation or any other operation that require a subset filtering ( update , delete ,aggregate )

q.filter = "{'purchase.date':{$lt:'$date'}}";
q.filter.date =long("1463572271651");
q.filter.date.("@type")="Date";

notice the use of  '$nameOfVariable' to dynamically inject the value in search filter
Using the sane representation it can be inject the value for an update operation

q.collection = "CustomerSales";
q.filter = "{surname: '$surname'}";
q.filter.surname = "Larsesen";
q.documentUpdate = "{$set:{age:'$age'}}";
q.documentUpdate.age= 22; 

and so it goes for the aggregation:

q.collection = "CustomerSales";
q.filter = "{$group:{ _id : '$surname', total:{$sum : 1}}}";

Use and limitation of the connector

The connector can be downloaded here ( Jolie Custom services Installation ), the current release has to be consider a Beta release, therefore prone to present  imperfections and bugs
Some of them are already known and are on the way to be corrected:
  1. Logoff operation missing
  2. Handling  MongoDB security
  3. DateTime storing error[UTC DateTime
the user must consider the following native data type mapping

Mongo TypeJolie TypeDetail
Doubledouble
int32int
int64int
Stringsting
DateTimelongwith child node ("@type")="Date"
ObjectIdlong






   
   

May 15, 2016

Posted by Fabrizio Montesi

Provide-until now provided in Jolie

A pattern that arises often when writing a (micro)service is that of a workflow that provides access to a resource until something special happens. The new provide-until primitive in Jolie supports exactly this pattern. It doesn’t add any expressivity to the language (we could do this before using bookkeeping variables, just like in any other language), but it makes the code more readable and pleasant in this kind of situations that come up so frequently.

Here is an example of a service that handles shopping carts. We can start a process in our service by invoking operation createCart. We then set a timeout (given by the constant TIMEOUT, omitted here). When the timeout expires, the Time service (from the Jolie standard library) is going to call us on operation timeout. Now we are all set and we enter a provide-until block. Provide-until takes two input choices. The reasoning is very simple: the operations in the first input choice are provided (thus can be invoked multiple times) until any of the operations in the until block is called. Below, we can invoke the operations add and remove to add and remove, respectively, items in our shopping cart, until one of the following happens: the process timeouts (operation timeout is invoked); the user decides to delete the cart (operation delete); or, the user decide to proceed to checkout (operation checkout).

/* Interfaces and ports ... */

cset { id: Add.id Remove.id Timeout.id Delete.id Checkout.id }

main
{
  createCart( request )( cart ) {
    cart.id = csets.id = new;
    cart.user = request.user
  };

  setNextTimeout@Time( TIMEOUT { .message.id = cart.id } );

  provide
    [ add( request )( cart ) {
      cart.items[ #cart.items ] << request.item
    } ]

    [ remove( request )( cart ) {
      undef( cart.items[ request.which ] )
    } ]

  until
    [ timeout() ]

    [ delete()() ]

    [ checkout( paymentInfo )() {
      // ...
    } ]
}

Provide-until is a little thing that I enjoyed developing lately. It originally comes from the desire of handling elegantly REST processes in Jolie, but it works just as well in any similar situation. I give a longer explanation for researchers of how Jolie works with the Web in this paper: https://arxiv.org/abs/1410.3712.

April 26, 2016

Posted by Saverio Giallorenzo

Full Support for Executing Jolie from Sublime Text

Version 1.2.2 of the ‘jolie’ plugin for the Sublime Text editor now supports the direct execution of Jolie programs from the editor’s interface on the main operating systems, i.e., Windows, OSX, and Linux.

Executing a Jolie program is as easy as pressing Ctrl+B (or Cmd+B on OSX) keys, which is the shortcut for launching the build command in the editor (Tools > Build).

Currently, the Jolie programs execute in different terminals according to the OS and the installed terminal emulators:

  • Windows: the default prompt console;
  • OSX: the default Terminal app or iTerm in case it is installed;
  • Linux: uses (and requires) x-terminal-emulator, which automatically starts the OS-defined default terminal (e.g., gnome-terminal for Gnome, konsole for KDE, etc.).

Thanks to Enrico Benedetti, Matteo Sanfelici, and Stefano Pardini for their help in building and refining the launcher for Linux.

June 09, 2015

Posted by Fabrizio Montesi

Non-distributed Microservices



Thanks to Martin's great help, Jolie 1.4 will have support for Internal Services and enhanced local locations. If you feel like testing, you can already find it in our git repository (see compile from sources).
While these features do not add any essential expressiveness to Jolie, they make it easier to program non-distributed microservice architectures (NMSA here for brevity). "Wait" - I hear you - "NMSAs?"


Non-distributed Microservice Architectures (NMSAs)

When I teach somebody (students or professionals) how to program microservices, I pay attention to following a very gradual path. The fact is, there are simply too many things to worry about when you program Microservice Architectures (MSAs). If you are not an expert, you are going to get overwhelmed by the complexity. Even if you are an expert, you may be better off with the MonolithFirst approach.

But why are Microservices so complicated? The main reason is that an MSA is typically distributed, and distributed computing is hard (see the fallacies of distributed computing). So, intuitively, it is much simpler to start with a monolith, and switch to microservices later. Unfortunately intuition is not really met by reality here. Developing a monolith does not force programmers to make their components modular and loosely-coupled enough to be smoothly ported to microservices; for example, it is way too tempting to share resources and/or develop APIs that are not suitable for message passing. Most probably, your initial monolith will end up being a SacrificialArchitecture. This is not necessarily a bad thing: many successes were built on top of sacrificing architectures. I am more worried about the fact that this is not really a smooth learning process: develop a monolith, learn something, now change everything.

NMSAs is a style where you program your system as if it were an MSA, but without the distribution. In an NMSA, Microservices are run concurrently in the same container (in our case, a Jolie-controlled JVM). It is an attempt at providing an easier setting for learners and experimenters to start with: no distributed computing means no fallacies of distributed computing. What can you learn in such an environment? A great deal, including:

  • You can learn how to develop loosely-coupled service interfaces, and how to deal with message passing in the easier setting of a perfectly-working "network".
  • You can learn how to test your services, and how to automatise testing.
  • You can learn how to monitor service execution.
The idea is to first approach the programming of microservices in this simpler ("fake", if you like) setting, and learn how to deal with distribution later. In the Jolie programming language, this path from NMSAs to MSAs is supported by the language itself.

NMSAs first...

NMSAs are created in Jolie by using embedded or internal services, i.e., services that run locally in the same container of the main service program. I will refer to such services as "local services" here. All components that you can write in Jolie are services, therefore all the architectures that you write in Jolie are service-oriented by design. This means that components cannot share state, and must interact through service interfaces based on message passing. What happens behind the scenes depends on the deployment information that you specify for your services, which is kept separate from the application logic. If the case of a local service, messages are efficiently communicated via shared memory, but without breaking the message passing abstraction given to the programmer.

Here is a toy example of a program using local services (you can of course split this in multiple files; oh, and I'm omitting what happens in case we do not find some records or files):


service Images {
    Interfaces: ImagesIface
    main {
        get( request )( img ) {
            fetch@ImageDB( request )( img )
        }
    }
}

service Products {
    Interfaces: ProductsIface
    main {
        get( request )( response ) {
            query@ProductDB
                ( "select name,desc,imageurl from products where id=:id"
                    { .id = request.id } )
                ( response )
        }
    }
}

main {
    viewProduct( request )( product ) {
        get@Products( { .id = request.id } )( product );
        if ( request.embedImage ) {
            get@Images( { .url = product.imageurl } )( product.image )
        }
    }
}

The program above is a simple service to access product information. It offers one operation, viewProduct, that clients can invoke with a message containing the id of the product they want to see (as a subnode of request). The service then invoke the internal service Products, which queries a database for some basic information and returns it. The product information contains an URL to an image depicting the product. Then, the main program checks whether the original client requested the image to be embedded in the response (useful, e.g., for mobile applications that want to minimize the number of message exchanges); if positive, the image is added to the response by fetching it from the Images service.

The main service, Images, and Products communicate using shared memory in the Jolie implementation of internal services. Communications are therefore always going to succeed, but each service has its own interface and state and the style is still message passing. Hence, internal services can be used to teach how to design a loosely-coupled architecture.

...and MSAs later

NMSAs are expressive enough to teach, e.g., how to deal with concurrency issues (each service has its own processes), message passing interfaces, and good API design. They are also fairly easy to achieve, relieving beginners from the frustrations of service deployment.

After a while, however, an NMSA that needs to scale has to evolve into an MSA, by taking some internal services and making them distributed. Distributing an internal service in Jolie is easy. Simply put, take the code of the internal service you want to distribute and execute it with a separate interpreter in another machine (Jolie Enterprise automatises this job for you in a cloud environment, but you can use the tools of your choice).

The main service now needs to be updated to refer to Images and Products as external distributed services:

outputPort Images {
Location: "socket://images.example:8080"
Protocol: sodep
Interfaces: ImagesIface
}

outputPort Products {
Location: "socket://products.example:8080"
Protocol: sodep
Interfaces: ProductsIface
}

main {
    viewProduct( request )( product ) {
        get@Products( { .id = request.id } )( product );
        if ( request.embedImage ) {
            get@Images( { .url = product.imageurl } )( product.image )
        }
    }
}

Basically, we have replaced the code for each internal service with a respective output port. The rest of the code does not change (again, a key aspect of developing services with Jolie is that deployment is kept as separate as possible from behaviour). However, now the system is distributed. Among other consequences of this, communications with services Products and Images may now fail! In practice, this means that the invocations get@Products and get@Images can throw network-related faults now, and we must account for that. If we leave the code like this, such faults are going to be automatically propagated to clients. This may be undesirable. A better strategy could be:

  • If we fail to get the product information, we immediately return an informative fault to the client.
  • If we manage to get the product information but fail to get its image, we should return at least the product information with a placeholder blank image.
We update the code accordingly, using dynamic fault handlers:



main {
    viewProduct( request )( product ) {
        scope( s ) {
            install( default => throw( ProductsCurrentlyUnavailable ) );
            get@Products( { .id = request.id } )( product );

            install( default => product.image = BLANK_IMAGE );
            if ( request.embedImage ) {
                get@Images( { .url = product.imageurl } )( product.image )
            }
        }
    }
}

There we go. Now our code accounts for network errors (actually any error, since I used the default handler, which catches any fault). There are, of course, many other things to watch out for, e.g., distributed monitoring, crash recovery, and performance. However, this small example gives already an idea of the overall methodology.

Gotcha

Although Jolie does its best in not allowing resource sharing among services (they simply cannot be shared, by the language syntax), this can always be circumvented via external side-effects. For example, two internal service may share access to the same file system. However, this can also be abstracted from: in Jolie all accesses to the file system happen through the File service from the standard library which, being a service, can also be distributed.

Conclusions

The microservices style arises out of practical needs, and although it should be used only when necessary, sometimes it is appropriate. I believe that we should find ways to guide a smoother transition from monoliths to microservices, and that finding such ways is possible. This is a rough attempt at giving a first piece of the puzzle towards building such a transition model.


PS: I should really try to come up with better acronyms.
PPS: If you have interesting examples to share in other languages, please feel free to do so!

May 04, 2015

Posted by Fabrizio Montesi

Evaluating the Design of a Microservice


I made a little rubric for evaluating the design of Microservices. Keep in mind that it is still very preliminary. If you have comments, I would be more than happy to discuss them with you.

Find it here:

https://github.com/microservices-course/material/blob/master/rubrics/design.md

April 28, 2015

Posted by Balint Maschio

Using defined macro vs personalized Jolie service

When designing software solution using microservices  one finds himself to wander how do handle repetitive code.  This is particularly true when one uses microservices for data handling and data mapping; where often it is required to transform data from a format to another for example:
  1. Date Format
  2. Double Value Format 
  3. Other Alphanumerical manipulation  
Jolie language provides  to the programmer already with some rather powerful primitives to manipulate such data but is not uncommon to have to personalize some behavior. And this is where the dilemma araises. Do I go for 
  1. A series of  macros defined by the primitive define 
  2. Or do I create a service with a number of exposed operation
Now this post does not aim to set a standard or either to propose a methodology for microservices coding in Jolie, Its only aim is to share a personal checklist when taking a coding decision on repetitive code .
Let's look  to an example of date transformation
   
define transformDate{

    if (is_defined(__date)){

requestSplit= __date;
requestSplit.length = 4 ;
splitByLength@StringUtils (requestSplit)(resultSplitYear);
year = resultSplitYear.result[0];
requestSplit= resultSplitYear.result[1];
requestSplit.length =2;
splitByLength@StringUtils (requestSplit)(resultSplitMonth);
month = resultSplitMonth.result[0];
day = resultSplitMonth.result[1];
__transformedDate =day+"-"+month+"-"+year
}
}


Now this section of code could be defined with in an iol file and included by several services and used in this way

__date = value_original_date
transformDate
value_new_date = __transformedDate

Now let us see the same functionality implemented with a service

type DateTransformRequest:string

type DateTransformResponse:string

interface TransformInterface {
RequestResponse:
 transformDate (DateTransformRequest)(DateTransformResponse)
}

and with an implementation that looks like

inputPort TransformerInputPort{ 
  Location:"socket://localhost:8099" 
  Interfaces:TransformInterface
  Protocol:sodep 
}
main{
transformDate(request)(response){
  //the code that does it here
}]{nullProcess}


Now let us go down to the topic of the post how can we compare the two implementations

Define implementation

Pro
Con
Need to restart all the services if there a change to be implemented

Yes
No network resources
Yes

Shorter time of implementation
Yes

Clear error tractability

No
Network deploy

No

and for the Jolie service implementation

Jolie Service implementation

Pro
Con
Need to restart all the services if there a change to be implemented
No

No network resources

No
Shorter time of implementation

No
Clear error traceability
Yes

Network deploy
Yes


Now the reader may be tempted to object to such a limited arbitrary check list and he or she would be right to do so, but my aim was not to give a definitive answer but more indicate a methodology of thinking.
Now let us step back to the drawing board and consider the following questions.

  1. Do I need to re-use the code several time and across several implementations
  2. My hardware infrastructure will be subject to  changes
  3. Do I need trace possible errors specializing error raising and handling
  4. Do I need to use complex data type  

If the answer to all these questions is yes is my opinion that would be better to opt for a Jolie Service Implementation approach for the following reason:
  1. In both implementation the cross usability is possible but the Jolie service implementation is favorable for :
    1. In case of code changes I need to change in a single place 
    2. If I need to use in a new project I do not have to copy specific implementation but refer to an interface and leave the implementation stable
  2. This point is often underestimated during the design phase of the project but in microservices architecture has a huge impact ; also in this case the Jolie Service Implementation is favorable because:
    1. Its naturally more adaptable to infrastructure changes ( the service can be deploy where ever )
    2. If there is a need of splitting the microservices implementation over several machine, in the case of the define base implementation we will find ourself duplication the implementation on several machine , with all the problems concerning code duplication
  3. The error handling  it is essential in any software paradigm , but it is even more important in SOA architecture where the high level of concurrent execution would almost impossible to trace errors. In the case of the Jolie Service Implementation the possible error are definable at level of interface making in my opinion simpler the use of the functionalists
  4. /* define implementation */
    define myDefine{
       /*some code*/
       throw (someError)
       /* some other code */
       throw (someOtherError) 
    }
    /* Use of interface  */
    interface myInterface{
    RequestResponse:
    myOperation (requestType)(responseType) throws someError , someOtherError
    }  
    1. As the code above tries to exemplify the user of the functionalists  would   find easier to identify the possible error to catch via interface instead of having to scan the implementation
  5. Similarly to the issue of the fault handling the use of complex types is aided massively by the use of  a Jolie Service Implementation approach where we can define complex types in a clear way instead of having to reverse engineer the code of the define
I hope you will find my post somehow useful to define define recurrent code in jolie

    March 17, 2015

    Posted by Claudio Guidi

    Integrating Wordpress with a Jolie service

    Wordpress developers could have the problem to integrate the website they are developing with some extra data which come from external resources. Data synchronization could be not easy, it must be scheduled, data are visible on the web application with a period equal to the synchronization timeout and it usually requires a level of maintenance that is high with respect to the amount of synchronized data.

    In these cases it could be useful to synchronously call an external server in order to retrieve the data we need. Jolie could be very helpful for developing a fast API server to be integrated within your web site.

    This short post describes a way for easily integrate Wordpress with a Jolie service.

    First of all, let me consider the simple case of getting a list of addresses from a Jolie service. This service could be written as it follows:

    type GetAddressRequest: void {
        .name: string
        .surname: string
    }

    type GetAddressResponse: void {
        .address: string
    }

    interface ServerInterface {
        RequestResponse:
            getAddress( GetAddressRequest )( GetAddressResponse )
    }


    execution{ concurrent }

    inputPort AddressService {
        Location: "socket://localhost:9001"
        Protocol: http
        Interfaces: ServerInterface
    }

    init {
        with( person[ 0 ] ) {
            .name = "Walter";
            .surname = "White";
            .address = "Piermont Dr NE, Albuquerque (USA)"
        };
        with( person[ 1 ] ) {
            .name = "Homer";
            .surname = "Simpsons";
            .address = "Street 69, Springfield USA"
        };
        with( person[ 2 ] ) {
            .name = "Sherlock";
            .surname = "Holmes";
            .address = "221B Baker Street"
        }


    }

    main {
        getAddress( request )( response ) {      
            index = 0;
            while( index < #person
                    && person[ index ].name != request.name
                    && person[ index ].surname != request.domain ) {
                index ++
            };
            if ( index == #person ) {
                throw( PersonNotFound )
            } else {
                response.address = person[ index ].address
            }
        }

    }


    In this case I simulated a database with a simple vector of persons. The service is located in socket://localhost:9001 and you can easily invoke him through a browser by using this url: http://localhost:9001/getAddress?name=Sherlock&surname=Holmes

    Clearly, this is an example which runs on your local machine, it is simple to export it in a distributed scenario by simply running Jolie on the remote machine, let me suppose it is located at IP X.X.X.X.

    When the Jolie service is executed, we can invoke it within a wordpress page. In order to do this, simply add the following PHP code on the page where you want to show the results:

    $data = array(
        'method' => 'POST',
        'timeout' => 45,
        'httpversion' => '1.0',
        'blocking' => true,
        'headers' => array(),
        'body' => array( 'name' => 'Sherlock', 'surname' => 'Holmes' )
        );

    $url = 'http://X.X.X.X:9001/getAddress';
    $result = wp_remote_post( $url, $data );

    print_r( $result );

    if( is_wp_error( $result ) ) {
    echo $result->get_error_message();
    }


    Where $result contains the response message.

    You can also exploit a JSON format message by adding the following parameter to the http protocol of the Jolie service and re-arranging the PHP code for managing JSON messages.

     inputPort AddressService {
        Location: "socket://localhost:9001"
        Protocol: http { .format="json" }
        Interfaces: ServerInterface
    }


    Moreover you could connect the Jolie service to a database for retrieving the data you need, check the Jolie tutorial page for getting how to use databases in Jolie.



    February 06, 2015

    Posted by Fabrizio Montesi

    Programming Microservices with Jolie - Part 1: Data formats, Proxies, and Workflows.

    In Microservice architectures, software components are built as reusable and scalable services that interact with each other. While this introduces many advantages, it also brings all the issues of distributed computing to the internals of software applications and developers need to be careful about this. Luckily, we are standing on the shoulders of all the knowledge that we accumulated by using Service-oriented Architectures (SOAs), so we are more aware of these issues and we have more technologies and methodologies to deal with them. Microservices could (and I believe they will) fly after all, in one form or the other, and we at the Jolie team are surely not the only ones sharing this optimism.

    But like SOAs were, Microservices are also at risk of exposing programmers to too much complexity to deal with when they take their first steps in this world. And we are at risk again of making arbitrary technology choices that we may regret later, just like SOAP was in many cases.

    In developing Jolie, a microservice-oriented programming language, we are trying to make microservice programming simple and productive. A major challenge in this is how we can map design ideas for a microservice architecture to code as fast as possible, without committing to specific implementation details on communications and service deployment/distribution.

    The problem of focusing on design issues and not getting locked inside of interoperability problems is particularly dear to me. Jolie helps in this, e.g., by abstracting application logic from the underlying data formats that you will use for communications.

    Another problem that is dear to the team in general and I will talk about below is that of making composition of microservices easy. This does not simply mean composition of the operations offered by microservices, but also dealing with how we can deploy new systems based on previously developed systems.

    But talk is cheap. Let's do code and see an example.


    Get Jolie

    You don't need to have Jolie installed to read this article, but if you want to try what I will show, go install Jolie.
    Ready? Let's proceed.


    A calculator microservice

    We are going to program a little toy calculator microservice for making additions. Feeling lazy? Download the code here.
    Open a file and call it calculator.ol.
    Then we write:

    include "calculator.iol"

    execution { concurrent }

    inputPort CalcInput {
    Location: "socket://localhost:8000"
    Protocol: sodep
    Interfaces: CalcIface
    }

    main
    {
    sum( req )( resp ) {
    resp = req.x + req.y
    }

    }

    Above, we are telling Jolie that this microservice is concurrent, i.e., each request is handled in parallel by a lightweight process (Jolie processes are implemented as threads with a local state).
    The input port CalcInput states that the service can be accessed using TCP/IP sockets on port 8000, using the SODEP protocol. CalcIface is the exposed interface, which we define in  the included file calculator.iol:

    type SumT:void {
    .x:int
    .y:int
    }

    interface CalcIface {
    RequestResponse: sum( SumT )( int )

    }

    CalcIface is an interface with only one Request-Response operation called sum, which receives a tree with two nodes (x and y, both integers) and returns an integer.
    The sum operation is implemented in the main procedure of the calculator microservice: it simply receives the message on variable req and replies with the sum of the two subnodes x and y to the invoker.


    Using the calculator

    Using the calculator service from Jolie is easy. We create a client.ol program, include the interface and off we go:

    include "calculator.iol"
    include "console.iol"

    outputPort Calculator {
    Location: "socket://localhost:8000"
    Protocol: sodep
    Interfaces: CalcIface
    }

    main
    {
    with( req ) {
    .x = 3;
    .y = 2
    };
    sum@Calculator( req )( resp );
    println@Console( resp )()
    }

    Running this client program will reach the calculator service and print the number 5 on screen. SODEP is a binary protocol, so you don't need to worry about overhead as much as you have to in, e.g., SOAP.

    The attentive reader may have noticed that in our client code we are calling the println operation of the Console service, just like we are calling the sum operation of calculator. That's right, even our Console library is a microservice that you can relocate as you like.

    OK, but what if the rest of my system is not in Jolie?

    Jolie is interoperable with many other technologies, both server- and client-side. Let's see what happens if our client is not written in Jolie and does not support SODEP. Let's say instead that it's a web browser and that it consequently uses HTTP.
    Take the calculator program again and add the following before the main procedure:

    inputPort CalcWebInput {
    Location: "socket://localhost:8080"
    Protocol: http
    Interfaces: CalcIface
    }

    Et voilà, now calculator is not only exposed on TCP port 8000 with SODEP but also on TCP port 8080 with the HTTP protocol. Run calculator and you can now point your browser to:


    You will see calculator crunching the numbers and reply with:

    <sumResponse>8</sumResponse>

    You can use many other protocols (HTTP/JSON, SOAP, local memory, ...) and communication mediums (Bluetooth, UNIX sockets, ...), depending on your needs. See here.


    What if I cannot change the code of my microservices?

    If your calculator is a black box that you cannot or do not want to touch, then you cannot just edit its code to add an HTTP input port. This is the typical case in which you would like to compositionally evolve your system without touching what has already been deployed.

    The standard solution is to use a proxy. In our example, we wish to make a proxy that enables web browsers to communicate with the calculator service:

    Web browser <-> Proxy <-> Calculator

    Proxies (and variants) are very useful in general as intermediate glue components in service systems, but managing them may require you getting to know yet another technology. In Jolie, instead, proxies are powered by a native language primitive for forwarding messages and integrate well with our communication ports, so you can take advantage of the data abstraction layer provided by Jolie.

    Open a file proxy.ol and write:

    include "calculator.iol"
    include "console.iol"

    execution { concurrent }

    outputPort Calculator {
    Location: "socket://localhost:8000"
    Protocol: sodep
    Interfaces: CalcIface
    }

    inputPort ProxyInput {
    Location: "socket://localhost:9000"
    Protocol: http
    Aggregates: Calculator
    }

    main
    {
    in()
    }

    The important part here is that we have an input port on TCP port 9000 that aggregates the output port towards the Calculator. This means that all messages on port ProxyInput for an operation provided by Calculator will be forwarded to that service. So now we have our proxy and we can navigate to:

    http://localhost:9000/sum?x=3&y=5

    The proxy service will now receive your HTTP message on port ProxyInput, see that it is for the sum operation of calculator. The message is therefore converted to the SODEP format, as specified by the output port towards the calculator. The response from calculator will finally be converted again from SODEP to HTTP and sent to the initial client.

    Aggregation is really useful. You can also add hooks (called couriers) for intercepting messages and, e.g, log operations or check for authentication credentials.

    Workflows

    Alright, now we have our toy microservice system. Let's say that after a while, you want to add the requirement that a user must log in before calling the sum operation on the calculator. This is a workflow, the magic keyword being before in the previous sentence. Jolie inherits native workflow primitives from business process languages (e.g., BPEL), making handling flows of communications among services a breeze!

    Open calculator.ol and change the main definition as follows:

    cset {
    sid : SumT.sid
    }

    main
    {
    login( creds )( csets.sid ) {
    if ( creds.pwd != "jolie" ) {
    throw( InvalidPwd, "The password is jolie" )
    };
    csets.sid = new
    };
    sum( req )( resp ) {
    resp = req.x + req.y
    }
    }

    The cset block above declares a session identifier sid, which we will use to track invocations coming from a same client (that's a correlation set). We also introduced a new operation, login, which simply checks if the password sent by the client is jolie and otherwise throws a fault to the invoker. Observe now that we use the semicolon operator to tell Jolie that sum becomes available only after login has successfully completed.

    This is a fundamental difference with respect to, say, object-oriented computing, where if you want to enforce order in how different methods are called, you have to implement it yourself with bookkeeping variables and concurrent lock mechanisms. Here instead, you just write a semicolon.

    We also need to update the interface of the calculator, in calculator.iol, to account for our modifications:

    type SumT:void {
    .x:int
    .y:int
    .sid:string
    }

    type LoginT:void {
    .pwd:string
    }

    interface CalcIface {
    RequestResponse:
    login( LoginT )( string ) throws InvalidPwd( string ),
    sum( SumT )( int )
    }

    We are done! No need to update the code of the proxy, as the aggregation primitive is parametric on the interface and the protocols used by the aggregated services. Jolie will do that lifting for us. So now we can browse to: 


    You will get a random session identifier. I got:

    <loginResponse>193e6a7a-895c-4ff6-b32a-5bbab0eea7f3</loginResponse>

    So now I can go to (remember to replace sid with what you got in the previous call if you try this yourself):


    And receive my good old 8 as result.

    Have a look at the Jolie documentation if you are interested in using cookies and theming your responses with nice HTML. You can put that in a separate microservice, to keep calculator clean!

    Conclusions

    This is just the tip of the iceberg of what you may have to do with microservices and what Jolie can do for you.

    But we can already see some of the underlying concepts that Jolie is based on:

    • You should not commit to any particular communication technology. Microservices are born to be lean and reusable, and should be kept decoupled from the implementation details of how data is exchanged.
    • Building proxies and other kinds of architectural composers is very handy in microservice architectures just like it is in any other distributed system. However, with microservices you can end up having a lot of them. Jolie helps in creating and managing these composers by making them programmable with native primitives that make it clear what is happening.
    • Workflows can help in making the structure of communications followed by a service explicit. They should be easy to write and Jolie strives in doing so. See here for our other workflow primitives.


    Stay tuned for the next chapters. :-)

    Meanwhile, you may want to have a look at some more material here and here.

    February 03, 2015

    Posted by Saverio Giallorenzo

    Implementing Quicksort-as-a-(Micro)Service

    TL;DR Microservices are not so cumbersome and tedious to develop as you may think. They only require the right tool (Jolie) for the job. I report my experience in implementing a simple staple utility (quicksort) as-a-service. The service recursively calls itself using internal memory but it is very easy to make it accept also HTTP calls or to distribute the workflow on different instances of the same service.


    Scenario: for the release of the new Jolie website we decided to add external blog posts in a dedicated “blogs” section. After ~230 lines of Jolie code we had our own news retrieval service. The service reads the RSS feed from our collection of blogs and serves them as a single flow of news.

    Problem: we had different sources that returned a stream of articles ordered by date. We collected them into a single stream but still they were not ordered properly.

    Solution: one word (well actually a compound, but still) QuickSort-as-a-(micro)service

    QuickSort …

    Best thing I love about QuickSort is its elegant recursive implementation.

    As Wikipedia puts it (slightly modified):

    function quicksort(A)
    	less, equal, greater := three empty arrays
    	if length(A) > 1  
      	pivot := select any element of A
      	for each x in A
          if x <= pivot then add x to less
          if x > pivot then add x to greater
      	quicksort(less)
      	quicksort(greater)
      A := concatenate(less, pivot, greater)
    

    In English, you want to order A, which is an array of sortable (comparable) elements. You take a pivot element (e.g., from the middle of the array) and cycle through the whole array to divide it into two halves, less and greater. less will contain all elements lesser than or equal to the pivot and greater the remaining greater-than-pivot elements. Done that you can apply quicksort to less and greater which will order them. Finally you assemble less, the pivot, and greater (in this order) and return the result.

    … as a (Micro)Service

    Let us take that pseudo-code implementation of quicksort and translate it into a microservice. (Btw, if you are into THAT kind of microservice-porn, here you can find a recursive algorithm of the Tower of Hanoi).

    In the pseudo-code we define a function called quicksort. Being a little more detailed we can add some typing to it. We want quicksort to take an array and return it sorted, hence we have something like function quicksort( Array[ T ] : A ) : Array[ T ]. T being the type of the elements we want to sort.

    In Jolie we can model function calls as request-response operations. Hence we can define the interface of our quicksort service as

    interface QuickSortInterface {
    	RequestResponse: quicksort( QSType )( QSType )
    }

    Jolie interfaces describe which operations a service can implement and the type of their input and output. The type of QSType can be something like

    type QSType: void {
    	.e* : T
    }

    Remember that every variable in Jolie is a tree 1, therefore our type will have a root, say QSType, which is empty (void) and then we allow it to have 0 to n leaves e[0], …, e[n-1] of type T.

    As said, let us begin with the request-response operation

    quicksort( req )( res ){ ... }
    

    As you might guess, req and res have both type QSType. The body of the operation will contain the behaviour of the quicksort.

    function quicksort( A )
     if length(A) > 1
      pivot := select any element of A
      for each x in A
       if x <= pivot then add x to less
       if x > pivot then add x to greater
      quicksort(less)
      quicksort(greater)
     A := concatenate(less, pivot, greater)
    
    quicksort( req )( res ){
     if( #req.e > 1 ){
      // 1. partition the request
      pivot = #req.e / 2;
      for ( i = 0, i < #req.e, i++ ) {
       if( i != pivot ) {
        if( req.e[ i ] <= req.e[ pivot ] ) {
         less.e[ #less.e ] << req.e[ i ]
        } else {
         greater.e[ #greater.e ] << req.e[ i ]
        }
       }
      };
      // 2. order the two partitions
      // 3. merge the result
     } else {
      res << req
     }
    }
    

    First we check if the array in the request has less than 2 elements. Jolie provides the # operator that returns the number of leaves in the branch at its right. Its behaviour is not different from the usual req.e.length of C++/Java. In case there are less than 2 elements (else branch), we return as response exactly the request. res << req (<< being the deep-copy operator) means “copy the whole structure and values of variable req in res”.

    If there are more elements we can sort them. Although there are plenty of optimisations [1][2][3][4], we will stick to the good ol’ “take the middle element as pivot”. We use # to address the n-th+1 leaf and store the whole structure of the i-th element of req.e.


    Beware: pedantic observation!. We are using << to copy the values of the leaves. Why didn’t we use the expression less.e[ #less.e ] = req.e[ i ]? If we assume the type T of req.e[ i ] to be a basic type (int, string, double, …), using = would yield the same result as <<. On the contrary, if we want to bring along a sub-structure rooted in req.e[ i ] (e.g., blog posts) we need to deep-copy it.


    Let us get to the recursive calls. We need to create two ports, one inputPort and one outputPort.

    outputPort SelfOut {
    	Interfaces: QuicksortInterface
    }
    
    inputPort SelfIn {
    	Location: "local"
    	Interfaces: QuicksortInterface
    }

    We use the "local" location, which is a lightweight medium that exploits in-memory messages between services running in the same interpreter. In this case we use it to let the QuickSort Service call itself.

    Almost finished. Let us invoke quicksort on the two partitions created earlier.

    // 2. order the two partitions
    quicksort@SelfOut( less )( res );
    quicksort@SelfOut( greater )( greater )
    

    Note that we store the sorting of less in res. This comes handy when we will merge the result together

    // merge the result
    res.e[ #res.e ] << req.e[ pivot ];
    for ( i=0, i<#greater.e, i++) {
    	res.e[ #res.e ] << greater.e[ i ]
    }

    We append the pivot and the elements of greater to res, which concludes the behaviour of quicksort.

    Just a couple of things and we are ready to run our service. First, we set to concurrent the execution mode of our service. In concurrent mode Jolie creates a new instance of the service at any new request.

    execution{ concurrent }

    and second we add the init procedure which, like the main, is a special procedure. If present init is executed before the main. To enable quicksort call itself we need to set the local location of outputPort SelfOut to the value returned by the runtime service

    init { getLocalLocation@Runtime()( SelfOut.location ) }

    Runtime is a service of the Jolie Standard Library, imported with the line include "runtime.iol"

    To wrap up

    Let us pull together the parts we wrote above into a single Jolie service (for simplicity I set the type of the content of the array to int). Note that we enclosed operation quicksort into the main procedure.

    include "runtime.iol"
    
    execution{ concurrent }
    
    type QSType: void{
    	.e*: int
    }
    
    interface QuicksortInterface {
      RequestResponse: quicksort( QSType )( QSType )
    }
    
    inputPort SelfIn {
      Location: "local"
      Interfaces: QuicksortInterface
    }
    
    inputPort SelfHttp {
      Location: "socket://localhost:8000"
      Protocol: http
      Interfaces: QuicksortInterface
    }
    
    outputPort SelfOut{
      Location: "local"
      Interfaces: QuicksortInterface
    }
    
    init { getLocalLocation@Runtime()( SelfOut.location ) }
    
    main
    {
      quicksort( req )( res ){
      if( #req.e > 1 ){
       // 1. partition the request
       pivot = #req.e / 2;
       for ( i = 0, i < #req.e, i++ ) {
        if( i != pivot ) {
         if( req.e[ i ] <= req.e[ pivot ] ) {
          less.e[ #less.e ] << req.e[ i ]
         } else {
          greater.e[ #greater.e ] << req.e[ i ]
         }
        }
       };
      // 2. order the two partitions
      quicksort@SelfOut( less )( res );
      quicksort@SelfOut( greater )( greater );
      // merge the result
      res.e[ #res.e ] << req.e[ pivot ];
      for ( i=0, i<#greater.e, i++) {
        res.e[ #res.e ] << greater.e[ i ]
        }
      } else {
       res << req
       }
      }
    }

    Did you notice I also added a new port to the service?

    inputPort SelfHttp {
      Location: "socket://localhost:8000"
      Protocol: http
      Interfaces: QuicksortInterface
    }

    What it does is to enable the service to accept a request from any browser. How cool is that?! We developed an application for in-memory requests and with just 4 lines of declarative code we have it accepting calls from HTTP! Fire up your browser and write this URL:

    http://localhost:8000/quicksort?e=5&e=21&e=13&e=34&e=1&e=1&e=2&e=3

    (or if you are REALLY lazy click here, after you launched the quicksort service, of course.)

    Still here? Get yourself a Jolie and start tinkering with microservices :)

    I want more

    You want more? Really? Ok, one last candy. As you may be aware of (or maybe not) the ; in Jolie does not mark the end of a statement. It is the sequential composition operator by which A ; B means after the execution of A executes B. Jolie also provides the parallel composition operator which executes two statements concurrently.

    Want to optimise for multi-core computation?

    { quicksort@SelfOut( less )( res ) | quicksort@SelfOut( greater )( greater ) }

    you can send two parallel requests to quicksort. Note that we enclosed the two requests into a scope. In this way the instructions after the scope will be performed only when both requests return their results.

    Want more? Take a cloud, deploy several copies of quicksort and call them

    { quicksort@Instance1( less )( res ) | quicksort@Instance2( greater )( greater ) }

    And this is only the tip of the iceberg as you can aggregate a cloud of services into a single point of workflow distribution, as well as proxies, load-balancers, and many other architectural patterns.

    Always without leaving the Jolie language.

    Pardon me if I ask it again. What are you waiting for getting yourself a Jolie?

    Happy microservices :)

    1. I know, at the beginning it might feel awkward but native tree-structured variables come really handy when dealing with JSON/XML-like data. Good topic for another post though. 

    Posted by Claudio Guidi

    Programming Service Oriented Recursion

    Recursion in service oriented architectures is something unusual because it seems to be useless. So, why a post about recursion in SOA? I think the big result we can obtain from service oriented recursion is understanding service oriented programming paradigm nature and technologies. Recursion indeed, is a well known programming pattern hugely adopted by all the programmers in the world and its usage could reveal us some interesting features about a programming language. So, I want to use recursion in a SOA because I want to know more about SOA. Let's do it.

    In the following example you can find the implementation of Fibonacci with Jolie.
    [ You can copy it into a file and save it with name fibonacci.ol. Then run it typing the command
    jolie fibonacci.ol ]


    include "console.iol"
    include "runtime.iol"

    execution{ concurrent }

    interface FibonacciInterface {
      RequestResponse:
        fibonacci( int )( int )
    }

    outputPort MySelf {
      Location: "socket://localhost:8000"
      Protocol: sodep
      Interfaces: FibonacciInterface
    }


    inputPort FibonacciService {
      Location: "socket://localhost:8000"
      Protocol: sodep
      Interfaces: FibonacciInterface
    }

    main
    {
      fibonacci( n )( response ) {
        if ( n < 2 ) {
          response = n
        } else {
          {
    fibonacci@MySelf( n - 1 )( resp1 )
    |
    fibonacci@MySelf( n - 2 )( resp2 )
          };
          response = resp1 + resp2
        }
      }
    }




    The code is very intuitive and simple. The outputPort MySelf defines the external endpoint to be invoked which corresponds to the input endpoint FibonacciService where the operation fibonacci is deployed. The invocations are performed in parallel (operator |) and they are blocking activities which wait until they receive a response from the invoked service
    . You can invoke it by simply exploiting the following client which sends 10 as input parameter:

    include "console.iol"

    interface me {
      RequestResponse:
        fibonacci( int )( int )
    }

    outputPort Service {
      Location: "socket://localhost:8000"
      Protocol: sodep
      Interfaces: me
    }

    main
    {
      fibonacci@Service( 10 )( result );
      println@Console( result )()
    }

    Starting from this simple example we can understand some interesting features:

    server sessions represent recursive call stack layers: each invocation opens a new session on the server which represents a new layer in the recursive call stack.

    loosely coupling: each invocation is separated from the others which guarantees that the value of n is not overwritten by different calls.

    runtime outputPort binding: the binding of the output endpoint (MySelf)  must be achieved at runtime and not at deploy time in order to avoid frustrating programming issues like those reported here:  http://blogs.bpel-people.com/2007/02/writing-recursive-bpel-process.html.

    the invocation stack could be distributed: you can imagine to deploy more than one fibonacci service and switch the invocations from one to another depending on some inner parameter such as, for example, the number of the open sessions. As an example consider the code modified as it follows:

    init {
      getLocalLocation@Runtime()( global.location );
      MySelf.location = global.location;
      println@Console( MySelf.location )();
      global.count = 0
    }

    main
    {
      fibonacci( n )( response ) {
        synchronized( lock ) {
          global.count ++;
          println@Console( "begin n="+n )();
          if ( global.count >= 100 ) {
    MySelf.location  = "socket://localhost:8001"
          }
        };
        if ( n < 2 ) {
          response = n
        } else {
          {
    fibonacci@MySelf( n - 1 )( resp1 ) 

    fibonacci@MySelf( n - 2 )( resp2 )
          };
          response = resp1 + resp2;
          synchronized( lock ) {
    global.count --;
    println@Console( "end n="+n )();
    if ( global.count < 100 ) {
     MySelf.location  = global.location
    }
          }
        }
      }
    }

    Here the location of the outputPort MySelf can be changed dynamically during the execution. global.count stored the number of the current open sessions, if it is greater than 100 the location is changed into socket://localhost:8001 where the second fibonacci service is deployed. In this way you can easily create a chain of Fibonacci services whose sessions participate to a unique Fibonacci number recursive calculation.


    Conclusions
    Here I used service oriented recursion for programming a Fibonacci service with Jolie which can be invoked by external clients. This service could be also chained with other copies of it in order to obtain a distributed SOA for calulating the Fibonacci number recursively. Such an example could be an interesting reference point for understanding how service creation and invocations work in a SOA.


    January 30, 2015

    Posted by Fabrizio Montesi

    Running Jolie inside of Java

    Lately I have been asked a lot how to run a Jolie program from inside of a Java program.
    The opposite, running Java from Jolie, is well documented in our documentation site.

    Running Java code from inside a Jolie program is also a very simple process.
    First, you need to add to your classpath (import in your project, if you are using an IDE) the jolie.jar and libjolie.jar libraries that you can find in your Jolie installation directory (JOLIE_HOME).
    Now, from Java, you can create an Interpreter object. Interpreter is the class that implements the Jolie language interpreter. It will need two parameters for the constructor. The first is the command line arguments that you want to pass to the interpreter. The second is the parent class loader to use for loading the resources that the interpreter will need. Here is an example based on the Linux launcher script (but it should work on any OS), where we assume that the Jolie program that we want to run is called main.ol:


    String jh = System.getenv( "JOLIE_HOME" );
    String args[] = "-l ./lib/*:$JOLIE_HOME/lib:$JOLIE_HOME/javaServices/*:$JOLIE_HOME/extensions/* -i $JOLIE_HOME/include main.ol".replaceAll( "\\$JOLIE_HOME", jh ).split( " " );
    final Interpreter interpreter = new Interpreter( args, this.class.getClassLoader() );

    Now you just need to use the run method of the interpreter object you have just created. Here I do it in a separate thread, as that method will return only when the Jolie program terminates. This is a toy example so I am disregarding exceptions here, but you should obviously care about them in real-world code:

    Thread t = new Thread( () -> {
    try { interpreter.run(); }
    catch( Exception e ) { /* Handle e.. */ }
    } );
    t.setContextClassLoader( interpreter.getClassLoader() );
    t.start();

    If you need to stop the interpreter manually, you can use the exit method it comes with. It takes a hard timeout if you need one. Enjoy your Jolie interpreter inside of Java!

    January 26, 2015

    Posted by Fabrizio Montesi

    Faster websites with Jolie - HTTP Compression enabled by default

    Thanks to Matthias Dieter Wallnöfer, Jolie now supports the negotiation of HTTP compression and this is activated by default. This means that all your Jolie-powered websites will benefit from this, along with your Jolie programs which access remote HTTP content.

    We have just uploaded the patch to the web server on the Jolie website: http://www.jolie-lang.org

    We have also added caching, with the good old .cacheControl HTTP configuration parameter that I introduced a while ago (http://arxiv.org/abs/1410.3712).

    Two minutes later, the loading speed of the main webpage doubled. We went from ~140ms down to ~70ms ! You should feel the difference navigating from one page to the other (aside from those pages using slideshare plugins, which take time to load from slideshare.net).

    Enjoy a faster Jolie website and faster $all_your_Jolie_programs.

    PS: Yes, this will be released with Jolie 1.1.1

    January 25, 2015

    Posted by Balint Maschio

    Using Jolie to implement BTE transactions

    For whom of you that is familiar with SAP and its complex behaviours, the term BTE ( Business Transaction Event  ) should be familiar. For those less familiar with SAP a BTE can be defined as a specific moment ( status ) in  the life cycle of enterprise document. such concept transcends SAP and its implementation of an ERP  For example an Sales Invoice can be represented by a cohesive object that contains all the required information.
    In the service oriented programming object as such do no exist instead they are  substituted by operation and they relative input and output messages. In a standard WSDL notation operations and their relative messages are represented in this way 

    <message name="createSalesInvoiceRequest">
      <part name="clientCode"    type="xs:string"/>
      <part name="invoiceDate"   type="xs:string"/>
      <part name="invoiceAmount" type="xs:double"/>
    </message>

    <message name="createSalesInvoiceResponse">
      <part name="invoiceId" type="xs:string"/>
    </message>

    <portType name="salesInvoiceInterface">
      <operation name="createSalesInvoice">
        <input message="createSalesInvoiceRequest"/>
        <output message="createSalesInvoiceResponse"/>
      </operation>
    </portType>

    In Jolie the operation can be represented using the following definition

    type createSalesInvoiceRequest:void{
       .clientCode:    string
       .invoiceDate:   string
       .orderNumber : string
       .invoiceAmount: double
    }
    type createSalesInvoiceResponse:void{
          .invoiceId:int
    }

    interface SalesInvoiceInterface{
     RequestResponse:
        createSalesInvoice (createSalesInvoiceRequest)(createSalesInvoiceResponse)
    }

    Is not a case that as an example of operation createSalesInvoice was selected as an example: often the communication medium of such document varies from client. Let’s look at some examples

    Your customer is :

    1.     A big corporation and wants a communication via XML File 
    2.     A medium size company that requires PDF
    3.     A small company may just want an e-mail
    Now let's us define a DocumentInterface

    type createDocumentRequest:void{
       .invoiceId:    int
       .invoiceDate:   string
       .orderNumber : string
       .invoiceAmount: double
       .clientInfo:clientInfoType
       .
    }
    type createDocumentResponse:void{
          .docId:string
    }

    interface DocumentInterface{
     RequestResponse:
        createDocument (createDocumentRequest)(createDocumentResponse)
    }

    Now let's us assume that three separate services are created and each of the them is implementing the documentInterface

    Now in the service that implements createSalesInvoice we need to define an output port in this way

    outputPort DocumentServicePort {
      Location:  "local"
      Protocol:sodep
      Interfaces: DocumentInterface
    }

    and an implementation of the operation as follow

    [ createSalesInvoice(request)(response){
      /*code to save on the DB */
      response.invoiceId = responseDB.invoice
    }]{ 
         requestClientInfo.clientCode = request.clientCode;
         getClientInfo@ClientServicePort(requestClientInfo)(responseClientInfo);
         requestDocumentCreator.invoiceId =  response.invoiceId ;
         requestDocumentCreator.invoiceDate = request.invoiceDate;
         requestDocumentCreator.orderNumber = request.orderNumber;
         requestDocumentCreator.invoiceAmount =request.invoiceAmount;
         requestDocumentCreator.clientInfo << responseClientInfo.clientInfo;
         DocumentServicePort.location = responseClientInfo.serviceInfo.location
        createDocument@DocumentServicePort( requestDocumentCreator) (responseDocumentCreator);
    }

    It can be seen on the line underlined in green how the location of the DocumentService is dynamically this will allow to select the location dynamically depending on the client code the location and therefore its implementation of createDocument.
    Let’s go back to our three clients with there relative localtion

    1.     A big enterprise =>"socket://localhost:3001"
    2.     A medium enterprise =>  "socket://localhost:3002"
    3.     A small enteprise=>  "socket://localhost:3003"

    Let’s assume now that in the life cycle of the implementation will be necessary to specialize even further the document service
    The IT development team will be able to design and develop the new service without having to touch the existing code. Once tested and approved by the business it will be possible to activating by simply adding the new location to all the client master data that need to have this new service

    I hope this post has showed some the excellent flexibility capabilities Jolie possess in implement BTE. How a change in a part of the business workflow does not mean long hours of redesign and redeployment. The author does realize that this example centrally does not represent the complexity of a real business work flow, but this post aims to propose Jolie as a serious contend for Business workflow design and implementation

    December 19, 2014

    Posted by Fabrizio Montesi

    The first full course on Jolie has ended

    The first university course almost entirely on Jolie (and choreographies!) ended today, with cakes and final discussion on the exam procedure. :-)
    I had a lot of fun teaching it. I look forward to the next edition, I got lots of good ideas from this one.

    It was also a lot of work, as I had to prepare a lot of material from scratch. There is still lots of room for improvement, but I got a decent set of lectures and exercises out of it. Here is the material from this year:
    http://fabriziomontesi.com/teaching/imds-f2014/index.html

    Feel free to use it, as long as you put a reference to the origin please. Or, just contact me.

    This year I also experimented with teaching students how to prepare and give a presentation on a scientific paper, and how to read scientific articles. I think it really paid off, they gave excellent presentations. But it's hard to tell, I had really clever students.. so maybe it's all them. ;-)
    Here are some of their carefully crafted presentations:
    I still have to get the final feedback on the course, but an initial probing tells me that most of them would use Jolie for their future software projects. This alone makes all the effort worth it. I can't wait to see what they build with it. Some of them have already started.

    As the students were so happy, I am going to try carrying out an educational project on research-led teaching next year and see how to push its limits a little more every year! Doing live demos on Jolie during lectures is so much fun (you should see the faces of people when I hack a multiparty web chat server or a prototype cloud computing solution in Jolie in about 10 minutes), but I think that I can transform some of them into videos and apply some more flipped classroom techniques along with other videos from the research community on session programming. We'll see.. and suggestions are welcome!

    PS: something that baffled me. I could not find a good tutorial for master students on CCS and the pi-calculus. I had to prepare introductory lectures by myself, which was also quite a bit of work. But maybe they are out there and I just could not find them?

    August 21, 2014

    Posted by Claudio Guidi

    An abstract means for dealing with service and microservice componentization

    Some months ago I read this article about microservices and I found it very interesting because of the experience we have had so far in the development of Jolie language. Jolie was born as a language for crystalizing SOA principles in a specific domain language but, day by day, we were becoming more and more confident that the boundaries of SOA were not enough for dealing with some issues that we had encountered during programming with Jolie. Thus, I was very surprised and excited when I read about microservices because I think they enlarge the domain  of service oriented programming in a way which is coherent with the results we have had so far. This is why I would like to share our experience with the microservices community. On the one hand my aim is to promote the usage of Jolie as one of the technologies which could help in the design and the development of microservices, and on the other hand I hope to give my contribution in the definition of concepts and practices for microservices. In particular, in this post, I would like to introduce a simple concept called Service Threshold which could be of help when reasoning about service and microservice componentization. The main idea behind the service threshold is to provide an abstract boundary which allows for the separation among a world of services and a world of components, where a component is a general piece of software.

    The Service Threshold
    The Service Threshold is the architectural line which separates services from other components (that are not services) as shown In Fig 1. where I simply represent services as hexagons and other components as squares.

    Fig 1. The Service Threshold abstractly separates services (hexagons) from 
    components that are not services (squares).

    It is a very simple concept but some questions need to be addressed:

    Why do I need to introduce it?
    A line always represents a division, in this case a logical division between two kinds of things that we consider different. I introduce it because it helps me to change my perspective about distributed software architectures. Indeed, I need to think of them in a way which allows me to design, deploy and manage them quickly and easily. Summarizing, I need that line because I am looking for a simplification of distributed and heterogeneous systems engineering. 

    Are there different properties above and under the service threshold?
    Clearly, there are differences between the top plane and the bottom one. If I operate "above" the line I would like to be in a world where a service is the basic element which can be composed and managed by following some specific rules related to services, no other kind of software components exist. Otherwise, if I am "under" the line, the service is just something I have to create, it is the target of my work.

    Which are the differences between a service and component?
    A service is an autonomous running piece of software able to provide its functions when invoked. Its execution does not depend on components out of those strictly required.  The service functions should be always available for invocations, if not the service should reply with a specific fault message. Invocations are always achieved by means of message exchanges which could be synchronous or asynchronous. It does not matter the transport protocol used for the message passing. A service can be stateless or stateful.
    A component is a piece of software which cannot be considered as a service.

    Which is the right place to put the threshold line?
    This is the core question every distributed software architect should deal with. Depending on the answer, I would design a different system with different properties. If I put this line very high I will have few services and a lot of components, if I put this line at a low level I will have a lot of services and few components. So, the question could be changed in: how many services do I need? Again, which are the basic properties that I need to take into account in order to identify all the required services?

    The service threshold is not a line but a boundary
    A distributed system is a system where software artifacts are separated, thus even though I have discussed the service threshold as a horizontal line, it fits better to think about it as a boundary which allows me to isolate and identify the different services involved into a system.


    Fig 2. The Service Threshold is a boundary which allows me to isolate and identify the services.

    Once defined, the service  threshold allows me to focus on the single service without being worried about its connections too much. I can just limit my effort to the development of the service as an autonomous specific provider of functions. The final target is to provide a basic set of services which represent the core functions of the application I am interested in. Some of them could be bought from a third party supplier, the others could be developed from scratch. The most important think is that each service function must be well separated from the others and, at the same time, it must be strictly necessary to the overall system. As an example, let me consider the human body where each organ supplies a well defined function which is fundamental for life. They are precisely separated by means of different tissues and they are characterized by different biological properties. 

    All of them are needed, because the functions that they supply are needed, nothing is too much, nothing is too less. They are connected in a perfect and inexpensive way, because they all participate to form the human body, a more complex and completely different organism. 

    Service connections and dependencies
    Once all the services have been identified, I can deal with service communication and connections. A communication is performed when a message is sent by a service and received by another one, where a message is a limited set of data transmitted in a limited interval of time. A connection represents the possibility to perform a communication between two services: a sender and a receiver. Usually a service receiver is not aware of the sender (this is not true in the case of a stateful long running transaction). A connection becomes a dependency when the sender needs to invoke another service in order to accomplish its own tasks.
    Fig 3. Service A has a dependency on service B.
    Service dependencies allow me to divide services in two main categories: 
    • Autonomous services
    • Depending services
    Autonomous services usually control resources such as databases, electronic devices, computational resources, legacy systems, etc. These kind of services are the software artifacts which enable the integration of  the underlying resource with a system of services and the APIs that they provide represent the only way for interacting with it. They are usually designed to be independent from the context they will be inserted in and they are strongly loosely coupled.

    Depending services are built upon other services because they require to call other services to accomplish their tasks. They could be:
    • Mixed services
    • Pure coordination services
    Mixed services control resources as the autonomous ones (e.g. a database) but they also need other information from other services for providing all their functions.
    Pure coordination services do not control any data resource but they just call other services for achieving their tasks (orchestrators are usually pure coordination services).
     

    Fig.4 Autonomous services, Mixed services and Pure coordination services
    By componing these kind of services it is possible to obtain a complex distributed system which provides a new set of functions different from those of the constituent ones. I call service complex system a distributed system whose components are all connected services.

    Fig. 5  A service complex system is a distributed system whose components are all connected services.


    The service threshold can be applied over a set of services
    A service complex system could be formed by tens or hundreds of services. Clearly, when the number of services is very high it could be difficult to manage and maintain the system because of its complexity. In this case, in order to simplify the management, I exploit another interesting property of the service threshold: I apply it over a set of existing services by grouping them. The services grouped by a service threshold can be seen and treated as a single service. In this case all the inner connections are hidden for an external observer and the only ones that are relevant for us are those that can be observed outside the service threshold boundary. 
    Fig 5. The service threshold can be applied over a set of services in order to group them
    and identify them as a single service.
    There are no limits to the application of the service threshold over a set of services, thus I could apply it again and again in order to reduce the system to a limited set of services. Since a service is usually stateless and always ready to serve different clients concurrently, it is possible that the same service could be enclosed by two (or more) different service thresholds as in the following picture:

    Fig 6. A service could be enclosed by two different service thresholds. 

    In this case the resulting services both contain the same service inside. Is this possible? Yes, in the following section I will answer to this question.

    Primary thresholds and abstract thresholds
    As I said before a service is always a running piece of code. Thus, a set of files in our file system is not a service, it is just a set of file which could contain executable code but they are not a service. They start to become a service when they run. This is a fundamental point because a service exists only if it is able to respond to our requests. This is why it is possible to double a service by wrapping it into two different service thresholds as represented in Fig 6. It is possible because I am grouping the service functions, not the piece of code. I call abstract service threshold all the service thresholds which group existing services. When I apply the abstract service threshold to a set of services I loose the deployment details of a service (where it is, how it is composed, which technology it uses, etc) because I am focusing only on its functions. On the contrary, the primary service threshold is applied to a set of components that, when executed, they become a service. By definition. the primary service threshold has a completely different nature with respect to the abstract one because it will be applied to components for tting services. Nevertheless, I both call them service threshold because they both allow me to identify a new service even if they are applied to different domains.

    Fig 7. The Abstract Service Threshold (on the left) groups existing services, the primary service threshold groups a set of components that, when executed, they become a service.

    The primary threshold is a technological threshold
    The Primary Threshold is a technological threshold because it clearly defines the boundary between service composition mechanisms and other technologies mechanisms. It does not matter which kind of technology I select, but it is important that I will be able to satisfy some minimum requirements that allow me to create a service (I suppose you would ask me which are the minimum requirements for creating a service, wait wait wait I will discuss them in another post). Within the primary threshold I could exploit any kind of technology and any kind of communication approach (such as in memory data exchanges, file exchange, etc.).  I can also use specific applications such as web servers, application servers, etc. The final target is to prepare the basic software layer which allow me to jump into the service world. Everything happens inside this threshold is a matter strictly related to the technologies I chose for building the service. This point is important because allow me to state that there exists a trade off between the benefits obtained by the introduction of a service and the technological overhead I have to pay. Is it simple to maintain a service developed with a specific technology? Which skills I require for achieving the development? Does the technology easily scale with the service load? Moreover, do I need to adopt the same technology for all the services of our system or do I need to change it? When? Will I be able to maintain the required skills for managing all the existing services I have? These are not trivial issues because more is the number of technologies I adopt, more is the human resource knowledge of my team I have to manage. Indeed, if I adopt a technology I need someone who is able to manage it. 

    Conclusions
    In this post I discussed service and microservices componentization by introducing the concept of service threshold which is a useful tool for approaching distributed systems based on services and microservices. I hope this post could be useful for those people that are trying to approaching service system design in a simple and intuitive way. Service thresholds could be a useful means for structuring the design and defining the development phases of a service system.  


    Posted by Claudio Guidi

    A service oriented Web Server

    In this post I would like to discuss how a web server should be considered in the context of a Service Oriented Architecture or a distributed system in general. Usually, the web server is used as a tool which enables the possibility to public files and applications  under the HTTP protocol. Indeed, applications are developed by using some specific technology like PHP, Ruby, Java, etc. and then deployed into a web server for making them available on the web. The same approach is adopted also in Service Oriented Architecture where web services and orchestrators are developed in a given technology and then published in a web server together with the WSDL documents.

    Here I want to share one of the results we obtained by developing Jolie as service oriented programming language. In Jolie the web server is just a service which provides its operations under an HTTP protocol and it follows the same programming rules we use for simple services and orchestrators. Here you can find the code of Leonardo, which is a web server completely developed in Jolie:

    http://sourceforge.net/p/leonardo/code/HEAD/tree/trunk/leonardo.ol

    You can download the code here and try to execute it by simply typing the following command in a shell:

          jolie leonardo.ol www/

    where  www/ is the root folder where files are retrieved by Leonardo (remember to create it if it is missing). Now try to add a simple index.html file into the www folder like the following one:

    <html>
    <head/>
    <body>
    Hello World!
    </body>
    </html>

    Then open your browser and set the url: http://localhost:8000 and you'll see the html page displayed in it. Try also to add images, css, html pages and subfolders into the www/ folder.
    You can change the port by setting the Location_Leonardo parameter of the config.iol file. 

    But,
              how can I program the server side behavior of a web application?

    And here comes the service oriented approach. First of all let me introduce a new operation called test into the HttpInput by modifying its interface as it follows:

    interface HTTPInterface {
    RequestResponse:
    default(DefaultOperationHttpRequest)(undefined),
    test
    }

    Then let me add the implementation of the test operation into the main:

    main {
    [ default( request )( response ) {
    /* ... code of the default operation */
    } ] { nullProcess }

    [ test( request )( response ) {
    format = "html";
    response = "Test!"
    }] { nullProcess }
    }

    Operation test executes a very simple piece of code that is:

    • setting the response format to "html" in order to make the browser able to manage the response as an html file
    • set the response message with a simple html page

    Now if we relaunch Leonardo and we point the browser to  http://localhost:8000/test we will see the page generated by the test operation. 

    This is a pretty standard way for programming a web application: joining a web page to each operation. You can do it but I don't like it, even if in some cases it could be useful. I prefer to completely decouple the server side from the client side. I prefer to public the operation test as a JSON API which can be used by the client using an AJAX call. In this way I can program the server side as a service and the client side as a dynamic application. In order to this I modify the interface for introducing the message types which are very useful for catching message faults:
    type TestType: void {
    .message: string
    }

    interface HTTPInterface {
    RequestResponse:
    default(DefaultOperationHttpRequest)(undefined),
    test( TestType )( TestType )
    }

    Then I modify the implementation of the test operation:

    [ test( request )( response ) {
    response.message = request.message + " RECEIVED!"
    }] { nullProcess }

    Finally, I just insert a little bit of Javascript on the client side by modifying the index.html:

    <html><head>
      <script type="text/javascript" src="http://code.jquery.com/jquery-2.0.3.min.js"></script>

      <script>
     function jolieCall( request, callback ) {
         $.ajax({
              url: '/test',
              dataType: 'json',
              data: JSON.stringify( request ),
              type: 'POST',
              contentType: 'application/json',
              success: function( data ){callback( data )}
         });
     }
     
     function afterResponse( data ) {
    $( "#test" ).html( data.message )
     }
      </script>
      
    </head>
    <body>
      <div id="test">
    <button onclick="jolieCall( {'message':'Ciao'}, afterResponse )">TEST</button>
      </div>
    </body>
    </html>

    That's all! Try to open the index.html page on the browser and click on TEST button. It is worth noting that Jolie recognize the JSON format and sends back a JSON message.

    Now, try to add other operations and create a more complex web application.
    Have fun!


    May 13, 2010

    Posted by Fabrizio Montesi

    Web Services for human beings

    No, this is not related to *buntu. But this is looong... and now I've got your attention. :-)


    Interacting with Web Services usually implies reading some WSDL (Web Service Description Language) document. Unfortunately, WSDL documents are written in XML and can be pretty complicated, leaving the user with no other choice than to use some graphical development tool in order to understand them.

    Lately at italianaSoftware we had to handle a lot of Web Services stuff, equipped with long WSDL documents and complex data types. Jolie already had support for the SOAP protocol, but one had to code the interface and data types of the Web Service to invoke by hand. So we started a new project aiming to enable Jolie for the automatic usage of WSDL documents, mainly composed by two parts: a tool and an improvement to the Jolie SOAP protocol. Of course both things are open source and available in the trunk branch in the Jolie svn repository!

    wsdl2jolie (whose executable is now installed by default if you install Jolie from trunk) is the newly developed tool in question. It simply takes a URL to a WSDL document and automatically downloads all the related files (e.g., referred XML schemas), parses them and outputs the corresponding Jolie port/interface/data type definitions.
    Let us see an example. See this WSDL document for a service that calculates prime numbers: http://www50.brinkster.com/vbfacileinpt/np.asmx?wsdl. Reading the raw XML is not so easy, or at least requires some time.
    Let us see what we get if we execute wsdl2jolie http://www50.brinkster.com/vbfacileinpt/np.asmx?wsdl :

    Retrieving document at 'http://www50.brinkster.com/vbfacileinpt/np.asmx?wsdl'.
    type GetPrimeNumbersResponse:void {
    .GetPrimeNumbersResult?:string
    }

    type GetPrimeNumbers:void {
    .max:int
    }

    interface PrimeNumbersHttpPost {
    RequestResponse:
    GetPrimeNumbers(string)(string)
    }

    interface PrimeNumbersHttpGet {
    RequestResponse:
    GetPrimeNumbers(string)(string)
    }

    interface PrimeNumbersSoap {
    RequestResponse:
    GetPrimeNumbers(GetPrimeNumbers)(GetPrimeNumbersResponse)
    }

    outputPort PrimeNumbersHttpPost {
    Location: "socket://www50.brinkster.com:80/vbfacileinpt/np.asmx"
    Protocol: http
    Interfaces: PrimeNumbersHttpPost
    }

    outputPort PrimeNumbersHttpGet {
    Location: "socket://www50.brinkster.com:80/vbfacileinpt/np.asmx"
    Protocol: http
    Interfaces: PrimeNumbersHttpGet
    }

    outputPort PrimeNumbersSoap {
    Location: "socket://www50.brinkster.com:80/vbfacileinpt/np.asmx"
    Protocol: soap {
    .wsdl = "http://www50.brinkster.com/vbfacileinpt/np.asmx?wsdl";
    .wsdl.port = "PrimeNumbersSoap"
    }
    Interfaces: PrimeNumbersSoap
    }

    which is the Jolie equivalent of the WSDL document. Those ".wsdl" and ".wsdl.port" parameters are the aforementioned improvement to the SOAP protocol: when the output port is used for the first time, Jolie will read the WSDL document for processing information about the correct configuration for interacting with the service instead of forcing the user to insert it by hand.
    Now we can just put this into a file, say "PrimeNumbers.iol", and use the output ports we discovered from Jolie code. As in the following:

    include "PrimeNumbers.iol"
    include "console.iol"

    main
    {
    request.max = 27;
    GetPrimeNumbers@PrimeNumbersSoap( request )( response );
    println@Console( response.GetPrimeNumbersResult )()
    }

    That little program will output "1,3,5,7,11,13,17,19,23". The example can be downloaded from this link (remember, it requires Jolie from trunk).

    wsdl2jolie acts as a nice tool for being able to use the typed interface of a Web Service from Jolie and for getting a more human-readable form of a WSDL document (its Jolie form).

    Another nice feature we get for free from the SOAP protocol improvement is that now MetaService can act as a transparent bridge towards Web Services. Simply call the addRedirection operation with the right protocol configuration (in this case, the ".wsdl" and ".wsdl.port" parameters) and MetaService will automatically download the WSDL document (we also have a cache system for them already in trunk!) and make it callable by clients.
    This is a huge step forward for client libraries such as QtJolie: just tell where the WSDL document is and you can already place calls to the Web Service of interest.
    Also, by using wsdl2jolie first and then tools such as jolie2plasma one could use the Jolie intermediate representation for transforming a Web Service interface definition into a (KDE) Plasma::Service XML one. With the same trick, one could also write C++ generators for QtJolie and introduce ease and type-safeness to Web Services invocations. I and Kévin sure have things to think about now. ;-)

    Beware that the code is still in its early stages. It already works with a lot of Web Services, but we're still improving it. For instance, we currently support only the SOAP document format (which is the one people advise to use, anyway). Patches, comments and reports are welcome in jolie-devel @ lists.sourceforge.net!

    September 09, 2009

    Posted by Fabrizio Montesi

    Random thoughts: getting ready for Jolie 1.0 and remote services inclusion in KDE

    Jolie is getting into shape for the 1.0 release. This is requiring a lot of work, and whenever we reach our current objectives we feel like improving things even further (every developer likes perfection, the problem is it requires too much time)! Nevertheless, the line for a good 1.0 release has been placed and is getting very near. The two major points we need to address are SSL support (both in SODEP an HTTP) and documentation. By documentation I'm not just referring to the language documentation but also to the source code documentation of the interpreter; the aim is to make the Jolie interpreter more approachable by hackers who want to improve it or want to write an extension (protocols, specifications, hardware support, ...). The 1.0 release will also contain experimental support for XML-RPC.

    SSL support is already working and waiting for inclusion (rough edges need to be investigated) in the main source tree. SSL support will particularly benefit B2B applications, which until
    now had to get security by wrapping Jolie in SSL containers, and the KDE project, which has just included support for remote services in trunk using Jolie and MetaService as backend technologies. SSL support in Jolie would allow KDE remote Plasma services to encrypt their data exchanges, an essential feature when dealing with sensitive data (though the initial handshake part is already made secure by using Qt).
    I must say that the KDE inclusion was pretty quick: it was just one year ago when the Jolie and the Plasma teams met at the first Tokamak and now we already have a first functioning version in trunk. This process required quite a lot of work and skill, from writing a service orchestrator that could dynamically create bridges from Plasma to services using different protocols and technologies onward to the QtJolie gluing layer and the final implementation in the Plasma library. Now we have the basis for constructing even more complex frameworks and applications, and all of this will benefit from the future Jolie static analysis tools that we (the Jolie team) are planning to implement. All of this to say that it really is a pleasure to work with the KDE team: I will be sure to keep in touch as much as possible.

    I'm now focusing heavily on source code documentation and polishing (for which I've just made some pretty big commits). Regarding code polishing, I got some help from the FindBugs static analysis tool; you can just ask it to read some Java bytecode and it comes up with a lot of good hints for performance improvements and bug removals. It is really easy to use and it even considers correct resource locking and releasing. Pretty useful if you have old code to maintain or for checking new extensions before merging them.


    Ah yes, I should also blog about a lot of other features and the porting of Jolie to ARM-based devices thanks to the Jalimo project... but that's for other blog posts!

    January 21, 2009

    Posted by Fabrizio Montesi

    MetaService enters the main source tree

    Today I have moved MetaService from JOLIE's playground source tree to the main source tree, the one that gets installed when you type ant install.

    A short description for people that do not know what MetaService is: MetaService is a JOLIE service that allows you to dynamically bridge applications that speak different protocols (sodep, soap, http, etc.) or use different communication transports/mechanisms (sockets, local sockets, Java RMI, etc.). It has been developed as an answer to the Plasma project's needs to have a transparent means to introduce Plasma to the Service-oriented world. The project turned out to be so useful that I have built a Java library for interacting with and dynamically loading it (metaservice-java, to be found in trunk/support/metaservice-java) which we, as italianaSoftware, have already used in the scope of web application development (I can not say much more on this topic... yet ;).

    As a bonus, *nix users get a launcher script for MetaService when installing JOLIE with ant install. Just type metaservice -h and a help screen will welcome you.

    The script supports passing MetaService's input port location and protocol as parameters. This means that for KDE launching MetaService just became a matter of executing something like:

    metaservice localsocket:/tmp/ksocket-$USER/metaservice

    Hooray for simplicity!

    January 20, 2009

    Posted by Fabrizio Montesi

    Jolie Jobs (now with Plasma)

    The Jolie web site now sponsors a "Jobs List", which can be found from the Contribute page, also known as "What we lack for world domination". The list features some Junior Jobs, so it's a good starting point for people that are looking for ways to start contributing to Jolie. We are continuously filling in new jobs (there's *always* something to do, as in every project ;), so be sure to refresh it if you're interested. If you don't find anything that interests you, remember that you can always post to the mailing list. =)

    Now that JOLIE supports message types, it is feasible (and not too hard) to make automatic generators for Plasma::Service description files. These jobs are listed, so if someone wants some cool tool for automatic type-safe code generation between Plasma and JOLIE sooner rather than later and to learn something more about how Plasma and JOLIE communicate, this is a good occasion to jump on the JOLIE<->Plasma revolution train. ;)

    January 19, 2009

    Posted by Fabrizio Montesi

    Compiling JOLIE from sources in Windows now easier

    For everyone using Windows out there: compiling JOLIE from sources in Windows is not a pain anymore!

    I've patched the build system so that it recognizes the running operating system and installs the appropriate launcher script (a shell script in *nix and a .bat script in Windows), so now having a working environment is far less "manual" than before. You can find the updated guide here (Windows -> From sources).

    January 12, 2009

    Posted by Fabrizio Montesi

    Jolie's new website is up!

    JOLIE has a new and shiny website: www.jolie-lang.org

    The new site has a wiki which we are filling in with a lot of information, minute by minute. Some tutorials are already final and we are preparing demos and packages of source code examples.

    This step required some effort and time, but we now have a serious and stable basis to build our knowledge base upon (the backend of the website is quite powerful and allows for easy administration and data versioning).

    The mailing list and IRC channel are already active since some time, respectively jolie-devel@lists.sourceforge.net and #jolie on freenode, providing the means for asking about anything related to JOLIE and getting the attention of the JOLIE team.

    October 10, 2008

    Posted by Fabrizio Montesi

    Opening MetaService to Java: the MetaService-Java API layer.

    Today, MetaService-java hit Jolie's SVN. You can find it in /trunk/support/metaservice-java. Basically, MetaService-java is a Java API abstraction layer for interacting with MetaService, so that you can use it as if it were a Java object.

    This project was born for two reasons:
    - it will be part of a solution for the integration of Java enterprise web applications with Service-Oriented Computing;
    - it is a good example of how to implement an API abstraction layer to MetaService from an Object-oriented language, so its source code could be a useful reading for the Plasma::Service developers.

    I'll let the code (and its comments) and a simple example speak for themselves. They're in Java, but they should be pretty easy to understand even for people non-proficient with it. Example follows.


    MetaService metaService = new EmbeddedMetaService(); // Create a MetaService instance.

    // Set up access to a SOAP Web Service.
    MetaServiceChannel myWebService = metaService.addRedirection(
    "MyWebService", // Resource name to assign
    "socket://www.mywebservice.com:80/", // Service location
    Value.create( "soap" ), // Protocol to use
    Value.create( "My metadata" ) // Descriptive metadata
    );

    // Done! Let's communicate with it.
    myWebService.send( "getNameById", Value.create( 4 ) );
    Value response = myWebService.recv();
    System.out.println( response.strValue() ); // Will print the name.


    The API is still to be refined, but pretty much usable already. =)

    October 02, 2008

    Posted by Fabrizio Montesi

    Local socket support in JOLIE

    This fall looks good so far! A lot of JOLIE sub-projects are taking shape, and are going better than anticipated. But I'm blogging about something which has just landed on JOLIE's svn.

    JOLIE now supports local sockets (a.k.a. unix sockets). How to use them? Let's see it by comparison; a communication interface over tcp/ip sockets is exposed like the following:

    inputPort MyInputPort {
    Location: "socket://localhost:9000"
    Protocol: soap
    Interfaces: MyInterface
    }


    Let's switch that to a local socket:

    inputPort MyInputPort {
    Location: "localsocket:/tmp/mylocalsocket"
    Protocol: soap
    Interfaces: MyInterface
    }


    That's it! Use the localsocket scheme in the Location URI, and supply the right path to it.

    August 12, 2008

    Posted by Fabrizio Montesi

    Open collaboration services have been Jolied!

    Frank's talk was great: bringing the community into the KDE desktop is actually a great idea.
    Even better, he's building up REST services for accessing the data! Of course, as we're speaking of services, my mind went directly to "Hey, these things are Joliable".

    I've already started collaborating with Frank in order to discuss the API and a set of specifications for publishing a machine-readable format for reaching said services.

    While I'm at it, someone could wonder "Just how difficult is to use these things with JOLIE?". The answer, of course, is "damn easy" (screencast). ;)

    So, once we're finished with the JOLIE<->Plasma integration, expect nice graphical frontends to pop up. =)

    August 09, 2008

    Posted by Fabrizio Montesi

    Are we converging to a service-oriented experience?

    The more I speak with people at Akademy, the more I'm convinced that contacting KDE to start the JOLIE<->Plasma integration thing has been the right decision.

    It looks like the whole KDE community (users and developers) is a perfect match to start "porting service-oriented computing to the masses". There is an incredible need for integration, communication, sharing and service accessibility. And when I see talks like the one given by Frank Karlitschek (KDE Community websites: The past, the present and a vision for the future) I get the feeling that we are all trying to converge to a service-oriented experience. We have the right tools, let's make that happen!

    I do really hope to clarify what this means and how we can do it with my presentation this afternoon, so... here I am checking that the demos are still working. Murphy's law, begone! ;)

    August 06, 2008

    Posted by Fabrizio Montesi

    Before Akademy

    Phew, the temperature is so high here in Bologna that it's difficult even to think.
    Nevertheless, this month is gonna be even hotter! So, here are some quickies of the current things that are happening.

    Akademy, Akademy, Akademy...
    I'm finishing my preparations for Akademy, and the presentation is coming out nicely. I'm trying to make it understandable even by people who does not know much about Service-oriented Computing. I will also demo a couple of service-oriented applications for KDE (of which you could have seen screencast/shots in the kde blogs), and explain how they work and how easy is to make them with JOLIE.
    By the way, if you need/want to talk with me, I'll be at Akademy the 9th, the 10th and the 11th (the 11th only in the morning).

    Rising interest
    We, as italianaSoftware, are receiving more and more interest in JOLIE by the business world. It looks like this could produce some open-source products. We shall see what comes up. The industry involvement has already produced some good things, like the sponsoring of the Google Web Toolkit integration in JOLIE (we are already using this in the open-source world, too: Echoes is based upon the JOLIE-GWT compatibility).

    A lot of features to come in the next months
    Some projects from the academic side are taking shape. A student from the University of Bologna, Davide Malagoli, is putting the basis for a complete bi-directional JOLIE<->XML translator. While JOLIE already supports the SOAP protocol, this is a great thing for the integration with tools designed for the Web Services technology. For instance, we will be able to generate automatically a WSDL document of a JOLIE program. As JOLIE supports more than just SOAP and TCP/IP sockets, we are already designing standard compliant WSDL elements to describe that.
    We have got other two projects in early development, I will blog about them in the near future.

    Looks like this fall and winter I will have a lot of features to put in the source tree, and a lot of screencast/shots to make. =)
    As for this coming week-end, see you all at Akademy!


    P.S.: welcome back to the blog world, Aaron!

    UPDATE! Got the permission from the student to blog his name, so here it is. ;)

    July 24, 2008

    Posted by Fabrizio Montesi

    Starting up: another blog in the cauldron.

    Hello, readers.
    So here I am, another piece in the blogosphere. This blog will be about the development of JOLIE, a language for service-oriented programming, and all of its related works.

    As of this first post, this blog is on Planet KDE; the following says why.

    If you read the planet, you should have seen JOLIE cited here and there already. Basically I'm on Planet KDE because we, the JOLIE and KDE (Plasma especially) projects, are cooperating in order to bring the power of service-oriented computing (see also: service-oriented architecture, service-oriented programming, etc.) to the desktop.
    This blog of mine is here also to explain what this means and will mean for the KDE desktop, and to show the practical results of this effort. Stay tuned with my blog if you want to know more!

    Posted by Fabrizio Montesi

    Vision: a distributed presentation framework.

    Vision is a proof of concept application (thanks aseigo for the initial suggestion), showing what can be easily obtained by using the service-oriented paradigm together with rich client applications.

    What is Vision?
    But, more importantly: what's a distributed presentation?

    Imagine that you are showing a presentation, and that someone wants to see what you're presenting... in his computer! To do so, he should connect in some way to your presentation. Whenever you change page (or make another relevant action, like changing document), the client should be notified and synchronized automagically.
    That is exactly what Vision does. It forms a network between the presenter and the clients, keeping the clients synchronized with the presentation.
    A screencast is better than a thousand words: an example usage of vision with the previewer plasmoid and kpdf.

    As you can see in the screencast, Vision already supports more than one viewer (there are shown kpdf and the previewer plasmoid, but we also already support okular).

    Another nice feature of Vision is that it forms a P2P network. A client acts also as a presentation server. Say that A is watching B's presentation. Now another client (C) can connect to A. Whenever A receives an event from B, it will now also send it to C.
    This can be exploited, for example, in order to perform bandwidth load balancing.

    Note also that Vision makes use of SODEP, a binary protocol especially designed for service communications (it can, nevertheless, use all of the other JOLIE supported protocols, such as SOAP, HTTP, GWT-RPC...). Thus, the needed bandwidth is very small and the application can handle a lot of clients at the same time.


    Exposing Vision to other platforms
    A great feature of JOLIE is that it separates the logical interfaces of a service from their deployment. Suppose that we want to extend Vision to support Bluetooth clients. The ideal solution would be to expose the same interface that we expose on the network via sockets to bluetooth. Luckily, doing that in JOLIE is very easy:


    service PresenterBluetoothService
    {
    Location: "btl2cap://localhost:3B9FA89520078C303355AAA694238F07;name=Vision;encrypt=false;authenticate=false"
    Protocol: sodep { .charset = "ISO8859-1"; .keepAlive = 1 }
    Ports: PresenterInputPort
    }


    What do these lines mean?
    Well, they're telling JOLIE to expose the PresenterInputPort interface (which is the Vision presenter interface) on the pointed bluetooth Location using the SODEP protocol to encode and decode data. You can see the whole presenter service code in the JOLIE SVN repository (/trunk/playground/vision).

    Guess what? It just works! (video) And what's better than a mobile phone to test it?

    By the way:


    just in case you want to see more service-oriented goodness in KDE. =)