Sunday, July 16, 2017

Why automated end to end tests may not solve your quality issues

Our customer asked us to help us with an approach for end to end testing. The problem the teams were facing was mainly due to the errors detected during the integration phase/ environment where end to end tests were executed. During the discussions with the teams we noticed that the organizing of teams was based on loosely coupled software components. The teams worked on individual components that were versioned, tested and deployed independently. They had development or test environments were the system tests were executed as part of the build or release process. These system tests were executed against the stubs of the dependent components and verified successfully.

The teams used to get requirements per component which did not specify the entire use case, most of them were in the format of “if a message in format XYZ is given to the component, then do something and send the output to another component in the system in a different format.” Each team developed these user stories and delivered the stories with tests to ensure that the components worked fine, given the correct data is provided in the expected format.

But there was no guarantee that the whole system was working as expected. Most of the time, there were integration issues, due of the wrong versions of components in an integration environments or schema mismatches etc. Different component teams worked in isolation and failed to communicate about the recent releases to other teams. This introduced the need of a separate team to test the system as a whole. The responsibility of this team was to create end to end tests to check and ensure whether the whole system worked properly with multiple components deployed.


The end to end tests now helps in identifying  integration issues before deploying to an higher environment. Only the problem was not completely solved!!!

The issues detected by the integration test team was reported back to the component teams and which was normally not picked up in the current sprint, but has to be put into the backlog for later iterations. This introduced waiting periods and tensions between the different silo teams. Developers were working on parts of the complete applications who were not aware of how the complete system functions were not able to test the system as a whole.

End to end testing was not a solution to this situation, but just a workaround to suppress the problem from getting into higher environments. A better solution will be to organize the teams into feature teams where the team can shift their focus on value delivery and work on business features rather than technical components. Now the teams have acceptance criteria that maps a customer centric use case, rather focusing on a part of the use case. Compared to the component teams, feature teams can provide better design decisions and thus better tests because of the knowledge of the whole application,  technology stack and use cases. The automated tests delivered as part of the user stories now cover the whole feature and not only part of the application.


The feature team as a whole now have the required skills to implement the entire customer centric feature. The integration test team members are also part of the feature teams and assist in ensuring the correct functional tests are captured as part of the development process.

Friday, March 3, 2017

How to create a reliable cluster management system using SQL server for orleans

Orleans runs on a cluster of servers in a data center, each running a container process that creates and hosts actor activation. A service built on Orleans is deployed on a cluster of nodes. When deployed to a cluster of nodes, Orleans internally implements a set of protocols to discover and maintain membership of Orleans silos in the cluster, including detection of node failures and automatic reconfiguration.

Cluster management is supported in Orleans via a built in membership protocol, also referred to as Silo membership. The goal of this protocol is for all silos to agree on the set of currently alive silos, detect failed silos, and allow new silos to join the cluster. For reliable management of cluster membership, Orleans uses Azure Table, SQL Server or Apache ZooKeeper for synchronization of nodes.

This post we’ll see how we can use the SQL server implementation of the membership table and make use of the cluster management feature.

Step 1: Creating the database

Create a new database OrleansMembership in your SQL server and run the below given script to create the required tables and data for SQL server membership provider.



Step 2: Configuring the membership setup in the  SystemStore element settings: 

In the silo configuration file (OrleansConfiguration.xml), add a new section SystemStore in the Globals section with the connection string for the OrleansMembership table we created in step 1.

<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
   
    <Liveness LivenessType="SqlServer" />
    <SystemStore SystemStoreType="SqlServer"
                 DeploymentId="OrleansClusterDemo"
                 DataConnectionString="Data Source=.\SQLEXPRESS;Initial Catalog=OrleansMembership;Integrated Security=True;Pooling=False;Max Pool Size=200;Asynchronous Processing=True;MultipleActiveResultSets=True"
                 AdoInvariant="System.Data.SqlClient" />
   
  </Globals>
</OrleansConfiguration>

Step 3: Update the silo projects with the nuget packages for sql client and sql utils:

"Microsoft.Orleans.OrleansSqlUtils": "1.4.0",
"System.Data.SqlClient": "4.3.0"

Step 4: Update the client configuration with the membership table information: 

The client configuration should also be updated with the OrleansMembership table using the SystemStore element.

<ClientConfiguration xmlns="urn:orleans">
  <Tracing DefaultTraceLevel="Info"
           TraceToConsole="true"
           TraceToFile="C:\OrleansLogs\WebApiClient-{0}-{1}.log"
           WriteTraces="false"/>
  <Statistics MetricsTableWriteInterval="300s" PerfCounterWriteInterval="30s" LogWriteInterval="300s" WriteLogStatisticsToTable="true" StatisticsCollectionLevel="Info"/>
  <Messaging ResponseTimeout="30s" ClientSenderBuckets="8192" MaxResendCount="0"/>
  <SystemStore SystemStoreType ="SqlServer"
                 DeploymentId="OrleansClusterDemo"
                 DataConnectionString="Data Source=.\SQLEXPRESS;Initial Catalog=OrleansMembership;Integrated Security=True;Pooling=False;Max Pool Size=200;Asynchronous Processing=True;MultipleActiveResultSets=True" AdoInvariant="System.Data.SqlClient" />
</ClientConfiguration>

Also notice that you don't need to use the ProxyingGateway anymore for the Primary silo address. Instead the client can now use the membership data for grain references.



Conclusion:

We have created a reliable deployment that is configured with membership using SQL server. In this case, all silos are created equal, with no notion of a primary of secondary silos. This is the recommended production deployment scenario where a failure of a single node or a combination of nodes does not affect other silos in the cluster. This also means that you don't need to provide a Seed node option in your silo configuration anymore!!!

Sunday, February 26, 2017

How to create a monitoring infrastructure for your Orleans silos and grains

Orleans dashboard is a configuration based bootstrap provider for Orleans silos. The dashboard provides details on the performance of silos, individual grains and also the details on failures etc.

If you are new to actor systems and orleans, it may be helpful to read the official documentation https://dotnet.github.io/orleans/index.html



How do you use it?

First thing you want to do is install the OrleansDashboard packages via NuGet to the GrainServer project (this is where the gains are implemented)

Install-Package OrleansDashboard

Register the dashboard bootstrapper into the Orleans system

In the OrleansConfiguration.xml file add the new bootstrapper. You can mention the port number on which the dashboard will be accessible. The default value is 8080.
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <StorageProviders>
      <Provider Type="Orleans.StorageProviders.SimpleSQLServerStorage.SimpleSQLServerStorage" Name="SqlStore"
                ConnectionString="Data Source=.\SQLEXPRESS;Database=OrleansStore; Integrated Security=True;"
                UseJsonFormat="false" />
    </StorageProviders>
    <BootstrapProviders>
      <Provider Type="InfraAutomation.GrainServer.BootstrapServer" Name="BootstrapServer" />
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" Port="8081" />
    </BootstrapProviders>
  </Globals>
....


Start the silo and you can now access the dashboard at the configured port like http://localhost:8081


Saturday, February 25, 2017

How to integrate ASP.NET WebAPI and Microsoft Orleans

Orleans is a runtime and programming model for building distributed systems, based on the actor model. In an actor based programming model, actors represent multiple instances of related real-world artifacts and interact with each other as independent single threaded entities.

In Orleans the actor is represented as a grain, that contain state can be uniquely identified and expose logic via asynchronous methods. These grains are isolated from one another and can only communicate via messages. These grains are hosted on silos which houses the Orleans runtime, that performs operations like grain instantiation and look-up. Even though these silos can be connected over a network, to make request into a solo, you can create client applications like an ASP.NET WebAPI which uses the GrainClient.GrainFactory to provide remote access. This also helps to perform operations like authentication and authorization on the grain system from the web Api. This also helps to create a service that encapsulates the Orleans system so that clients don’t need to interact with the Orleans clients and do RPC, instead can make use of the RESTful Api and use HTTP for communication.

If you are looking are looking for a sample tutorial on how to create a grain server and host it locally, refer to the sample here: https://dotnet.github.io/orleans/Tutorials/Minimal-Orleans-Application.html
To get started, we have to create a .net core WebAPI project and use this to as the client project to invoke calls to the silos. Follow the steps below to use this API to invoke grain calls.


Adding nuget packages:

In the project.json file add the references to the following nuget packages.
"Microsoft.Orleans.Core": "1.4.0",
"Microsoft.Orleans.OrleansCodeGenerator": "1.4.0",
"Microsoft.Orleans.OrleansCodeGenerator.Build": "1.4.0",
"Microsoft.Orleans.OrleansRuntime": "1.4.0",


Creating the client configuration:

Orleans can be used in a variety of configurations that fit different usage scenarios, such as local single node deployment for development and testing, cluster of servers, multi-instance Azure worker role, etc. All of the different target scenarios are achieved by specifying particular values in the Orleans configuration XML files. The client configuration is used to specify the silos gateway endpoints to connect to. For this sample I am using to connect to the local silo, so we should configure the gateway to localhost. You can also use the Client configuration file to specify the Trace settings etc.
In the WebAPI project create an XML file ClientConfiguration.xml and copy the contents below

<ClientConfiguration xmlns="urn:orleans">
  <Gateway Address="localhost" Port="30000"/>
  <Tracing DefaultTraceLevel="Info"
           TraceToConsole="true"
           TraceToFile="C:\OrleansLogs\WebApiClient-{0}-{1}.log"
           WriteTraces="false"/>
</ClientConfiguration>



Initializing the GrainClient:

The connection to the grain server/ silos are established by initializing a grain client. The GrainClient.Initialize method accepts a client configuration object that will initialize the client using the configuration provided. In the Startup.cs file, use the code below to initialize the GrainClient.
public void StartSiloClient()
{
    var config = ClientConfiguration.LoadFromFile("ClientConfiguration.xml");

    // Attempt to connect a few times to overcome transient failures and to give the silo enough
    // time to start up when starting at the same time as the client (useful when deploying or during development).
    const int initializeAttemptsBeforeFailing = 5;

    var attempt = 0;
    while (true)
    {
        try
        {
            GrainClient.Initialize(config);
            break;
        }
        catch (SiloUnavailableException)
        {
            attempt++;
            if (attempt >= initializeAttemptsBeforeFailing)
            {
                throw;
            }
            Thread.Sleep(TimeSpan.FromSeconds(2));
        }
    }
}
The StartSiloClient can be invoked from the Configure method in the Startup.cs file


The controller:

Now we have initialized the GrainClient, we can make use of the static methods in the GrainFactory to get a reference to a grain and start working.
[HttpGet]
public IActionResult GetAll()
{
    var vm = GrainClient.GrainFactory.GetGrain<IGetAllVirtualMachine>(Guid.NewGuid());
    var result = vm.GetAll().Result;
    if (!result.Any())
    {
        return NoContent();
    }
    return new OkObjectResult(result);
}