Azure Service Fabric

陈辉煌

　　Concept
Benefit of using SF
Sample SF architecture.
Code Structure
ServiceManifest
　　Placement Constraint
　　service affinity
Stateful Service
      Replica Roles
　　Actor Pattern
　　State management
Stateless Service
ApplicationManifest
Service Package
Service Upgrade
Load Balancer
　　service failover
　　Scalability
　　AutoScaling
　　Event Sourcing
　　Monitoring
　　Communications
　　FSM (Finite State Machine Actor)
　　Multi-Tenants Pattern - Tenant Manager
　　ICommunicationListener
　　Hosting Guest Application
　　Concept:

• Node- a smallest unit within a cluster to provide compute capablity
  
• Nodetype - a logical naming of the tier / layer in the architecture so it can easily tag to the node, e.g. Webtier (UI + API) and ServiceTier(Backend)
  
• SF - abbreviation for Service Fabric
  
• Application type -
  
• Instance - stateless service term for unit of service that is behind load balancer
  
• Partition - a unique service type that derive from base service type that match with partition criteria
  
• Replica - stateful service term, not applicable to stateless, definition a copy of code and data (state) of a service
  

　　Benefit of using SF


·Service Fabric is the Windows Azure microservice platform.

·Cost - Container concept provides the flexibility of placement of multiple services in single container or node.

·Environment reuse - Compatible for deployment in environment on Cloud (AWS or Azure), On premises, cross platform(Microsoft or Linux)

·Architectural support - Built in capability of load balancing and self-healing via clustered multi-instances or replica.

·Individually upgrade – supported, good level of isolation.　　Sample SF architecture.

• Website (stateless)
  
• API(stateless)
  
• Auction (stateful)
  

　　Stateless service – Only for collecting the data and is not responsible for data integrity
　　Stateful service – Responsible for data persistence and ensure data integrity. Share data across the nodes.
　　Traditionally each architectual layer will take up one tier / node and form a cluster, SF provides flexiblity of placement where you can group layers in just one node under the same NodeType

　　Code Structure


·A root level of Application project will be created accompany with creating a new service fabric project  for individual service, either it is stateless or stateful service.

·Application can be a top node in a microservice Hierarchy structure and manage and supervise multiple sub-services.

·Service can be a web application, background jobs, Backend code etc.

  

　　ServiceManifest
　　Each SF project either stateful or stateless has a ServiceManifest xml to indicate the entry point of the service being hosted, it is also designed for Docker image usage in SF.
　　ServiceInstanceListener is the communication channel to sync up on the status of the serivce, when it is not given, sf explorer would not be able to detect service existence.
　　RunAsync method is the entry point for the functional code of the service
　　this.Context.ServiceName.LocalPath this.Context expose properties of service per se
　　Configuration - endpoint elements, if fixed then dup when scale out.

You can define custom metrics in several places. First, you can define these metrics in service manifest files, as shown in the following example. The manifest defines two metrics: MemoryInMband DiskInMb. The MemoryInMb metric has high weight, and it has default value of 100 (MB) for primary replicas and 50 (MB) for secondary replicas.
<ServiceTypes>
  <StatefulServiceType ServiceTypeName="Stateful1Type" HasPersistedState="true">
    <LoadMetrics>
      <LoadMetric Name="MemoryInMb" Weight="High" PrimaryDefaultLoad="100"
SecondaryDefaultLoad="50"/>
      <LoadMetric Name="DiskInMb" Weight="Medium" PrimaryDefaultLoad="1000"
                    SecondaryDefaultLoad="500"/>
    </LoadMetrics>
  </StatefulServiceType>
</ServiceTypes>

Placement constraint
Used to define condition whether service can be placed to node



<ServiceTypes>
<StatelessServiceType ServiceTypeName="Stateless1Type">
<PlacementConstraints>(NodeTypeName == WebApp &amp;&amp; myValue &gt;=4)</PlacementConstraints>
</StatelessServiceType>
</ServiceTypes>
　　Service Affinity
　　To achieve better performance, 2 services can be bound to the same node



<ServiceTemplates>
<StatelessService ServiceTypeName="Stateless1Type" InstanceCount="5">
<SingletonPartition></SingletonPartition>
<ServiceCorrelations>
<ServiceCorrelation Scheme="Affinity"
ServiceName="fabric:/otherApplication/parentService"/>
</ServiceCorrelations>
</StatelessService>
</ServiceTemplates>
　　For Stateful service, Reliable dictionary is where state can be shared accross replica, StateManager manage all these dictionaries, reliabled collection is not centralized cache but is replicated copy in other replica.
　　Reliable queue -

　　SF restrict there can only be *one replica* per partition at any time, data changed are replicated to secondary replica.
　　Replica Roles
　　Primary - each service has 1 primary replica in the set that responsbile for writing
　　Active secondary -
　　Idle secondary -
　　If you click each of the replicas, you’ll see that only the primary
replica is bound to a listening address, indicating all reads and writes are
going through the primary replica

　　Usually the stateless service are for web tiers like UI or API, those can be loadbalanced with naming service

　　Actor Pattern
　　Actor service is pretty like object in Object Oriented Programming, it has its own behavior and maintaining its state, they are not acting on their own and tend to interact with other objects.

　　Actor service is build on top of stateful service.
　　It can use different state persistence machanism, state can persist to hard disk or memory

　　State management
　　State are shared and can be write and read concurrently, similar to data store like sql server, transaction isolate level for this  
　　Snapshot isolation level is selected for most of cases for Reliable Dictionary and Reliable Queue. Only Primary Replica will impose update lock on reliable queue

　　Application Manifest


·Partition - a unique service type that derive from base service type that match with partition criteria

Microsoft.ServiceFabric.Services.Communication.Client.ServicePartitionClient.ServicePartitionClient(ICommunicationClientFactory<TCommunicationClient> factory, Uri serviceName, long partitionKey); has the approach to help with declaring a partitionkey.

·Replica - stateful service term, not applicable to stateless, definition a copy of code and data (state) of a service. each Replica will take one node as primary;  others are secondary active by default i.e.

　　ServiceManifestImport xml node reference the service manifest
　　DefaultServices xml node implying the included service will be automaticaly initiated right after applicaiton is setup




Service Package Structure



·A service performs a complete and standalone function (it can start and run independently of other services) and is composed of code, configuration, and data. For each service,

·code consists of the executable binaries,

·configuration consists of service settings that can be loaded at run time, and

·data consists of arbitrary static data to be consumed by the service. Each component in this hierarchical application model can be versioned and upgraded independently.



Service Upgrade

ServiceManifest - Version attribute - 2.0.0.0

                           CodePackage - 2.0.0.0

ApplicationManifest - ApplicationTypeVersion - 2.0.0.0

                               ServiceManifestRef - ServiceManifestVersion - 2.0.0.0

However, in a production environment, usually you want to perform a
rolling update to maintain service continuity. With rolling updates, instances
of the older versions gradually are taken down as instances of the newer
version are brought online.



Another example:

• Update only the config setting
  

• Leverage the setting in stateless1 code
  

  protected override async Task RunAsync(CancellationToken cancellationToken)
{
// TODO: Replace the following sample code with your own logic
//       or remove this RunAsync override if it's not needed in your service.
var config = Context.CodePackageActivationContext.GetConfigurationPackageObject("Config");
var test = config.Settings.Sections["MyConfigSection"];
var step = Int64.Parse(test.Parameters["IncrementStep"].Value);
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
ServiceEventSource.Current.ServiceMessage(this.Context, "Working-{0}", iterations += step);
await Task.Delay(TimeSpan.FromSeconds(5), cancellationToken);
}
}

• Update the servicemanifest.xml in section of service version to 2.0.0.0
  
• Update the version in config.package to 2.0.0.0
  
• Update the Applicationmanifest.xml in section of application version to 2.0.0.0
  
• Publish the application using upgrade option
  

　　Setup a load balancer (Chapter 5 Service deployments and upgrades)




ICommunicationListener

  

　　Service Failover
　　In the context stateless service. A new node will be chosen to service if the node in used is breakdown or restarted. SF will choose the new node to replace the malfunctioned node a failover method.
　　Create a sample stateless service, set its replacaInstance as 1, meaning there is only 1 instane is needed to run the service, deactiate (restart) the service in SF Explorer. then a new node will be chosen to host the service. what it shows in the Diagnostic Event is as follow:

　　Scalability
　　Horizontal scale - Replicas, apply to stateless services
　　Partitions - apply to stateful services where state maintenance is a challenge if state is shared among too many replicas,  hence allocate partition to replica is a solotion, state bound to a partition will only be accessible to replica bound to the same partition.
　　AutoScaling
　　LeveragingARM whichmanaging the resource group with Azure Resouce Manager API to update the scaleset once the consumption hit the thresdhold.





{
"type": "Microsoft.Insights/autoscaleSettings",
"apiVersion": "2015-04-01",
"name": "autoscalewad",
"location": "West US",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachineScaleSets/', 'myVMSS')]"
],
"properties": {
"name": "autoscalewad",
"targetResourceUri": "[concat('/subscriptions/',subscription().subscriptionId,
'/resourceGroups/',  resourceGroup().name,
'/providers/Microsoft.Compute/virtualMachineScaleSets/', 'myVMSS')]",
"enabled": true,
"profiles": [
{
"name": "Profile1",
"capacity": {
"minimum": "1",
"maximum": "10",
"default": "1"
},
"rules": [
{
"metricTrigger": {
"metricName": "\\Processor\\PercentProcessorTime",
"metricNamespace": "",
"metricResourceUri": "[concat('/subscriptions/',subscription().subscriptionId,
'/resourceGroups/',  resourceGroup().name,
'/providers/Microsoft.Compute/virtualMachineScaleSets/', 'myVMSS')]",
"timeGrain": "PT1M",
"statistic": "Average",
"timeWindow": "PT5M",
"timeAggregation": "Average",
"operator": "GreaterThan",
"threshold": 50.0
},
"scaleAction": {
"direction": "Increase",
"type": "ChangeCount",
"value": "1",
"cooldown": "PT1M"
}
}
]
}
]
}
}
View Code　　
Event Sourcing
　　Visual Studio automatically generate eventsource class for the logging purpose
　　If it is stateless service,  ServiceEventSource class will be generated and help to write logs to Dianostic Events

• Program.cs
  

private static void Main()
{
try
{
ServiceRuntime.RegisterServiceAsync("Stateless1Type",
context => new Stateless1(context)).GetAwaiter().GetResult();
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(Stateless1).Name);
// Prevents this host process from terminating so services keep running.
                Thread.Sleep(Timeout.Infinite);
}
catch (Exception e)
{
ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
throw;
}
}
　　ServiceEventSource.cs



         [Event(ServiceTypeRegisteredEventId, Level = EventLevel.Informational, Message = "Service host process {0} registered service type {1}", Keywords = Keywords.ServiceInitialization)]
public void ServiceTypeRegistered(int hostProcessId, string serviceType)
{
WriteEvent(ServiceTypeRegisteredEventId, hostProcessId, serviceType);
}
　　Log shown on diagnostic event

　　if you want to persist the event log to Azure, change the Name below to Microsoft-ServiceFabric-Services



[EventSource(Name = "MyCompany-Application1-Stateless1")]
internal sealed class ServiceEventSource : EventSource
　　[EventSource(Name = "Microsoft-ServiceFabric-Services")]
　　Monitoring
　　In process monitoring using AppInsight

• 1. Create a new Service Fabric application named AppInsightsApplication with a stateless service named StatelessInsights.
  
• 2. Right-click the SatelessInsights project and select the Manage NuGet Packages menu.
  
• 3. Search for “servicefabric” to locate and install theMicrosoft.ServiceFabric.Telemetry.ApplicationInsights package.
  
• 4. Open Program.cs. Add the following line to the top of the Main() method:
  

　　Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.Active. InstrumentationKey = "[your instrument key]"; Microsoft.ServiceFabric.Telemetry.ApplicationInsights.Listener.Enable(EventLevel. LogAlways);
　　Using Application Insights alert rules
　　
In addition to the preceding event listener, you can useMicrosoft.ApplicationInsights.TelemetryClient directly to send application metrics to Application Insights. In the following walkthrough, we’ll use the client to send data to Application Insights and practice how to set up monitoring UI and alert rules.
1. Continue with the previous solution. Open the StatelessInsights.cs file under the StatelessInsightsproject.
2. Insert the following line at the top of the RunAsync() method:
Click here to view code image
Microsoft.ApplicationInsights.TelemetryClient client =
     new Microsoft.ApplicationInsights.TelemetryClient();
3. Modify the while loop in the RunAsync() method (the code in bold in the new code to be inserted):
Click here to view code image
while (!cancelServiceInstance.IsCancellationRequested)
{
    ServiceEventSource.Current.ServiceMessage(this, "Working-{0}", iterations++);
client.TrackMetric("MyTest", iterations,
new Dictionary<string,string>
{{"ServiceName", this.ServiceInitializationParameters.ServiceName.AbsoluteUri} });
    if (iterations > 1000)
        iterations = 0;
    await Task.Delay(TimeSpan.FromSeconds(1), cancelServiceInstance);
}
The preceding code sends “MyTest” values to Application Insights along with the current service name as a property. The value is reset whenever it reaches 1,000.
4. Deploy the application to your local cluster.
5. In Microsoft Azure management portal, open the Application Insights instance.
6. Click the Metrics Explorer icon.
7. On the Metrics Explorer blade, click the Add Chart icon.
8. On the Chart Details blade, select Chart Type Is Line. Then, select the MyTest metric under the Custom category. (It can take several minutes for this to show up.) As you make selections, you’ll see the chart refreshed with your metric data, as shown in Figure 10-16. Note that you can select a range on the
　　chart and use the zoom in icon to zoom in or use the zoom out icon to zoom out.

　　9. Click the Alert Rules icon.
10. On the Alert Rules blade, click the Add Alert icon.
11. On the Add An Alert Rule blade, set the following values (and leave others at defaults):
· Name: Enter a name for the alert rule.
· Metric: Select MyTest.
· Condition: Select “greater than.”
· Threshold: Enter 500.
· Period: Select Over The Last 5 Minutes.
· Additional administrator email(s): Enter an email address.
12. Click the OK button to define the rule.
13. After a few minutes, you’ll start to receive email notifications. Figure 10-17 shows a sample of how these emails look. If you don’t see the emails in your Inbox, look in your spam, junk, or clutter folder because your mail system might have identified them as spam email.
　　Communication
　　Different models and patterns

• Coupling - synchronous and over chatty messaging
  
• Low coupling - using queue to decouple the microservice
  
• Aggregate
  
• Competing consumer -
  

• 
  
  •   To improve the throughput of the queue items processing, multiple instances can be setup to consume the queue items, by placing the lock on the item being worked on, can prevent other consumer work on the same item. Azure Service Bus provide such kind of lock. detail of diff between Azure ServiceBus queue and Storage queue is listed here
    
  • Set up different queues to offload the work wont be a good idea cuz it adds the overhead managing different queue and complicate the job consumer management.
    
  
  

　　FSMA (Finite State Machine Actor)
　　With Event or Message base integration between microservices we can achieve low coupling between ms, nevertherless it introduce challenge when multiple ms jointly work on a workflow type of work, once the sender send out a message to the bus and wait to be pick up by the microservice which will keep on work on the task, it will lose control how is the workflow ends up. Unless there is another queue which can be used to commnuiate the outcome of the workflow by subsequent microservice who work on the task.
　　FSMA can be a solution in this scenario.FSMA should be aware of the status in particular workflow, can be query for the latest status.
　　Also the timer of FSMA can be setup that if the time elapsed exceed the defined time limit, it will invalid the workflow can set it to fail. it also accept cancel request and cencel the workflow subsequeently.

　　Multi-Tenants Pattern - Tenant Manager
　　Tenant Manager dynamically create or destroy the tenant via managment API or SDK. provision a new tenant by adding new application instance or retire a tenant by removing existed applicatioin instance. It also responsible for life time, failover and health check of the tenant level services.



FabricClient client = new FabricClient("localhost:19000");
await client.ApplicationManager.CreateApplicationAsync(new ApplicationDescription
{
ApplicationTypeName = "MyApplicationType",
ApplicationName = new Uri("fabric:/Tenant1"),
ApplicationTypeVersion = "1.0.0"
});
　　Hosting Guest Application
　　Guest application can be run as a service but with less provision than the normal SF services, $copy command to export the binaries to package code folder.
　　you need to build customize health check to generate health report in cluster level to be seen by the sf explorer. Service Fabric SDK can help programmatically on this.
　　Host Container
　　
Can multi stateful instances write
　　Chaos Test Page 263