martedì 12 gennaio 2016

How to Create & Test a Diagnostic Module with Scale Up action in WebLogic 12.2.1

You have learned :
Now we will see how to create a Diagnostic Module with a Scale Up action.
Inside the Diagnostic Module we will create a policy based on a Cluster High Throughput quota.
We will test it with JMeter and we will monitor it using the dashbord (http://host:port/console/dashboard).

If you have followed the previous steps on this blog, you need to deselect RCM from the Domain Partition (dp1), we do not want other side effects during the test of the Diagnostic Module:

Select WebLogic Domain->Environment->Domain Partitions:


Select dp1:


Select Domain Partition->Administration->Resource Sharing:


Under "Resource Manager Configuration" select "No resource manager" then click Save:
(Remember the Lock & Edit and then the Activate Changes):


Now we can proceed with the creation of a Diagnostic Module:
Lock & Edit the Console:
Select WebLogic Domain->Diagnostic->Diagnostic Module:


Click on Create:


Choose a name (TestModule) leave Global as Scope:


Click on the module (TestModule) just created:


Click on the tab "Collected Metrics" put 60000 (is in milliseconds) then click on Apply:


Click on the tab "Policies and Actions" then click on Create:


Choose a Policy Name (TestPolicy) then click on Next:
For our Test we need to select "Collected Metrics" as Rule Type:


In Smart Rule dropdown list choose "wls:ClusterHighTroughput()":


You can now set values in the wls:ClusterHighTroughput():

clusterName: MTCluster (is the name of our configured Dynamic Cluster)
period: 10s
duration: 30s
throughputLimit:250
percentServerLimit: 60

Click on Next:


Select "Every N seconds" with a frequency of 15 seconds, then click on Next:


Select "Use an automatic reset alarm" ant choose 60 as Automatic Reset Period, then click on Next:


Now we can create a Scale Up Action and assign it to the policy that we are creating:
Click on "Create Scale Up Action":


Choose a name (TestScaleUp), select MTCluster as Service Name, then click on Create:


Select "TestScaleUp" in the dropdown list of "Scale Up Action", then click on Create:


Target your Diagnostic Module and click Save:


This is our TestModule, remember to "Activate Changes" on the console:


We are ready now to test our Diagnostic Module (TestModule), we need to create a dashboard to see if the policy rule and the action are working fine:

To create a dashboard login in the "standard console":
http://localhost:7001/console (weblogic/welcome1)


Now use the url:
http://localhost:7001/console/dashboard

Click on "New View" icon:


Choose a name for your view (TestDiagnosticModule):


Click on "Metric Broser" tab:


on Servers choose MTCluster-1 (must be up and running) then click on Go button:


Select :
Types: ThreadPool
Instances: ThreadPoolRuntime
Metrics: Throughput(double)

Drag and drop the Metrics: Throughput(double) line on the right side (inside TestDiagnosticModule panel):


Repeat the previous operation with Server MTCluster-2:


You can choose how to position the window: select "New Chart 1" then TestDiagnosticModule->Move->Up:


Select now MTCluster-1 as server and:

Types: Cluster
Instances: MTCluster
Metrics: AliveServerCount(int)

Drag & drop the line on the right.

Click on the edit icon and rename the title of the 3 window.

In this way we can see the Throughput on MTCluster-1 and MTCluster-2 and the number of running server in the Dynamic Cluster MTCluster:

Our use case is:

  • shutdown MTCluster-2
  • utilize jmeter/OTD to generate traffic/throughput on MTCluster-1
  • verify that when the throughput exceeds the average throughput threshold value of 250 (our policy) we will have a Scale Up action and MTCluster-2 will be started.
  • when MTCluster-2 will be up and running we will see throughput on his dashboard window
  • we will see an increment from 1 to 2 in the MTCluster AliveServerCount Window



You need now to download a new test file for JMeter that I have created for this Test:
TestWlsDiagnosticModule.jmx

I have put the TestWlsDiagnosticModule.jmx file in the /home/weblogic/Oracle/TestMaterial directory:

Start JMeter:


Open in JMeter the TestWlsDiagnosticModule.jmx file:


The test will call localhost:8080 (OTD) and will call the InMemRepClient application that we have deployed on domain partition dp1 (previous post):


The test has a duration of 300 seconds (5 minutes):


Before starting the test we need to shutdown MTCluster-2:


Before starting the test we need to start our view in the dashboard:


We Can now start the Test i remember that our use case is:

  • MTCluster-2 is down
  • jmeter/OTD will generate traffic/throughput on MTCluster-1 for 5 minutes
  • we will verify that when the throughput exceeds the average throughput threshold value of 250 (our policy) we will have a Scale Up action and MTCluster-2 will be started.
  • when MTCluster-2 will be up and running we will see throughput on his dashboard window
  • we will see an increment from 1 to 2 in the MTCluster AliveServerCount Window
Click on the green icon on jmeter and start the test:


The test was started around 08:58:30, after around 2 minutes, the diagnostic module has started MTCluster-2 (09:00:30) I was able to see it monitoring the jvm with the "jps -l" command under the bin of the jdk installation.
This is a VM running on a simple laptop, so it took another 2 minutes (around 09:02:30) to be able to see throughput on MTCluster-2 frame and to see the increment from 1 to 2 in the MTCluster Alive Server Count frame.
The test has a duration of 5 minutes and around 09:03:30 ha finished.

The Test was successful :)

Obviously if you increase the period of the test and you change the type of Policy Alarm you will have another Scale Up action (according with the rule in Diagnostic Module), and there will be the creation and the start up of new Managed Server in the Dynamic Cluster. (MTCluster-3, MTCluster-4 .... and so on)


martedì 5 gennaio 2016

How to Create an HEAP Resource Consumption Manager in WLS 12.2.1 and How to Test it on a Domain Partition (Notify, Slow, Shutdown)

You have learned :
Now we will create an HEAP RCM (Resource Consumption Manager) on a Domain Partition and we will test the:
  • NOTIFY
  • SLOW
  • SHUTDOWN 
Remember that to use RCM we have already setted, in this POST, the required parameters:
“-XX:+UnlockCommercialFeatures -XX:+ResourceManagement -XX:+UseG1GC”
You must download the new version (released on 05:01:2016) of:
TestApp.war 
I have changed, on it, the heap.jsp file that we need for the purpose of this post.
So you need to undeploy the previous one, used in the previous postredeploy and activate the new TestApp.war.
You must have TestApp.war, deployed only on dp1.
We can proceed now with the creation of a new RCM: 
Select WebLogic Domain->Environment->Resource Consumption Managers

Lock & Edit the console:


Click on "Add resource manager":


We will create a RCM with (values on a selected Domain Partition):

  • Notify trigger at 50MB of HEAP Allocation
  • Slow Trigger at 100MB of HEAP Allocation
  • Shutdown Trigger at 150MB of HEAP Allocation

Put the values then click on OK:


This is the new RCM:


Now we must assign that RCM to a specific Domain Partition, in our case we choose dp1:

Select WebLogic Domain->Environment->Domain Partitions


Select dp1:


select Domain Partition->Administration->Resource Sharing:


Click on "Use a resource manager configured for the domain" then choose the new RCM (Heap in our case):

Click on Save:


Activate Changes:


In the following image you can see:

  • a broser to call (on MTCLuster-1 on dp1) the TestApp.war application (http://localhost:7101/dp1/TestApp/heap.html)
  • JMC console to see the Heap Allocation on MTCluster-1
  • a terminal window with a "tail -f" of the log file MTCluster-1.log


In the previous POST I have already explained how to use JMC and where find the MTCluster-1.log.

NOTIFY TRIGGER:

The Notify Trigger is setted on 50MB, so in http://localhost:7101/dp1/TestApp/heap.html we can put 80 (we need a number >50) and click submit, the application TestApp will allocate 80MB of Heap:


Il the log file (highlighted part) we have received the Notify Trigger:


SLOW (and NOTIFY) TRIGGER:

The Slow Trigger is setted on 100MB, so in http://localhost:7101/dp1/TestApp/heap.html we can put 120 (we need a number >100) and click submit, the application TestApp will allocate 120MB of Heap:


Il the log file (highlighted part) we have received the Slow Trigger:  (obviously we have received also a Notify Trigger)


SHUTDOWN TRIGGER:

The Shutdown Trigger is setted on 150MB, so in http://localhost:7101/dp1/TestApp/heap.html we can put 250 (we need a number >150) and click submit, the application TestApp will allocate 250MB of Heap:

(my test application is not the best one to allocate Memory :) so sometime GC is faster then the asyncronous evaluation for the Shutdown Trigger, and maybe that you need to resend the request of allocation of memory [sorry...])


As you can see, in the next image,we have received a FORCE SHUTDOWN:

This shutdown is only for dp1 on MTCluster-1, because we have allocated memory only on this managed server.

Remember that we have:
MTCluster-1 with port 7101 with dp1 and dp2
MTCluster-2 with port 7102 with dp1 and dp2

TestApp.war is deployed only on dp1


If you try to reload the application you will receive a 404 Error, because the dp1 on MTCluster-1 is now down, but in JMC you can see that the JVM on MTCluster-1 is alive and running:


dp1 is still live on MTCluster-2:
try to load http://localhost:7102/dp1/TestApp/heap.html


dp2 is live on MTCluster-1:
try to load http://localhost:7101/dp2/InMemRepClient/Session


If you select the Deployments page you can see that TestApp that is deployed only on dp1 is UP on MTCluster-2 and DOWN on MTCluster-1: