Configuring the Cluster/Load Balancer with GlassFish V2

Contents (Updated on Apr 25th 2007, send comments)


Installing and Configuring GlassFish

Install the GlassFish server by downloading it from GlassFish Download Page at: http://glassfish.java.net/public/downloadsindex.html

GlasssFish installation requires JDK 5 (1.5.0_10) and ant 1.6.5 (included).

  1. Download one of the bundles to disk
  2. Run:
    java -Xmx256m -jar filename.jar
  3. cd glassfish
  4. ant -f setup-cluster.xml

Installing and Configuring GlassFish Loadbalancer Plugin

To install and configure Load Balancer plugin to work with GlassFish, user can use GlassFish Loadbalancer Configurator. It has easy to use graphical interface which accepts user inputs, and then installs and configures loadbalancer plugin based on that. As of now beta version is available for the user. GlassFish Loadbalancer Configurator can install and configure load-balancer plugin onSun Java System Web Server, Apache and IIS. However GlassFish V2 is officially tested and supported only with Sun Java System Web Server and not supported for other webserver for the load balancer plugin.

Before you install the load balancer plugin, you must install Sun Java System Web Server 7.x (SJSWS) first. You can download the Web Server from: http://www.sun.com/software/products/web_srvr/get_it.jsp

User can also install and configure it manually.  Steps to install and configure GlassFish Loadbalancer Plugin on Sun Java System Web Server 7.x plugin are provided below :

Legends:
websrvr_install_dir - Sun Java System Web Server install Directory
websrvr_admin_user - Sun Java System Web Server admin user
websrvr_instance_config - Directory corresponding to Web Server instance config
websrvr_instance_dir - Directory corresponding to Web Server instance
as_install_dir - GlassFish installation Directory
OS - Operating System. It refers to 'linux' for the linux platform, 'solaris' for the solaris and 'windows' for windows platform.

  1. Download aslb (appserver load balancer plugin) from the link: SunOS_X86 , SunOS , Linux , Windows
  2. Unjar to install and configure in GlassFish installation (this is ${glassfish.home}same as ${S1AS_HOME) directory
  3. Create the following directories:
  4. Copy <as_install_dir/lib/lbplugin/lib/webserver-plugin/<OS>/sjsws/libpassthrough.so

    as <websrvr_install_dir>/plugins/lbplugin/bin/libpassthrough.so

    Use .dll as extension for windows.

  5. Add execute permission to <websrvr_install_dir>/plugins/lbplugin/bin/libpassthrough.so.
  6. Copy <as_install_dir>/lib/lbplugin/lib/webserver-plugin/<OS>/sjsws/errorpages/default-error.html

    as  <websrvr_install_dir>/plugins/lbplugin/errorpages/default-error.html

  7. Copy <as_install_dir>/lib/lbplugin/lib/webserver-plugin/<OS>/sjsws/errorpages/sun-http-lberror.html

    as  <websrvr_install_dir>/plugins/lbplugin/errorpages/sun-http-lberror.html

  8. Copy <as_install_dir>/lib/lbplugin/lib/webserver-plugin/<OS>/sjsws/*.res

    To  <websrvr_install_dir>/plugins/lbplugin/resource

  9. Copy <as_install_dir>/lib/lbplugin/lib/install/templates/loadbalancer.xml.example

    as  <websrvr_install_dir>/admin-server/config-store/<websrvr_instance_config>/config/loadbalancer.xml.example

  10. Copy <as_install_dir>/lib/lbplugin/lib/dtds/sun-loadbalancer_1_2.dtd

    as  <websrvr_install_dir>/admin-server/config-store/<websrvr_instance_config>/config/sun-loadbalancer_1_2.dtd

  11. Append the following lines to <websrvr_install_dir>/admin-server/config-store/<websrvr_instance_config>/config/magnus.conf.
    1.       ##BEGIN EE LB Plugin Parameters
          Init fn="load-modules" shlib="<websrvr_install_dir>/plugins/lbplugin/bin/libpassthrough.so" funcs="init-passthrough,service-passthrough,name-trans-passthrough" Thread="no"
          Init fn="init-passthrough"
            ##END EE LB Plugin Parameters
      The above lines need to be before following line.
      Init fn="load-modules" shlib=".../libj2eeplugin.so" shlib_flags="(global|now)" 
  12. Insert this line before the line of the first occurence of the string "nametrans" in <websrvr_install_dir>/admin-server/config-store/<websrvr_instance_config>/config/obj.conf
    NameTrans fn="name-trans-passthrough" name="lbplugin" config-file="loadbalancer.xml"
  13. Append the following lines to <websrvr_install_dir>/admin-server/config-store/<websrvr_instance_config>/config/obj.conf
         <Object name="lbplugin">
           ObjectType fn="force-type" type="magnus-internal/lbplugin"
            PathCheck fn="deny-existence" path="*/WEB-INF/*"
            Service type="magnus-internal/lbplugin" fn="service-passthrough"
            Error reason="Bad Gateway" fn="send-error" uri="$docroot/badgateway.html"
        </Object>
  14. Deploy the configuration to the Web Server's instance by executing the deploy-config command from Web Server's wadm CLI utility.

    <websrvr_install_dir>/bin/wadm deploy-config --user=<websrvr_admin_user> <websrvr_instance_config>

  15. Update the default Web Server instance startserv script by suffixing <as_install_dir>/lib/lbplugin/lib to LD_LIBRARY_PATH. You can also get this done by setting the environment variable $LD_LIBRARY_PATH to this value. On windows, user need to set system enviroment variable Path. This will require system restart.
  16. (Optional) For DAS based Load Balancer admininstration (auto-apply feature), configure the WebServer for SSL. Refer to inputs at http://blogs.sun.com/pjjairath/entry/ssl_setup_for_webserver_7
  17. Restart the Web Server instance by executing the start-instance command from Web Server's wadm CLI utility.

    <websrvr_install_dir>/bin/wadm stop-instance --user=<websrvr_admin_user> --config=<websrvr_instance_config>

    <websrvr_install_dir>/bin/wadm start-instance --user=<websrvr_admin_user> --config=<websrvr_instance_config>


Setting Up a One Machine Cluster

If you haven't already done so, download and install GlassFish latest promoted build.

Now to create the cluster, we need to first configure and start GlassFish V2 domain (which is in cluster mode), run following commands:

% ant -f setup-cluster.xml
% cd ${glassfish.home}/bin
% asadmin start-domain --user ${adminuser} --passwordfile ${Admin-PasswordFile} domain-name

To create a cluster on the machine running the above Domain Administartion Server (DAS), follow these steps:

  1. In a shell window, change your working directory to ${glassfish.home}/samples/quickstart/clusterjsp
  2. Run the command: ${glassfish.home}/bin/asant setup-one-machine-cluster which automates following steps.
  3. Starts the cluster ${glassfish.home}/bin/asant start_cluster
  4. Troble-shooting:
  5. You can do the above steps using Adminstration GUI by accessing https://<glassfish-host-name>:<admin-port>
  6. GlassFish Admin CLI commands to create cluster can be found HERE.


Configuring the Load Balancer Plugin

After installing the Load Balancer Plugin, configure the <websrvr_install_dir>/admin-server/config-store/<websrvr_instance_config>/config/loadbalancer.xml file as follows:
<!DOCTYPE loadbalancer PUBLIC "-//Sun Microsystems Inc.//DTD Sun One Application Server 8.1//EN"
"sun-loadbalancer_1_2.dtd">
<loadbalancer>
<cluster name="cluster1" policy="round-robin">
<instance name="instance1" enabled="true" disable-timeout-in-minutes="60"
listeners="http://instance-one-host:instance-one-port" weight="100"/>
<instance name="instance2" enabled="true" disable-timeout-in-minutes="60"
listeners="http://instance-two-host:instance-two-port weight="100"/>
<web-module context-root="clusterjsp" enabled="true" disable-timeout-in-minutes="60"
error-url="sun-http-lberror.html" />
<health-checker url="/" interval-in-seconds="10" timeout-in-seconds="30" />
</cluster>

<property name="reload-poll-interval-in-seconds" value="60"/>
<property name="response-timeout-in-seconds" value="30"/>
<property name="https-routing" value="true"/>
<property name="require-monitor-data" value="false"/>
<property name="active-healthcheck-enabled" value="false"/>
<property name="number-healthcheck-retries" value="3"/>
<property name="rewrite-location" value="true"/>
</loadbalancer>

To turn on load balancer plug-in monitor messages, set the require-monitor-data property value to true. The load balancer plug-in logs record the following information:

Follows steps 12 and 15 in load-balancer configuration to deploy the config and restart the Web Server to configure the LB plugin from loadbalancer.xml file.

Deploying and Running the clusterjsp Sample Application

To test cluster1 of two GlassFish V2 Server instances, use the "clusterjsp" application located in  ${glassfish.home}/samples/quickstart/clusterjsp directory as clusterjsp.ear

  1. You will need to use two Firefox browsers or at least two seperate sessions of the browsers, and clear cache/cookies from the browser before every new request (this is needed to simulate unique clients) to access a deployed clusterjsp sample application.
  2. You access the clusterjsp application through webserver http://web-server-host:web-server-port/clusterjsp
    and see the request is served by first instance that is running (instance-ONE in our case) in the http response you get in browser.  Healthy and the Unhealthy instance is been detected by webserver load-balancer plugin.
  3. Open another Firefox browser, clear cache/cookies from the browser (Do it couple of times to really clear cache/cookies) and access clusterjsp application again through webserver http://web-server-host:web-server-port/clusterjsp This time, you can see the request is served by second instance that is running (instance-TWO in this case) in the http response you get in browser. This ensures that requests are served by instances in the order set in the loadbalancer.xml file in Round-Robin fashion.  Screenshots below.
instance-ONE
instance-TWO

This demonstrates the cluster of two instances instance-ONE, and instance-TWO which are managed by the Node-agents and Domain Admininstration Server of GlassFish V2 and load balanced through Load Balancer plugin installed on SJSWS 7.x.

High Availability/Failover using In-memory Replication feature

To verify GlassFish server High-Availability or failover while accessing a deployed clusterjsp sample application, follow the steps below:

  1. You can make a particular instance serve a request by restarting the web server that has the load balancer plugin installed before deploying an application. This ensures that requests are served by instances in the order set in the loadbalancer.xml file. From the above loadbalancer.xml instance-ONE will serve the first request.
  2. We have already deployed a sample application that stores session data, e.g. clusterjsp web application as above. With this we should be able to see that successive requests from same client are served by the same instance that the served first request and the session data is maintained across the requests.
  3. Send few requests and note down the instance that served those requests and shutdown that particular instance. Use this command to stop the instance:
  4. Now send in the next request and verify that the new data is stored properly and that the previously added data is still there in the session.
    Session Data

This demonstrate the same shopping-cart scenario to avoid single-point of failure. If one of the server serving requests and adding items in shopping cart initially crashed, or not available anymore, then another server in the same cluster will take-over the request with all earlier session data (added items of shopping cart) and complete the request. Here the session data is stored in Memory, so this demonstrates the in-memory replication feature to get GlassFish server Highly Available.

Stop and Cleanup Cluster Settings

  1. To stop the cluster1 Cluster of two instances and cluster1-nodeagent nodeagent, run following command
  2. To remove cluster1 of two instances and node-agent, run following command

Create Cluster using GlassFish Admin CLI commands


Trouble Shooting

  1. If you see the following error message during start-cluster, you can safely ignore it, we are working on resolving this.
  2. start_cluster:
         [echo] Starting cluster cluster1
         [exec] instance-ONE is running, does not require restart
         [exec] error 0 [#|2006-08-18T11:22:58.078-0700|WARNING|sun-appserver-ee9.1|javax.jms|_ThreadID=10;_ThreadName=main;_RequestID=94652747-8288-451a-83d0-12fb326ce7d7;|[I500]: Caught JVM Exception: java.net.ConnectException: Connection refused|#]

  3. If you see webserver startup failures for loading the libraries, you may need to do the following:
  4. Edit the <websrvr_instance_dir>/startserv script to update the LD_LIBRARY_PATH value to include <as_install_dir>/lib/lbplugin/lib. This contains the various binary dependencies for the loadbalancer plugin.