Sockets on OpenShift

The use of Sockets, both client and Server, have restrictions on OpenShift due to the multi-tenant architecture that provides for a high-density of applications to run on a single OpenShift Node. First, this blog will use Java as an example but the concepts hold true for the underlying C sockets. Second, a little background …

Each Cartridge instantiation is allowed to bind to a single, Node-unique loopback address provided by the Node and available as an environmental variable (e.g. OPENSHIFT_JBOSSAS_IP). This design allows each instantiation to bind to standard ports with no conflicts between instantiations running on the same Node. External communication is handled through proxies (Apache & HAProxy) running on each Node.

ServerSockets must bind to the unique loopback. SELinux will prevent an instantiation from binding to any address other than the unique loopback provided by the Node.

Client Sockets must not call an explicit bind if they are intended to connect to a remote address. If the Socket is intended to connect to the unique loopback then the Socket must bind to the unique loopback provided by the Node. If the Socket attempts to bind to another address then permission will be denied by SELinux. If a Socket bound to the unique loopback attempts to connect to a remote address the connection will fail with a no route to host.

Several clustering technologies (e.g. JGroups, Hazelcast) are designed so that each cluster node sends the other cluster nodes information on how to connect back. In OpenShift there needs to be two sets of configuration params (host/IP and port) for these types of mechanisms. The first set is the local binding that consists of the unique loopback and port. The second set is the host/IP and port that is advertised to the other cluster nodes for how to connect back. This second set it the address and port of the Node proxy.

Several popular and widely used components did not or do not provide the configuration options required to deploy on OpenShift. Several versions of JGroups and AS/EAP support the required configuration options. The required configuration options for Vert.x and Hazelcast is in process. Several Spring and Apache components do not yet support the required options.

Scala on OpenShift

This post will describe how to build a simple Scala application on OpenShift using a DIY cartridge.

First, create a DIY cartridge:

rhc app create -a scale -t diy-0.1

Edit the pre_build action hook to download and install the Scala Simple-Build-Tool (SBT):

if [[ -d sbt ]]; then
  echo “SBT installed”
  curl -o sbt.tgz
  tar zxvf sbt.tgz sbt
  rm sbt.tgz

Edit the build action hook to create and build a sample HelloWorld project:

echo ‘object Hi { def main(args: Array[String]) = println(“Hi!”) }’ > ${SBT_DIR}/hw.scala
$SBT_PATH/bin/sbt -sbt-dir $SBT_DIR -ivy $IVY_DIR -mem 256 compile

Commit and push the changes:

[root@ip-10-34-137-190 scala]# git push
Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 624 bytes, done.
Total 5 (delta 2), reused 0 (delta 0)
remote: restart_on_add=false
remote: Done
remote: restart_on_add=false
remote: Running .openshift/action_hooks/pre_build
remote:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
remote:                                  Dload  Upload   Total   Spent    Left  Speed
remote: 100 1060k  100 1060k    0     0  6599k      0 –:–:– –:–:– –:–:– 7914k
remote: sbt/
remote: sbt/jansi-license.txt
remote: sbt/bin/
remote: sbt/bin/sbt.bat
remote: sbt/bin/sbt-launch.jar
remote: sbt/bin/sbt
remote: sbt/bin/win-sbt
remote: sbt/bin/sbt-launch-lib.bash
remote: sbt/bin/classes/
remote: sbt/bin/classes/SbtJansiLaunch.class
remote: sbt/bin/jansi.jar
remote: Running .openshift/action_hooks/build
remote: /var/lib/openshift/51424742371b593a740001d9/app-root/runtime/repo/.openshift/action_hooks/build: line 10: /var/lib/openshift/51424742371b593a740001d9/app-root/data//project/hw.scala: No such file or directory
remote: Getting jna 3.2.3 …
remote: downloading …
remote:     [SUCCESSFUL ];3.2.3!jna.jar (406ms)
remote: :: retrieving :: org.scala-sbt#boot-jna
remote:     confs: [default]
remote:     1 artifacts copied, 0 already retrieved (838kB/107ms)
remote: Getting org.scala-sbt sbt 0.12.2 …
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#sbt;0.12.2!sbt.jar (97ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#main;0.12.2!main.jar (301ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#compiler-interface;0.12.2!compiler-interface-src.jar (50ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#compiler-interface;0.12.2!compiler-interface-bin.jar (50ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#precompiled-2_8_2;0.12.2!compiler-interface-bin.jar (50ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#precompiled-2_10_0;0.12.2!compiler-interface-bin.jar (143ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#actions;0.12.2!actions.jar (57ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#interface;0.12.2!interface.jar (60ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#io;0.12.2!io.jar (54ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#ivy;0.12.2!ivy.jar (78ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#launcher-interface;0.12.2!launcher-interface.jar (52ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#logging;0.12.2!logging.jar (52ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#process;0.12.2!process.jar (73ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#run;0.12.2!run.jar (66ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#command;0.12.2!command.jar (56ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#classpath;0.12.2!classpath.jar (57ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#completion;0.12.2!completion.jar (59ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#api;0.12.2!api.jar (59ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#compiler-integration;0.12.2!compiler-integration.jar (50ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#compiler-ivy-integration;0.12.2!compiler-ivy-integration.jar (50ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#task-system;0.12.2!task-system.jar (51ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#tasks;0.12.2!tasks.jar (59ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#tracking;0.12.2!tracking.jar (70ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#testing;0.12.2!testing.jar (52ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#control;0.12.2!control.jar (52ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#collections;0.12.2!collections.jar (67ms)
remote: downloading …
remote:     [SUCCESSFUL ] jline#jline;1.0!jline.jar (22ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#incremental-compiler;0.12.2!incremental-compiler.jar (65ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#compile;0.12.2!compile.jar (50ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#persist;0.12.2!persist.jar (70ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#classfile;0.12.2!classfile.jar (52ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-tools.sbinary#sbinary_2.9.0;0.4.0!sbinary_2.9.0.jar (226ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.apache.ivy#ivy;2.3.0-rc1!ivy.jar (116ms)
remote: downloading …
remote:     [SUCCESSFUL ] com.jcraft#jsch;0.1.46!jsch.jar (37ms)
remote: downloading …
remote:     [SUCCESSFUL ] commons-httpclient#commons-httpclient;3.1!commons-httpclient.jar (43ms)
remote: downloading …
remote:     [SUCCESSFUL ] commons-logging#commons-logging;1.0.4!commons-logging.jar (21ms)
remote: downloading …
remote:     [SUCCESSFUL ] commons-codec#commons-codec;1.2!commons-codec.jar (20ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#cache;0.12.2!cache.jar (58ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-sbt#test-agent;0.12.2!test-agent.jar (53ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-tools.testing#test-interface;0.5!test-interface.jar (19ms)
remote: :: retrieving :: org.scala-sbt#boot-app
remote:     confs: [default]
remote:     40 artifacts copied, 0 already retrieved (8381kB/176ms)
remote: Getting Scala 2.9.2 (for sbt)…
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-lang#scala-compiler;2.9.2!scala-compiler.jar (1808ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-lang#scala-library;2.9.2!scala-library.jar (1086ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.scala-lang#jline;2.9.2!jline.jar (27ms)
remote: downloading …
remote:     [SUCCESSFUL ] org.fusesource.jansi#jansi;1.4!jansi.jar (18ms)
remote: :: retrieving :: org.scala-sbt#boot-scala
remote:     confs: [default]
remote:     4 artifacts copied, 0 already retrieved (20090kB/162ms)
remote: [info] Loading project definition from /var/lib/openshift/51424742371b593a740001d9/app-root/data/project
remote: [info] Updating {file:/var/lib/openshift/51424742371b593a740001d9/app-root/data/project/}default-19a144…
        [info] Resolving org.scala-sbt#precompiled-2_10_0;0.12.2 …
remote: [info] Done updating.
remote: [info] Set current project to default-e6de01 (in build file:/var/lib/openshift/51424742371b593a740001d9/app-root/data/)
remote: [info] Updating {file:/var/lib/openshift/51424742371b593a740001d9/app-root/data/}default-e6de01…
remote: [info] Resolving org.scala-lang#scala-library;2.9.2 …
remote: [info] Done updating.
remote: [success] Total time: 1 s, completed Mar 15, 2013 1:27:11 PM
remote: Running .openshift/action_hooks/deploy
remote: hot_deploy_added=false
remote: Done
remote: Running .openshift/action_hooks/post_deploy
To ssh://
   527c554..8370f08  master -> master

Inter-Gear/Node Communication for OpenShift Scaled Applications

This post will describe the design of inter-Gear communication within application instances running in OpenShift. When a scaled app is created … let’s assume it’s only 2 Gears …

1) A primary Gear is created that contains an instance of the app and an instance of HAProxy.
2) A secondary Gear is created that contains only an instance of the app
3) The HAProxy instance running in the Gear handles only load-balancing between the 2 app instances. All HTTP requests to the app go to this app-level HAProxy which load-balances between the 2 instances. This HAProxy instance is configured in ~/haproxy-1.4/conf/haproxy.cfg.
4) Each app instance and the app-level HAProxy instance are bound to a unique loopback address which allows for multi-tenancy on a Node. This means the app instances are not directly routable from outside the Node.
4) There is also another instance of HAProxy – one per node. This instance is responsible for handling all of the inter-Node/Gear communication. The Node-level HAProxy instance is bound to a routable (e.g. 10.x.x.x) address and is configured in /etc/openshift/port-proxy.cfg.
5) The cartridge specifies which ports are available for inter-Gear communication within the app by defining an Endpoint in the cartridge manifest.yml file. The name of the manifest Endpoint will determine the corresponding name of the environmental variable (defined in ~/.env) by using the prefix “OPENSHIFT_${CART_NS}_”. For each port a port in the 30k range is opened. The Node-level  HAProxy is reconfigured to listen on this port and route traffic on that port to the app.

For example:
1) Create a scaled JBossAS7 app which creates the 2 Gears described above. Assume the Gears are on 2 nodes ( and Gear1 binds to (app) and (haproxy) and Gear2 binds to The app-level HAProxy binds to and is configured to load-balance HTTP traffic between and That’s all it does.
2) AS7 needs to communicate over a port (7600) for clustering. So an ephemeral port is created (37600) and each Node-level HAProxy listens on this port ( and node2:37600).
3) When the the Node1 AS7 instance communicates with the Node2 AS7 it sends the traffic to which hits the node2 Node-level HAProxy. The node2 Node-level HAProxy then routes the traffic to And vice versa for node2 to node1.

Tips for Using Jenkins on OpenShift

This post will describe tips for running Jenkins Continuous Integration on OpenShift.

Embedded Apps

OpenShift will automatically configure Jenkins for building OpenShift applications when the Jenkins client cartridge is embedded in the application. A major benefit of using Jenkins is that the build takes place in it’s own Gear. Without Jenkins, application builds occur in the same Gear where the application runs which can cause the Gear to run out of available resources (e.g. memory).

When a Jenkins-enabled application build is triggered the following steps occur:

  1. If there is a existing builder gear then proceed to step 3
  2. Create a builder gear
  3. Execute the build in the builder gear

Builder gears have a configurable lifespan (15 minute default) and will be deleted after a period of inactivity. Builder gears can be manually deleted through the Jenkins console as well.

Customizing Jobs

The build configuration such as targeted maven goals can be customized by editing the Job configuration

Manually Creating Jobs (e.g. Building projects from

You may use OpenShift-based Jenkins instances to build projects that are not OpenShift applications such as maven projects from

  1. Create a new Job
  2. Select “Build a free-style software project”. Do not select the “Maven 2/3″ option – this option creates a build that attempts to create a socket listener on localhost which is not permitted in the base OpenShift configuration.
  3. Create a Label for your Job. The Label is required and typically matches the Job name. To create a Label check “Restrict where this project can be run”
  4. Select the source (e.g. Git, URL, branch, etc.)
  5. Add a Build step – Execute shell
  6. Enter the build details (e.g. maven clean package”)
  7. Apply

NOTE: Make sure “Manage Jenkins/Configure System/# of executors” is set to 0. If not Jenkins will build on the Jenkins Gear which does not have permissions for the correct .m2 directory.


The best places to troubleshoot issues with Jenkins builds are in the logs. For build-specific problems the Job build console logs is the best place to look. For non-build specific problems such as with a failure to create a builder gear then the jenkins.log is the best place to look. The jenkins.log file is located in ~/jenkins-1.4/logs/ of the Jenkins applications and can also be seen via ‘rhc tail-files

Load-Balanced HornetQ for OpenShift JBossAS7 Applications

This post will describe how to configure both client side and server side message load-balancing for HornetQ messaging for AS7 applications running in OpenShift. Client side load-balancing will round-robin (default algorithm) messages send by the JMS client between the 2 instance/Gears. Server side load-balancing will distribute the messages between the instance/Gears where there are message consumers.

Lets assume that you have created a scaled, stable 2-instance/Gear AS7 application on OpenShift.

On each instance/Gear create the following files in the .env directory:

export GEAR_HOST=value
where value = the value of the other gear’s OPENSHIFT_GEAR_DNS

export GEAR_PORT=value
where value = the value of the other gear’s OPENSHIFT_JBOSSSAS_MESSAGING_PORT


In the messaging subsystem of standalone.xml create connectors for the 2 instances/Gears:

  <netty-connector name=”netty” socket-binding=”messaging”/>
  <in-vm-connector name=”in-vm” server-id=”0″/>
  <netty-connector name=”gear1-connector” socket-binding=”gear1-socket-binding” />
  <netty-connector name=”gear2-connector” socket-binding=”gear2-socket-binding” /> </connectors>

In the messaging subsystem of standalone.xml configure the connection factory to point to the new connectors:

<connection-factory name=”RemoteConnectionFactory”>
    <connector-ref connector-name=”gear1-connector”/>
    <connector-ref connector-name=”gear2-connector”/>
    <entry name=”RemoteConnectionFactory”/>
    <entry name=”java:jboss/exported/jms/RemoteConnectionFactory”/>

In the <socket-binding-group> element of standalone.xml add the 2 bindings used by the connectors:

<outbound-socket-binding name=”gear1-socket-binding”>
  <remote-destination host=”${env.OPENSHIFT_GEAR_DNS}” port=”${env.OPENSHIFT_JBOSSAS_MESSAGING_PORT}”/>
<outbound-socket-binding name=”gear2-socket-binding”>
  <remote-destination host=”${env.GEAR_HOST}” port=”${env.GEAR_PORT}”/>


In the messaging subsystem of standalone.xml add a new connector to the <connectors> element:

<netty-connector name=”netty-remote” socket-binding=”messaging-remote”/>

In the messaging subsystem of standalone.xml configure the cluster:

  <cluster-connection name=”netty-remote”>
    <static-connectors allow-direct-connections-only=”true”>

In the messaging subsystem of standalone.xml configure RemoteConnectionFactory to point to the new connector:

<connection-factory name=”RemoteConnectionFactory”>
    <connector-ref connector-name=”netty-remote”/>

In the <interface> element of standalone.xml add a remote interface:

<interface name=”remote”>
  <inet-address value=”${env.OPENSHIFT_GEAR_DNS}”/>

In the <socket-binding-group> element of standalone.xml add anew binding:

<socket-binding name=”messaging-remote” interface=”remote” port=”${env.OPENSHIFT_JBOSSAS_MESSAGING_PORT}”/>

Maven Tips on OpenShift

This post will describe some of the tips for using Maven with your OpenShift applications


Goals/Phases/Options: The default maven goals are “clean package -Popenshift -DskipTests“. This default set of goals will:

clean: remove all files generated by the previous build

package: Execute the build lifecycle up to creating the deployment archive(s) (e.g. jar, war, ear)

-Popenshift: Execute the “openshift” build profile as defined in pom.xml

-DskipTests: Will not execute the project tests


The default JVM heap size is set to a ratio of the available memory available in the Gear. If hot_deploy is disabled then the maven heap size will be 75% of the Gear memory limit. If hot_deploy is enabled then the maven heap size will be 25% of the Gear memory limit. The latter is due to both the application and the maven build running concurrently within the Gear’s memory space. To override these default ratios, export MAVEN_JVM_HEAP_RATIO with the custom setting in the .openshift/action_hooks/pre_build_carttype action hook. For example: export MAVEN_JVM_HEAP_RATIO=”0.50″

Custom Goals/Phases/Options

If not using Jenkins to manage the builds: In order to override the default maven goals/phases/options export MAVEN_ARGS with the custom setting in the .openshift/action_hooks/pre_build_carttype action hook. For example: export MAVEN_ARGS=”package -Popenshift”

If using Jenkins to manage the builds: In the Jenkins console, edit the Build/Execute shell section of the Job configuration.


If not using Jenkins to manage the builds: In order to override the default MAVEN_OPTS export MAVEN_OPTS with the custom setting in the .openshift/action_hooks/pre_build_carttype action hook. For example: export MAVEN_OPTS=”-Xmx=512m”

If using Jenkins to manage the builds: In the Jenkins console, edit the Build/Execute shell section of the Job configuration.

Install Local Components

To install maven components that are not available in a public or accessible repository:

1) add the component archive to the root of the application’s git repo

2) add the following to the .openshift/action_hooks/pre_build

mvn install:install-file -Dfile=../../carttype/repo/filename -DgroupId=groupId -DartifactId=artifactId -Dversion=version -Dpackaging=packaging

For example:

mvn install:install-file -Dfile=../../jbossas-7/repo/caching-1.0.jar -DgroupId=org.judcon -DartifactId=caching -Dversion=1.0 -Dpackaging=jar

3) git commit and push

Custom Repositories

One or more repositories are provided via settings.xml depending on the cartridge type. Additional repositories may be added to OpenShift Online applications via the application’s pom.xml. For OpenShift Origin, the default repositories can also be modified by editing the settings.xml file(s) within the cartridge info/configuration directory.