Friday, November 30, 2012

Using CXF in Containers with JAX-WS handling - WebSphere/WebLogic

cxf jax-ws websphere weblogic annotation DisableIBMJAXWSEngine Ignore-Scanning-Packages Ignore-Scanning-Archives prefer-web-inf-classes prefer-application-packages prefer-application-resources

So CXF is your chosen JAX-WS framework for your application - perhaps because you want your appplication to work the same way in every servlet container - tomcat included - or because you can't avoid the need to reference the implementation rather than just the pure JAX-WS spec (access to the http-session for example). While these reasons seem valid they seem to have been considered as an afterthought in containers with built in JAX-WS handling - such as WebSphere and WebLogic.

So you follow the CXF notes and perhaps blog/mail-list posts and either are in the state where: (a) the application is not working or (b) is working but you're not sure how or why. What steps can you perform to guarantee successful deployment, and (if possible) can you understand the context for them - so you can decide if they're needed for example.

The following are useful references:
Normally for these container/class-loader issues you can get the desired behaviour by simply getting the container to load the application's libraries first (parent last). The complication in this case is the annotation processing, which seems (in my testing at least) can happen independently of the annotation processing implementation - particularly for the association of @Resource references.

To get past this, there's essentially three areas to cover:
  1. Supply JAX-WS annotation processing libraries (geronimo) that will override the container defaults - this includes all libraries that the CXF framework and annotation processing require - because nothing can be used from the container (parent)
  2. Tell the container you're handling annotations - explicitly
  3. Setup Parent-Last Class-loading - get the container to use your applications libraries before its own

Step 1: Supply JAX-WS annotation processing libraries and dependencies

There's a longer list of libraries in the CXF Notes but many of these are not specifically essential to the JAX-WS issue (the latest JAXB libraries for example will be required by CXF in a tomcat deployment). These are the libraries that appear to be need in addition to those that would have otherwise been included with the application:
  • geronimo-annotation_1.0_spec-1.1.1.jar - Annotation Processing
  • geronimo-jaxws_2.2_spec-1.1.jar - Runtime Override
  • geronimo-stax-api_1.0_spec-1.0.1.jar - Runtime Override
  • geronimo-ws-metadata_2.0_spec-1.1.3.jar - Annotation Processing
  • stax2-api-3.1.1.jar - Runtime Override
  • woodstox-core-asl-4.1.1.jar - Library Requirement
geronimo and woodstox/stax2 are the same Apache license as CXF and these libraries are supplied as part of the CXF distribution. In WebSphere you'll know it's taken effect as in deployment you'll see:
[19/10/12 8:34:46:641 EST] 00000044 AbstractInjec W   CWNEN0070W: The javax.annotation.Resource annotation class will not be recognized because it was loaded from the 
    file:/E:/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/SERVERNode01Cell/example-app.ear/lib/geronimo-annotation_1.0_spec-1.1.1.jar location rather than from a product class loader.
[19/10/12 8:34:46:645 EST] 00000044 AbstractInjec W   CWNEN0070W: The annotation class will not be recognized because it was loaded from the 
    file:/E:/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/SERVERNode01Cell/example-app.ear/lib/geronimo-jaxws_2.2_spec-1.1.jar location rather than from a product class loader.
The "Runtime Override" libraries listed above are essential as the overridden annotation processing code can not load certain classes from the parent - this may include classes from javax.xml.*. These 'parent prevention' issues are most likely going to be reported as java.lang.VerifyError. In WebSphere for example you'll encounter 'parent prevention' issues as:
Caused by: java.lang.VerifyError: JVMVRFY013 class loading constraint violated; 
    at java.lang.J9VMInternals.verifyImpl(Native Method)
    at java.lang.J9VMInternals.verify(
    at java.lang.J9VMInternals.initialize(
    at org.apache.cxf.jaxws.context.WebServiceContextResourceResolver.resolve(

Step 2: Tell the container you're handling annotations

This seems the strangest part but both WebSphere and WebLogic will continue to report errors - in particular about the @Resource annotation. For example, on WebSphere:
[19/10/12 17:11:03:261 EST] 00000053 webapp        E populateJavaNameSpace SRVE8084E: An unexpected internal server error occurred while populating the namespace. CWNEN0044E: A resource reference binding could not be found for the following resource references [org.example.hello.HelloSoapService/context], defined for the example-app component.
For example, on WebLogic:
<24/10/2012 2:17:27 PM EST> <Error> <J2EE> <BEA-160223> 
    <The resource-env-ref 'org.example.hello.HelloSoapService/context' declared in the standard descriptor or annotation has no JNDI name mapped to it. 
    The resource-env-ref must be mapped to a JNDI name using the resource-env-description element of the weblogic proprietary descriptor or corresponding annotation.>

For WebSphere - Disable and Ignore Annotation Scanning

There are ways to configure this globally - in the application server settings and configuration. I'll only note here the application-specific approach. This involves setting attributes in the META-INF/MANIFEST.MF of the war file.
  1. DisableIBMJAXWSEngine: true
  2. Ignore-Scanning-Packages: comma-separated list of packages where there are service implementations - and use of the @Resource and @WebService annotations
  3. Ignore-Scanning-Archives: comma-separated list of jar-file libraries where there are service implementations - and use of the @Resource and @WebService annotations
You don't necessarily have to do both 2 & 3 - use 2 if implementations are in (war-app)/WEB-INF/classes for example. It does seem setting the Ignore-Scanning-Archives setting can significantly speed application deployment - and is harmless if there is no annotation processing required. Hand-coding MANIFEST files is risky - I'd recommend using an ant task (and define all three attributes):
<target name="dist" description="make the war and ear">
        <!-- list of packages with WS implementations - which should be ignored by container annotation processing -->
        <property name="service.packages">
        <loadresource property="service.packages.delim">
            <propertyresource name="service.packages"/>
                <deletecharacters chars=" "/>
        <path id="webapp.archives">
            <fileset dir="./example-app/WEB-INF/lib">
                <include name="**/*.jar"/>
        <pathconvert property="webapp.archives.delim" refid="webapp.archives" pathsep="," dirsep="/">
            <map from="${basedir}/example-app/WEB-INF/lib/" to=''/>
     <!-- Define META-INF attributes -->  
        <manifest file="./example-app/META-INF/MANIFEST.MF" mode="update" flattenAttributes="true">
            <attribute name="DisableIBMJAXWSEngine" value="true"/>
            <attribute name="Ignore-Scanning-Packages" value="${service.packages.delim}"/>
            <attribute name="Ignore-Scanning-Archives" value="${webapp.archives.delim}"/>
        <copy todir="./example-app/WEB-INF/lib">
            <fileset dir="./runtime">
                <include name="**/*.*"/>
        <jar destfile="example-app.war" basedir="./example-app/"
        <ear destfile="example-app.ear" appxml="metadata/application.xml">
            <fileset dir="." includes="example-app.war"/>
            <metainf dir="metadata" includes="*.*" excludes="application.xml"/>

For WebLogic - Prefer Packages and Resources

In the (ear-app)/META-INF/weblogic-application.xml you must explicitly preference the packages supplied as part of the CXF solution and the service resources these libraries include:
<?xml version="1.0" encoding="UTF-8"?>
<weblogic-application xmlns="">
        <!-- // for logging  --> 
        <!-- // for jaxb  --> 
        <!-- // for apache commons lang/io  --> 
  <!-- // for spring/hibernate --> 
        <!-- // for jstl -->
        <!-- // for jax-ws -->
  <!-- // xml processing -->
        <!-- // geronimo (at present) has no service such declaration (glassfish and others do) - include for future reference -->

Step 3: Setup Parent-Last Class-loading

This is probably the easiest step.

For WebSphere - set flags during deployment

Note that this should be set at the ear and war (web module) level.

For WebLogic - set prefer-web-inf-classes

This doesn't seem to be a comprehensive setting based on what is required in the weblogic-application.xml file, but in the (war-app)/WEB-INF/weblogic.xml file, ensure prefer-web-inf-classes is set to true:
<?xml version="1.0" encoding="UTF-8"?>

Happy CXF servicing.

Tuesday, October 23, 2012

Stop Hibernate info level logging to Console in WebSphere

websphere hibernate logging info commons logging log4j info SystemOut.log Console stdout WAS7 WAS8.5

The application is packaged with hibernate and configured with log4j logging (to a rolling, application-specific file - hibernate set to WARN level logging) and deployed to WebSphere with parent last class loading.

Despite all this info-level messages are being output to the WebSphere system-out/console log (in WAS_HOME\profiles\[profile_name]\logs\[server_name]\SystemOut.log).

Why is the application log4j configuration (in being ignored and how can the application's hibernate logging be diverted away from the console?

Various resources on the web describe configuring WebSphere logging or hibernate logging but the bottom of this post on Hibernate and Logging specifically describes this 'leaking' of log configuration/output and how to resolve it.

Create a with the content:
... and put this file in the application's classpath (WEB-INF/classes for example).

By default, WebSphere Application Server is installed to use info level tracing (for all loggers that it knowns about). In WAS 8.5 you'll see this in the SystemOut.log:
[22/10/12 12:00:00:000 EST] 00000001 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info.

You can edit this by going to (in WAS 8.5) Application Servers > [server_name] > Troubleshooting | Change log detail levels. You can specify the logging definition in plain text. For example, turning default logging to audit-level and disabling logging for the verbose module (this is a ':' delimited list):

Alternatively, use the "Components and Groups" section of the interface to expand items of the "tree" and right-click to select custom logging levels for a given "node" (NB for log4j users, the log levels may not match log4j - eg "audit" and "warning" instead of "warn").

Also interesting to note is that without the file defined in your web-app you will see every package from your application(s) and their libraries (where classes include logging definitions) included in this "Components and Groups" interface. Therefore if you see org.hibernate.* disappear from this interface you can be certain the configuration has worked.

Wednesday, July 25, 2012

Managing Oracle JDBC Memory Usage - Executive Summary

Oracle JDBC memory leak usage java.lang.OutOfMemoryError fetchSize white paper summary 10g 11.2

So there appears to be memory issues with your system. Oracle JDBC drivers seem like the prime candidate (based on the heap dumps). The Oracle JDBC Memory Management white paper says as much:
The Oracle JDBC drivers can use large amounts of memory. This is a conscious design choice, to trade off large memory use for improved performance. For the most part and for most users this has proved to be a good choice. Some users have experienced problems with the amount of memory the JDBC drivers use.

The white paper is only a dozen or so pages, but it seems each JDBC version introduces new connection properties or changes the meaning of previously used properties. Is there an executive-summary/cheat-sheet/quick-reference that would essentially say what you can do for a given version?

There doesn't seem to be, but I'll have a go ...

First, some summary points:
  • From 9i to 10g performance of the JDBC drivers has been improved "... on average about 30% faster".
  • This was achieved by a greater use a caches. The justification being "memory is relatively cheap".
  • In large scale applications with complex (and batch) data usage, there seem to be two key caches that may cause memory usage issues - the "Implicit Statement Cache" and the "Internal Buffer Cache".
  • Tuning the caches can be done via connection properties (all of which can also be defined as system "-D" properties).
    Tuning should be done in consideration of:
    • the table design (column types and sizes)
    • the query design (columns needed / batches)
    • the fetch size. Setting this incorrectly will definitely cause OutOfMemoryErrors - as seen in JCR-2892.

Now, for the connection properties:




Applicable for Version




When true row-data buffers are cleared when a PreparedStatement is cached (in the "Implicit Statement Cache").

Yes (new)

(from onwards, buffers are put in a new "Internal Buffer Cache")


(2,147,483,647 ~ 2Gb)

Sets an upper limit on the "Internal Buffer Cache".
Look out for the change in meaning from to 11.2 - though if you use a value >30 in 11.2 it will revert to treating it as an integer. Oracle recommends: "... start with 18. If you have to set the value to less than 16, you probably need more memory."
Each connection will (may?) have its own buffer cache. So in a connection pool setup multiply the pool-size by the maxCachedBufferSize.

N/A (no "Internal Buffer Cache")

No (no way to set the size)

Set as an integer value.
102400 ~ 100Kb

Set as a log2 value.
18 = 2^18 = 262,144 ~ 256Kb



By storing the buffer cache as a TreadLocal instead of on the Connection you'll save memory if (and only if) there are less Threads than Connections. Avoid if using code that uses Connections across Threads.



Yes (new)




An initial size for the "Implicit Statement Cache". But it doesn't seem that setting it to 0 or -1 will disable it, so it can perhaps be ignored for memory management issues - it may improve performance.




Yes (new)

Friday, April 20, 2012

Running Native SQL deletes or updates via Hibernate

hibernate manual native SQL update delete bulkUpdate child table spring

Probably not the best idea but for performance, limitations in hibernate mappings or errors in the objects you find it'd be much easier to just run some manual SQL. Can you do this via hibernate?

Yes you can, hibernate call this "Native SQL". This is fairly well document, but I noticed its missing an example of doing a manual update/delete - where the result is not going to give you a list() of anything.

For Query you make via createSQLQuery(...), use executeUpdate() which returns the number of rows effected.

Here an example (using the CATS/DOGS table names from the hibernate document):
Query deleteQuery = session.createSQLQuery(
    "delete from CATS "
    + "where DOG_ID in ( "
        + "select d.ID from DOGS d "
        + "where d.STATUS = ?)");
deleteQuery.setString(0, "VICIOUS");
int updated = deleteQuery.executeUpdate();

If you're integrating with hibernate via spring, the same can be done via the template-callback mechanism (where your DAO extends
Integer deletedData = (Integer)getHibernateTemplate().execute(new HibernateCallback () {
    public Object doInHibernate(Session session) throws HibernateException, SQLException {
        // delete the data
        Query deleteQuery = session.createSQLQuery(
            "delete from CATS "
            + "where DOG_ID in ( "
                + "select d.ID from DOGS d "
                + "where d.STATUS = ?)");
        deleteQuery.setString(0, "VICIOUS");
        int updated = deleteQuery.executeUpdate();
if (log.isDebugEnabled()) {
    log.debug("rows deleted from CATS: " + deletedData);

If you're wondering if you can avoid hard coding the table references by asking hibernate for the table-name mapping, this does seem possible - Get the table name from the model in Hibernate - but this approach doesn't seem to be "public" so may disappear.

Friday, March 30, 2012

javascript object keys being sorted in some browsers

javascript object associative array map keys sorted ordered chrome safari webkit

Is it the case that some browsers - WebKit, and possibly IE9 - are sorting the keys javascript objects (aka "associative-arrays", aka "maps")?

Trying the following code (note the integer keys as strings is intentional):
    var x = new Object;

Or as a 1-line snippet you can paste in a (modern) browser address bar:
    javascript:var x = new Object;x["3"]="C";x["2"]="B";x["1"]="A";JSON.stringify(x);

You'll get the following in Firefox:

In Chrome you'll get:

Is this a bug? Can you not expect the order the keys are added to be preserved?

The short answer is no (though more appropriately but rude would be "why would you!?").

A translation in terms that a Java programmer may understand is think of the javascript object as a java.util.HashMap despite the fact it behaves a bit like a java.util.LinkedHashMap in Firefox and like a java.util.TreeMap in WebKit.

Don't code based on any expectation of the key order - which may need some thought if dealing with JSON representations of objects where there is an implicit order in the objects in "stringify-ied" form.

Some thoughts on how to maintain an order are here - How to keep an Javascript object/array ordered.