Apache Axis2 Evaluation

1
Apache Axis2 Evaluation

• Goal
• To evaluate Axis2 performance and usability in
  the context of its possible use in a future
  Globus Toolkit release.
• Tasks
• Implement a service testing framework based on
  WSRF WSDL specifications using Axis2.
• Implement sample services within the framework.
• Benchmark performance of sample services against
  similar services running under GT 4.0.5.

2
Test Environment

• Software used
• Sun Java JDK 1.4.2_10
• Apache Ant 1.6
• Apache Axis2 1.2
• Tomcat 5.0.28
• GT 4.0.5
• Hardware used
• Server Dell Precision 530 dual Xeon 2.2ghz 512KB
  cache 1G RDRAM, OS Linux (Debian Sarge 3.1)
• Client Dell Precision 530 single Xeon 1.5Ghz
  512KB cache 1G RDRAM, OS Windows XP SP2
• Network environment
• Switched 100Mb Ethernet

3
Test Methodology

• XMLBeans was used as the data binding layer for
  the Axis2 WSRF sample services.
• Throughput of 4 different data types was
  measured
• xsdint
• xsdstring
• xsdcomplexType in this case a GRAM Job
  Description type was used.
• xsdanyType in this case an XML document
  containing 80KB of GLUECE information was used.
  The GLUECE sample data represented a PBS cluster
  of approximately 100 nodes.

4
Test Methodology, continued

• Each operation was invoked 1000 times with no
  delay between invocations the average and
  slowest round-trip times were measured.
• Memory usage/allocation was tested by noting the
  memory footprint at startup and subsequently
  after 1000 iterations of a method invocation
  using xsdcomplexType.
• Tests were run using Tomcat 5.0.28 as the HTTP
  server for both Axis2 and GT4.0.5.
• All tests were run without security enabled.

5
Performance Details Throughput
6
Performance Details Memory Usage

• Axis2
• Memory usage at startup 60184 KB
• Memory usage after 1000 iterations 71852 KB
• Total memory growth during test 11668 KB
• GT4.0.5
• Memory usage at startup 60416 KB
• Memory usage after 1000 iterations 65524 KB
• Total memory growth during test 5108 KB

7
Performance ComparisonSummary

• Throughput performance was approximately 2-3
  times faster for Axis2 using the primitive and
  complex types tested.
• No appreciable performance difference found when
  dealing with xsdanyType.
• Memory consumption of Axis2 under the stress test
  approximately double that of GT4.0.5 and Axis1.
• Reason for the memory consumption currently
  unknown it could be XMLBeans overhead, Axis2
  engine overhead, or a possible combination of
  both.

8
Axis2 Pros

• General Architecture
• The architecture of Axis2 is designed to be
  modular and flexible with regard to customization
  on many levels. This kind of design can help to
  ensure the longevity of this incarnation of the
  Axis system.
• Performance
• Performance (throughput) of Axis2 does indeed
  seem better than Axis1 with regard to certain
  types of XML structures.
• REST support
• REST support could be an attractive feature for
  certain WSRF services, namely singleton or
  well-known service instances that do not
  require a resource key and therefore can be
  addressed using a simple URI.
• For example, REST-enabled querying of a WSRF
  resource data aggregator such as the MDS
  DefaultIndexService.

9
Axis2 Pros, continued

• Deployment model
• The J2EE-style deployment model is very
  attractive. It allows service developers to
  package and deploy service versions as
  self-contained binary archives.
• Custom handlers can also be deployed as archive
  modules which allow further customization of
  the message processing pipeline. This feature
  can be key in the implementation of new WS-
  specifications.
• The deployment model also supports hot-deployment
  for service archives that do not require a
  container restart, a widely requested feature
  from GT users.
• Hot-Update is also supported by Axis2, whereby
  an already deployed service can be updated to a
  newer version without requiring a container
  restart.

10
Axis2 Cons

• Data Binding
• Data binding options are somewhat limited, and
  there are many unknowns still remaining regarding
  the usability of code generated from WSRF
  schemas, since coverage of WSRF functions in the
  evaluation test environment is low.
• The Axis2 native Axis Data Binding is not
  robust enough at this time to handle WSRF
  schemas.
• XMLBeans has the most development and testing
  time. While it was possible to generate valid
  objects from WSRF schemas using XMLBeans, it
  required modifications to the Axis2 code
  generation libraries to make it work.
• JAX-ME is supposedly another data binding option,
  but as of the last release of Axis2 (1.3), JAX-ME
  support was labeled as experimental.

11
Axis2 Cons, continued

• Code Generation
• WSDL support in Axis2 codegen module is not as
  flexible as it should be when it comes to
  standards compliance. Specifically, the biggest
  issue is that the codegen module does not support
  multiple port types defined within a single
  service.
• This is a big problem for many of the WS-
  specifications, where multiple ports are often
  defined. The Axis1 tooling supports multiple
  ports per service, so why not Axis2? This seems
  like a big step backwards in usability.
• Note that it is possible to work around the issue
  by invoking wsdl2java multiple times on the same
  WSDL file, specifying each time the name of the
  port type to be generated, however this is a very
  clumsy and cumbersome workaround -- especially
  when dealing with large WSDL files that contain
  many port types.
• Axis developers seem indifferent to fixing this,
  though it is a legitimate concern for users. See
  http//issues.apache.org/jira/browse/AXIS2-1443
  and http//issues.apache.org/jira/browse/AXIS2-270
  for more information about this issue.

12
Axis2 Cons, continued

• Subjective Opinions
• Axis2 is big and ambitious. In some ways, it is a
  bit bleeding-edge. The architecture is so
  flexible that it essentially tries to be
  everything to everybody. That is a lot of ground
  for the Axis developers to cover. Developer
  priorities and user priorities are not always in
  sync, so there is no guarantee that the
  development focus will always be in the same
  areas that we require.
• The Axis2 code base is robust, but still highly
  in flux. There is quite a bit of development
  still going on in many areas. While this is
  generally a good thing, it seems like there is a
  bit of disparity between the various components.
  Some components are stable and production
  quality, while others are beta quality or worse.
  This mixed-bag situation could cause problems
  for a large porting effort.

13
Conclusion

• Axis2 is very promising, but still relatively
  immature as a SOAP platform. While on the
  surface it may seem like there are many good
  reasons to attempt a port of Globus to Axis2,
  looking deeper we can see that there are a large
  number of factors that still need to be
  addressed.
• Axis2 in its currently released form may be
  suitable for applications that have relatively
  simple requirements, or have total control of
  service interface design.
• However, in the case of a large, complex
  application such as Globus, which relies on
  published WSDL interfaces and standards
  compliance, it is not clear that Axis2 in its
  current state is going to provide a robust enough
  solution to warrant a major porting effort at
  this time.

14
Future Directions

• Track Axis2 development, in particular with
  regard to the code generation and data binding
  modules. As the code generation issues seem to
  be the major stumbling block, if these issues
  could be resolved adequately then the viability
  of a porting effort could be reassessed.
• Update the test environment to match the current
  release of Axis2 and re-run the performance
  tests. The current test environment was
  developed against Axis2 1.2 it may be worth the
  relatively small effort required to make it
  compatible with the current Axis2 release, 1.3.
• Test JAX-ME data binding and compare performance
  and usability to XMLBeans.