[转帖]Handling native dependencies with Apache Ivy

renshanshan

　　转一个blog，关于如何使用ivy来处理native的依赖，对于有使用JNI开发的朋友应该很有价值。
　　原文blog地址：http://www.cooljeff.co.uk/2009/08/01/handling-native-dependencies-with-apache-ivy/
　　---------------------------------
　　Being able to handle native dependencies with Ivy
has cropped up a couple of times with
no best practise solution being available. This blog entry discusses the various
proposals that have been put forward in order to provide a solution that:

• Works with Ivy Ant tasks in order to help populate the
  java.library.path
  .
  
• Works with IvyDE in order to populate the Native Library Location
  of a classpath entry.
  
• Is able to deal with different platforms.
  
• Is suitable for building enterprise repositories in order to download by
  platform.
  

　　It is important to remember that the concept of a platform constitutes a
permutation of various components not limited to Operating System and Endianess.
A good example would be the C libraries available from the NAG (Numeric Algorithms
Group)
which have distributions that take into account the compiler version
to. Occasionally you even see JARs being distributed by platform (e.g. IBM’s
Java MQ Series Client), which although is arguably a bad practise, does happen
and Ivy needs to be able to handle these edge cases. Having said this, in this
blog entry I will only take into account Windows/Linux and 32-bit/64-bit
combinations for brevity.
　　The following proposals are explored (from least to most favourite):

• Using configurations to declare native artifacts.
  
• Using the extra attribute to declare a JNI path.
  
• Using types to declare native artifacts.
  
• Using types and a platform attribute to declare native artifacts.
  

Using configurations to declare native artifacts

　　This solution would look as follows:

<!--<br />
<br />
Code highlighting produced by Actipro CodeHighlighter (freeware)<br />
http://www.CodeHighlighter.com/<br />
<br />
--><
ivy-module 
version
="2.0"
>


  
<
info 
organisation
="mit"
 module
="kerberos"
 revision
="1.7.0"
 
/>


  
<
configurations
>


    
<
conf 
name
="ia32.linux"
 description
="32-bit linux native dependencies"
/>


    
<
conf 
name
="ia64.linux"
 description
="64-bit linux native dependencies"
/>


    
<
conf 
name
="ia32.windows"
 description
="32-bit windows native dependencies"
/>


  
</
configurations
>


  
<
publications
>


    
<
artifact 
name
="kerb5_lib"
 type
="native"
 ext
="so"
 conf
="ia32.linux"
/>


    
<
artifact 
name
="kerb5_lib"
 type
="native"
 ext
="so"
 conf
="ia64.linux"
/>


    
<
artifact 
name
="kerb5_lib"
 type
="native"
 ext
="dll"
 conf
="ia32.windows"
/>


  
</
publications
>


</
ivy-module
>

　　Here we are using configurations to declare artifacts by platform. Something
depending on this module would then explicitly depend on a specific platform
through a configuration.
　　This solution has the following issues:

• Switching between platforms requires updates to Ivy files.
  
• Switching between platforms is not really possible for a transitive
  dependency.
  
• IvyDE does not support filtering by configuration (only by types).
  
• Repository structuring information is now in the dependency hierarchy.
  

　　Switching between platforms is a big issue here because we are abusing what
configurations are designed for. A configuration is a way of using a module, not
for describing what an artifact actually is. This is why you don’t see
configurations labeled: jars
, javadocs
or source
.
Whilst I have to admit that I’ve not tried implementing this solution, I cannot
see how it can work when resolving native dependencies transitively. Perhaps
some conf mapping trickery could be used.
　　The owner of a 3rd party Ivy module (i.e. one found in some repository) does
not know about the environment you are going to be working in. Hence if that
module itself has native dependencies, the owner of the module would not be able
to say which configuration to depend on when writing the ivy module, even though
it clearly has a native dependency.
　　Even for the end user who does have a little more control over the
configurations they directly pull in, you would not be able to switch between
environments without updating the configurations you depend on. Being able to
switch between environments is a common use case. Many people develop on Windows
and deploy on Linux. The Ivy module should remain identical for both
environments because in most use cases, the logical dependency stays the same.
What changes is the physical artifact.
　　Finally, many shared libraries have the same file name for both 32-bit and
64-bit distributions. This means that you are going to have to use the
configuration as a directory name to structure your repository, which is not
ideal because repository structuring information is now directly in the
dependency hierarchy that a user will use.


Using the extra attribute to declare a JNI path

　　This solution would look as follows:

<!--<br />
<br />
Code highlighting produced by Actipro CodeHighlighter (freeware)<br />
http://www.CodeHighlighter.com/<br />
<br />
--><
ivy-module 
version
="2.0"
 xmlns:extra
="http://ant.apache.org/ivy/extra"
>


  
<
info 
organisation
="mit"
 module
="kerberos"
 revision
="1.7.0"


              extra:jni
="native/lib"
 
/>


</
ivy-module
>

　　Here we are using the ability to add additional custom attributes to the info
element in order to describe where the native libraries can then be found. An Ivy
trigger
would then need to be implemented by the Ant build infrastructure to
detect module resolution in order to pick up the extra attribute and populate
the java.library.path
correctly.
　　This was proposed by Arthur Branham on a comment to his JIRA requesting Ivy
to support native library path construction [IVY-600]
. This is
something that I’ve actually seen implemented as a quick solution to get over
native library dependency issues when running integration tests as part of a
build using Ant.
　　This solution has the following issues:

• Resolving for different platforms cannot be done using the same Ivy file.
  
• Resolving for a specific platform against a remote repository is not
  possible.
  
• IvyDE does not have support for extra attributes.
  

　　In order to support multiple platform resolution, you need to allow the
attribute to have tokens that the resolver is capable of substituting. Even
then, whilst this works for a local file system, it does not help when dealing
with an external repository on the web because Ivy will need to download the
artifacts into the cache for local use. With this solution there are no native
artifacts, the repository structure is instead pushed into the ivy module file
which ideally should remain in the ivy settings file.


Using types to declare native artifacts

　　This solution would look as follows:

<!--<br />
<br />
Code highlighting produced by Actipro CodeHighlighter (freeware)<br />
http://www.CodeHighlighter.com/<br />
<br />
--><
ivy-module 
version
="2.0"
>


  
<
info 
organisation
="mit"
 module
="kerberos"
 revision
="1.7.0"
 
/>


  
<
configurations
>


    
<
conf 
name
="runtime"
 description
="Core runtime dependencies"
 
/>


  
</
configurations
>


  
<
publications
>


    
<
artifact 
name
="kerb5_lib"
 type
="ia32.linux"
 ext
="so"
 conf
="runtime"
 
/>


    
<
artifact 
name
="kerb5_lib"
 type
="ia64.linux"
 ext
="so"
 conf
="runtime"
 
/>


    
<
artifact 
name
="kerb5_lib"
 type
="ia32.windows"
 ext
="dll"
 conf
="runtime"
 
/>


  
</
publications
>


</
ivy-module
>

　　Here we are using the type to provide the platform information. To switch
between platforms, you simply need to filter by type when performing an Ivy
Retrieve
or Ivy Cache Path
to match the platform you wish to
resolve by.
　　This solution has the following clear advantages:

• The supported platforms are clearly stated in the Ivy module descriptor.
  
• Repository structuring has not leaked into the dependency information.
  
• Types fit in well with IvyDE which already uses types for javadocs and
  source.
  
• Types are already used by default to segregate artifacts in a cache.
  

　　This solution has the following issue:

• The type can no longer be used for other platform specific artifacts.
  

　　The concern I have with this solution is that we are not really using the
type for its intended usage. This can result in blocking the use of other
artifacts which are associated to the same platform but not associated to the
java.library.path
.
　　Imagine if you have jar file which is platform specific but still goes onto
the classpath. As mentioned in the introduction, distributions of the IBM Java
MQ Series Client have slightly different jars for each platform. For this use
case you would still expect the type to be jar
, since this is the type
you will filter on when populating the classpath using the cachepath
ant task. Similarly take a repository that has scripts or executables available
for download by platform, you would expect the types to be scripts
or
exe
respectively.
　　If we were to take a step back, forget what we are trying to solve and use
the type classification for its intended usage, we probably would have named the
type for native library dependencies to go on the java.libary.path
as
type=”library”
.


Using types and a platform attribute to declare native artifacts

　　This solution would look as follows:

<!--<br />
<br />
Code highlighting produced by Actipro CodeHighlighter (freeware)<br />
http://www.CodeHighlighter.com/<br />
<br />
--><
ivy-module 
version
="2.0"
>


  
<
info 
organisation
="mit"
 module
="kerberos"
 revision
="1.7.0"
 
/>


  
<
configurations
>


    
<
conf 
name
="runtime"
 description
="Core runtime dependencies"
 
/>


  
</
configurations
>


  
<
publications
>


    
<
artifact 
name
="kerb5_lib"
 type
="library"
 ext
="so"
 platform
="ia32.linux"


              conf
="runtime"
 
/>


    
<
artifact 
name
="kerb5_lib"
 type
="library"
 ext
="so"
 platform
="ia64.linux"


              conf
="runtime"
 
/>


    
<
artifact 
name
="kerb5_lib"
 type
="library"
 ext
="dll"
 platform
="ia32.windows"


              conf
="runtime"
 
/>


  
</
publications
>


</
ivy-module
>


 

　　This solution has the same benefits as using the type alone to define the
platform information, however it does not have the same disadvantage of blocking
other platform specific artifacts that are unrelated to the
java.library.path
from using the same type name. To integrate the above
solution into Ivy the following would need to be done:

• IvyDE would need to support a Native Library Type (default to
  library
  ).
  
• IvyDE would need to support a Platform.
  
• IvyDE would need to populate the Native Library Location
  of a
  classpath entry.
  
• Ivy would need to support platform as a filter option in retrieve
  and cachepath
  .
  
• Ivy would need to allow the user to set the platform easily in the Ivy
  settings.
  

　　When populating the Native Library Location
IvyDE will need to
create a directory tree to all of the artifacts and then remove all duplicate
paths.


Conclusion

　　Handling native library dependencies is a gap in Ivy that will begin to
impact enterprise users who have some native dependencies. Whilst types alone
could be used to solve the problem, this potentially blocks using Ivy to resolve
other platform specific dependencies that are not associated with the
java.libary.path
. The neatest solution would be to use
type=”library”
to group artifacts that are intended for the
java.libary.path
, along with a new attribute called platform
.
This solution will allow repositories to be structured well with platform
specific information and also allow clients to resolve dependencies easily,
regardless of the environment they happen to be currently working in.
　　

Resources

• 
  Official Apache Ivy website
  
• 
  JNI
  and Endorsed directories ivy user mail thread
  
• 
  Add ability to
  construct a native library path based on dependencies JIRA