Dynamically Loadable Modules

LINKIMAGE can be used to make IDL load your system routines in a simple and efficient manner. However, it quickly becomes inconvenient if you are adding more than a few routines. Furthermore, the limitation that the LINKIMAGE call must happen before any code that calls it is compiled makes it difficult to use and complicates the process of redistributing your routines to others. IDL offers an alternative method of packaging your system routines, called Dynamically Loadable Modules (DLMs), that address these and other problems.

The IDL_SYSFUN_DEF2 structure, which is described in Registering Routines, contains all the information required by IDL for it to be able to compile calls to a given system routine and call it:

A routine signature (Name, minimum and maximum number of arguments, if the routine accepts keywords).

A pointer to a compiled language function (usually C) that supplies the standard IDL system routine interface (argc, argv, argk) and which implements the desired operation.

IDL does not require the actual code that implements the function until the routine is called: It is able to compile other routines and statements that reference it based only on its signature.

DLMs exploit this fact to load system routines on an "as needed" basis. The routines in a DLM are not loaded by IDL unless the user calls one of them. A DLM consists of two files:

A module description file (human readable text) that IDL reads when it starts running. This file tells IDL the signature for all system routines contained in the loadable module.

A sharable library that implements the actual system routines.This library must be coded to present a specific IDL mandated interface (described below) that allows IDL to automatically load it when necessary without user intervention.

DLMs are a powerful way to extend IDL's built in system routines. This form of packaging offers many advantages:

Unlike LINKIMAGE, IDL automatically discovers DLMs when it starts up without any user intervention. This makes them easy to install — you simply copy the two files into a directory on your system where IDL will look for them.

DLM routines work exactly like standard built in routines, and are indistinguishable from them. There is no need for the user to load them (for example, using LINKIMAGE) before compiling code that references them.

As the amount of code added to IDL grows, using sharable libraries in this way prevents name collisions in unrelated compiled code from fooling the linker into linking the wrong code together. DLMs thus act as a firewall between unrelated code. For example, there are instances where unrelated routines both use a common third party library, but they require different versions of this library. A specific example is that the HDF support in IDL requires its own version of the NetCDF library. The NetCDF support uses a different incompatible version of this library with the same names. Use of DLMs allows each module to link with its own private copy of such code.

Since DLMs are separate from the IDL program, they can be built and distributed on their own schedule independent of IDL releases.

System routines packaged as DLMs are effectively indistinguishable from routines built into IDL by ITT Visual Information Solutions.

Use of sharable libraries in this manner has ample precedent in the computer industry. Most modern operating systems use loadable kernel modules to keep the kernel small while the functionality grows. The same technique is used in user programs in the form of sharable libraries, which allows unrelated programs to share code and memory space (e.g. a single copy of the C runtime library is used by all running programs on a given system).

How DLMs Work

When IDL starts, it looks in the current working directory for module definition (.dlm) files. It reads any file found and adds the routines and structure definitions thus defined to its internal routine and structure lookup tables as "stubs". In the system routine dispatch table, stubs are entries that inform IDL of the routines existence, but which lack an actual compiled function to call. They contain sufficient information for IDL to properly compile calls to the routines, but not to actually call them. Similarly, stub entries in the structure definition table allow IDL to know that the DLM supplies the structure definition, but the actual definition is not present.

After the current working directory, IDL searches !DLM_PATH for .dlm files and adds them to the table in the same manner. The default value of !DLM_PATH is the directory in the IDL distribution where the binary executables are kept. This default can be changed by defining the IDL_DLM_PATH preference (similarly to the way the IDL_PATH preference works with !PATH). This process happens once at startup, and never again. This means that IDL's knowledge of loadable modules is static and unchangeable once the session is underway. This is very different from the way !PATH works, and reflects the static nature of built in routines. The format of .dlm files is discussed in The Module Description File.

The IDL session then continues in the usual fashion until a call to a routine from a loadable module occurs. At that time, the IDL interpreter notices the fact that the routine is a stub, and loads the sharable library for the loadable module that supplies the routine. It then looks up and calls a function named IDL_Load(), which is required to exist, from the library. It's job is to replace the stubs from that module with real entries (by using IDL_SysRtnAdd()) and otherwise prepare the module for use.

Once the module is loaded, the interpreter looks up the routine that caused the load one more time. If it is still a stub then the module has failed to load properly and an error is issued. Normally, a full routine entry is found and the interpreter successfully calls the routine.

At this point the module is fully loaded, and cannot be distinguished from a compiled part of IDL. A module is only loaded once, and additional calls to any routine, or access to any structure definition, from the module are made immediately and without requiring any additional loading.

The Module Description File

The module description file is a simple text file that is read by IDL when it starts. The information in this file tells IDL everything it needs to know about the routines supplied by a loadable module. With this information, IDL can compile calls to these routines and otherwise behave as if it contains the actual routine. The loadable module itself remains unloaded until a call to one of its routines is made, or until the user forces the module to load by calling the IDL DLM_LOAD procedure.

Empty lines are allowed in .dlm files. Comments are indicated using the # character. All text from a # to the end of the line is ignored by IDL and is for the user's benefit only.

All other lines start with a keyword indicating the type of information being conveyed, possibly followed by arguments. The syntax of each line depends on the keyword. Possible lines are:

MODULE Name

Gives the name of the DLM. This should always be the first non-comment line in a .dlm file.There can only be one MODULE line.

DESCRIPTION DescriptiveText

Supplies a short one line description of the purpose of the module. This information is displayed by HELP,/DLM. This line is optional.

VERSION VersionString

Supplies a version string that can be used by the IDL user to determine which version of the module will be used. IDL does not interpret this string, it only displays it as part of the HELP,/DLM output. This line is optional.

BUILD_DATE DateString

If present, IDL will display this information as part of the output from HELP,/DLM. IDL does not parse this string to determine the date, it is simply for the users benefit. This line is optional.

SOURCE SourceString

A short one line description of the person or organization that is supplying the module. This line is optional.

CHECKSUM CheckSumValue

This directive is used by ITT Visual Information Solutions to sign the authenticity of the DLMs supplied with IDL releases. It is not required for user-written DLMs.

STRUCTURE StructureName

There should be one STRUCTURE line in the DLM file for every named structure definition supplied by the loadable module. If you refer to such a structure before the DLM is loaded, IDL uses this information to cause the DLM to load. The IDL_Init() function for the DLM will define the structure.

FUNCTION RtnName [MinArgs] [MaxArgs] [Options...]

PROCEDURE RtnName [MinArgs] [MaxArgs] [Options...]

There should be one FUNCTION or PROCEDURE line in the DLM file for every IDL routine supplied by the loadable module. These lines give IDL the information it needs to compile calls to these routines before the module is loaded.

RtnName

MinArgs

The minimum number of arguments accepted by this routine. If not supplied, 0 is assumed.

MaxArgs

The maximum number of arguments accepted by this routine. If not supplied, 0 is assumed.

Options

OBSOLETE

IDL should issue a warning message if this routine is called and !WARN.OBS_ROUTINE is set.

KEYWORDS

The IDL_Load() function

Every loadable module sharable library must export a single symbol called IDL_Load(). This function is called when IDL loads the module, and is expected to do all the work required to load real definitions for the routines supplied by the function and prepare the module for use. This always requires at least one call to IDL_SysRtnAdd(). It usually also requires a call to IDL_MessageDefineBlock() if the module defines any messages. Any other initialization needed would also go here:

This function takes no arguments. It is expected to return True (non-zero) if it was successful, and False (0) if some initialization step failed.

DLM Example

This example creates a loadable module named TESTMODULE. TESTMODULE provides 2 routines:

TESTFUN

A function that issues a message indicating that it was called, and then returns the string "TESTFUN" This function accepts between 0 and IDL_MAXPARAMS arguments, but it does not use them for anything.

TESTPRO

A procedure that issues a message indicating that it was called. This procedure accepts between 0 and IDL_MAX_ARRAY_DIM arguments, but it does not use them for anything.

The intent of this example is to show the support code required to write a DLM for a completely trivial application. This framework can be easily adapted to real modules by replacing TESTFUN and TESTPRO with other routines.

The first step is to create the module definition file for TESTMODULE, named testmodule.dlm:

The next step is to write the code for the sharable library. The contents of testmodule.c is shown in the following figure. Comments in the code explain what each step is doing.

Table 15-7: testmodule.c

Table 15-7: testmodule.c
C	1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56	#include <stdio.h> #include "idl_export.h" /* Define message codes and their corresponding printf(3) format * strings. Note that message codes start at zero and each one is * one less that the previous one. Codes must be monotonic and * contiguous. / static IDL_MSG_DEF msg_arr[] = { #define M_TM_INPRO 0 { "M_TM_INPRO", "%NThis is from a loadable module procedure." }, #define M_TM_INFUN -1 { "M_TM_INFUN", "%NThis is from a loadable module function." }, }; / The load function fills in this message block handle with the * opaque handle to the message block used for this module. The other * routines can then use it to throw errors from this block. / static IDL_MSG_BLOCK msg_block; / Implementation of the TESTPRO IDL procedure / static void testpro(int argc, IDL_VPTR argv) { IDL_MessageFromBlock(msg_block, M_TM_INPRO, IDL_MSG_RET); } /* Implementation of the TESTFUN IDL function / static IDL_VPTR testfun(int argc, IDL_VPTR argv) { IDL_MessageFromBlock(msg_block, M_TM_INFUN, IDL_MSG_RET); return IDL_StrToSTRING("TESTFUN"); } int IDL_Load(void) { /* These tables contain information on the functions and procedures * that make up the TESTMODULE DLM. The information contained in these * tables must be identical to that contained in testmodule.dlm. / static IDL_SYSFUN_DEF2 function_addr[] = { { testfun, "TESTFUN", 0, IDL_MAXPARAMS, 0, 0}, }; static IDL_SYSFUN_DEF2 procedure_addr[] = { { (IDL_SYSRTN_GENERIC) testpro, "TESTPRO", 0, IDL_MAX_ARRAY_DIM, 0, 0}, }; / Create a message block to hold our messages. Save its handle where * the other routines can access it. / if (!(msg_block = IDL_MessageDefineBlock("Testmodule", IDL_CARRAY_ELTS(msg_arr), msg_arr))) return IDL_FALSE; / Register our routine. The routines must be specified exactly the same * as in testmodule.dlm. */ return IDL_SysRtnAdd(function_addr, TRUE, IDL_CARRAY_ELTS(function_addr)) && IDL_SysRtnAdd(procedure_addr, FALSE, IDL_CARRAY_ELTS(procedure_addr)); }

Table 15-7: testmodule.c

#include <stdio.h>  
#include "idl_export.h"  
   
/* Define message codes and their corresponding printf(3) format  
 * strings. Note that message codes start at zero and each one is  
 * one less that the previous one. Codes must be monotonic and  
 * contiguous. */  
static IDL_MSG_DEF msg_arr[] = {  
#define M_TM_INPRO                       0  
  {  "M_TM_INPRO",   "%NThis is from a loadable module procedure." },  
#define M_TM_INFUN                       -1    
  {  "M_TM_INFUN",   "%NThis is from a loadable module function." },  
};  
   
/* The load function fills in this message block handle with the  
 * opaque handle to the message block used for this module. The other  
 * routines can then use it to throw errors from this block. */  
static IDL_MSG_BLOCK msg_block;  
   
/* Implementation of the TESTPRO IDL procedure */  
static void testpro(int argc, IDL_VPTR *argv)  
{ IDL_MessageFromBlock(msg_block, M_TM_INPRO, IDL_MSG_RET); }  
   
/* Implementation of the TESTFUN IDL function */  
static IDL_VPTR testfun(int argc, IDL_VPTR *argv)  
{  
  IDL_MessageFromBlock(msg_block, M_TM_INFUN, IDL_MSG_RET);  
  return IDL_StrToSTRING("TESTFUN");  
}  
   
int IDL_Load(void)  
{  
  /* These tables contain information on the functions and procedures  
   * that make up the TESTMODULE DLM. The information contained in these  
   * tables must be identical to that contained in testmodule.dlm.  
   */  
  static IDL_SYSFUN_DEF2 function_addr[] = {  
    { testfun, "TESTFUN", 0, IDL_MAXPARAMS, 0, 0},  
  };  
  static IDL_SYSFUN_DEF2 procedure_addr[] = {  
    { (IDL_SYSRTN_GENERIC) testpro, "TESTPRO", 0, IDL_MAX_ARRAY_DIM, 0, 0},  
  };  
   
  /* Create a message block to hold our messages. Save its handle where  
   * the other routines can access it. */  
  if (!(msg_block = IDL_MessageDefineBlock("Testmodule", 
                                           IDL_CARRAY_ELTS(msg_arr),  
                                           msg_arr))) return IDL_FALSE;  
   
  /* Register our routine. The routines must be specified exactly the same  
   * as in testmodule.dlm. */  
  return IDL_SysRtnAdd(function_addr, TRUE,   
                       IDL_CARRAY_ELTS(function_addr))  
    && IDL_SysRtnAdd(procedure_addr, FALSE,   
                     IDL_CARRAY_ELTS(procedure_addr));  
}

If building a DLM for Microsoft Windows, a linker definition file (testmodule.def) is also needed. All of these files, along with the commands required to build the module can be found in the dlm subdirectory of the external directory of the IDL distribution.

Once the loadable module is built, you can cause IDL to find it by doing one of the following:

Move to the directory containing the .dlm and sharable library for the module.

Define the IDL_DLM_PATH preference to include the directory.

IDL> HELP,/DLM,'testmodule'  
** TESTMODULE - Test code for loadable modules (not loaded)  
Version:1.0,Build Date:JAN 8 1998,Source:ITT Visual Information 
Solutions.  
Path: /home/user/testmodule/external/testmodule.so  
IDL> testpro  
% Loaded DLM: TESTMODULE.  
% TESTPRO: This is from a loadable module procedure.  
IDL> HELP,/DLM,'testmodule'  
** TESTMODULE - Test code for loadable modules (loaded)  
Version:1.0,Build Date:JAN 8 1998,Source:ITT Visual Information 
Soluctions.  
Path: /home/user/testmodule/external/testmodule.so  
IDL> print, testfun()  
% TESTFUN: This is from a loadable module function.  
TESTFUN

The initial HELP output shows that the module starts out unloaded. The call to TESTPRO causes the module to be loaded. As IDL loads the module, it prints an announcement of the fact (similar to the way it announces the .pro files it automatically compiles to satisfy calls to user routines). Once the module is loaded, subsequent calls to HELP show that it is present. Calls to routines from this module do not cause the module to be reloaded (as evidenced by the fact that calling TESTFUN did not cause an announcement message to be issued).