Reading and Writing Very Large Files

IDL on all platforms is able to read and write data from files up to 2³¹-1 bytes in length. On some platforms, it is also able to read and write data from files longer than this limit.

If IDL prints the number 64, the platform supports large files. For more information, see !VERSION.

When reading and writing to files smaller than this limit, there is no difference in behavior between the platforms that can and those that cannot handle larger files. IDL uses longword integers for file position arguments (e.g. POINT_LUN, FSTAT) and keywords, as before. However, when dealing with files that exceed this limit, IDL uses signed 64-bit integers in order to be able to properly represent the offset. Consider the following example:

;Open the file  
OPENW, 1, 'test.dat'  
  
;Initial position should be 0.  
POINT_LUN, -1, POS  
  
;Print the position and its type.  
HELP, POS  
  
;Move the file pointer past the signed 32-bit boundary.  
POINT_LUN, 1, '000000ffffffffff'x  
  
;The position is now too large to represent as a longword.  
POINT_LUN, -1, POS  
  
;Print the position and its type.  
HELP, POS  
  
CLOSE, 1

Initially, the file position is 0, which fits easily into a 32-bit integer. Once the file position exceeds the range of a signed 32-bit number, IDL automatically shifts to the 64-bit integer type.

Limitations of Large File Support

There are limitations on IDL's support for very large files that must be understood by the IDL programmer:

On any platform, the amount of data that IDL can transfer in a single operation is limited by the amount of memory it can allocate. On most platforms, IDL is a 32-bit program, and as such, can theoretically address up to 2^31-1 bytes of memory (approximately 2.3GB). On these 32-bit platforms, reading, writing, and processing data larger than this limit must be done in multiple operations. Most systems do not have 2.3 GB of memory available, and other programs running on the system also compete for the same memory, so the actual memory available is likely to be considerably smaller.

To see if your platform is 32- or 64-bit, use the following:

   PRINT, !VERSION.MEMORY_BITS

IF "32" is returned, your platform is 32-bit. If "64" is returned, your platform is 64-bit. For more information, see !VERSION.

The ability to read or write to very large files is constrained by the ability of the underlying file system to support such files. Many platforms can only support large files on certain file systems. For example, many platforms will be unable to support these operations on NFS mounted file systems because NFS version 3 and later must be in use on both client and server. Some platforms, such HP-UX, can only support such operations on special large file systems, and only if they are mounted using the appropriate mount options. Consult your system documentation to determine the limitations present on your system and the procedures for supporting very large file.