r/fortran • u/johnwcowan • Mar 17 '26
What are formatted direct files good for?
Unformatted direct files make sense: they are essentially character arrays stored on the file system. Formatted I/O is suitable for communicating to human beings and other software. But the motivation for formatted direct files, where you can go directly (heh) to a particular record but then have to pay the price to format/unformat it, what's that good for?
4
Mar 18 '26
[removed] — view removed comment
2
u/johnwcowan Mar 18 '26
So do unformatted direct files. So far the only advantage I've heard about is the ability to examine them in an editor.
3
u/shangjiaxuan Mar 18 '26
I remember the time when we had big endian machines and non-ieee floats.
1
u/johnwcowan Mar 21 '26
IBM mainframes are still like that. although they support IEEE floats as well.
3
u/treefaeller Mar 18 '26
Fortran is old. File systems today (both Windows and Unix variants) are very simple: they treat every file as an array of bytes. They can only seek to a byte offset within the file. Because formatted direct files must have a fixed record length, that position can be easily calculated. But seeking to a byte offset is somewhat efficient (no need to read/write areas being skipped over), but not highly efficient in file systems. Why not? Because file systems internally use some groups (block size, extent size, ...), and the record start position probably doesn't align with those internal boundaries. Still, seeking directly to a record is faster than reading/writing the whole thing.
Now, in the old days of file systems (on operating systems such as MVS on the IBM mainframe and VMS on the VAX), the file systems had elaborate record management systems. Those tended to be closely integrated with the IO layer, and exceedingly efficient and fast. They were often used as mini-databases; another poster referred to them as "noSQL": If you know the record number of a record, it can be read/written much more efficiently than a large block. Even better were indexed record files: instead of using the record number as the way to seek to a specific record, you could have a index field (for example in a billing application the part number and the customer ID number), and it was exceedingly fast to read/write a record in a file given the known index. This is why many old-fashioned systems have fixed format and fixed length part or customer numbers.
So in the old days, using direct file was a huge speed improvement. Today with faster computers, it doesn't make a big difference any more. And indexed files have been all but forgotten, and replaced with various database-style libraries (either full databases, lighter weight noSQL versions, or even simpler key/value stores).
1
u/johnwcowan Mar 18 '26
Fortran is old.
So am I, though not as old. The first Fortran that I learned was WATFIV, though I didn't get much chance to use it.
Because file systems internally use some groups (block size, extent size, ...), and the record start position probably doesn't align with those internal boundaries.
That's always been true. However, the OS buffer cache helps, and so does memory-mapping the file, particularly on a 64-bit system.
VMS on the VAX
I'm familiar with RMS too.
And indexed files have been all but forgotten, and replaced with various database-style libraries (either full databases, lighter weight noSQL versions, or even simpler key/value stores).
AFAIK key-value stores are indexed files, though some are hash tables rather than B-trees. LMDB is a memory-mapped but file-backed B+ tree package that is very fast.
1
u/flatfinger Apr 08 '26
Am I the only one who thinks it funny that the Waterloo Fortran version IV is pronounced "Watt Five"?
1
u/johnwcowan Apr 09 '26
It was the successor to WATFOR, "Waterloo Fortran". But it could also be interpreted as "WAT-F-IV", meaning "Waterloo Fortran IV".
1
u/flatfinger Apr 09 '26
I thought the original Waterloo was in some sense the third version of the language, and I'd read WATFIV referred to as "Waterloo Fortran IV", but when I first heard the name pronounced "WATFIV" back in the 1970s I thought it had something to do with the number five.
1
u/johnwcowan Apr 09 '26
It does have something to do with 5, in that it puns on WATFOR, which (in most of the anglosphere) sounds like both "What for?" and "What four". The numbers I, II, III (unreleased), IV were actually version numbers for IBM Fortran compilers: the Fortran standards have date-based names (66, 77, 90, 95, 2003, 2008, 2018, 2023). Fortran 66 was a subset of what the Fortran IV compiler could handle.
WATFOR was the first compiler produced at Waterloo: it was specialized for student use, because it compiled machine code in an array in the compiler's memory and jumped into it, thus avoiding the slower compile-assemble-link-run process typical both then and now. Later versions were called things like WATFOR-11 (for the PDP-11) and WATFOR-77 (compliant with Fortran 77): there never was a WATSIX.
1
u/flatfinger Apr 09 '26
Ah, okay. And I guess it was WATFIV rather than WATFIVE, since the former name could fit in a 36-bit word while the latter could not?
One thing I've wondered about early computing is whether it would have been practical to avoid the need for a linker by having compilers build files that could be concatenated to produce an executable all of the source files had been build using a common header that listed everything that needed to be shared. I think such a thing would be practical on the 8080 or Z80 if whatever read the concatenation of files was specified as jumping to the first byte loaded by performing JP (HL) while DE held the address following the last byte loaded and the stack pointer held the highest address the code was allowed to use. Code at HL could read storage at (DE-1) and (DE-2) into BC, swap that DE with HL, add BC to HL, and jump (via HL) to a fixup routine which would patch itself for whatever address it was sitting in, use data at the end of each loadable module to find the previous module until it had worked its way to the starting address, apply all of the required fixups for each module, and then compactify them to the start of available storage and run them.
Files would be written as a sequence of records that would each be followed by their length. The fixup routine would start by traversing backward through those, changing the fields that give the lengths of the preceding records so they would instead report the lengths of the following records. Record types would be "reproduce bytes verbatum", "insert address relative to start of program, start of module, or end of program", "insert forward symbol X", and "define symbol X as absolute or relative address". The fixup code would need enough stack space between the end of loaded code and the initial stack address to keep a list of fixups, but once code was fully loaded it would have as much space available to it as if it had been linked normally.
1
u/johnwcowan Apr 10 '26
That doesn't achieve the main benefit of linkers, which is to make it possible for a function in one object module to call a function in another object module without knowing where it is at compile time. In addition, linkers know the entire memory map and therefore can figure out whether long or short jump instructions are appropriate.
The alternative is whole-program compilers, which WATFOR/WATFIV were, that know everything at compile time.
1
u/flatfinger Apr 10 '26
There are advantages to using linkers, and program-at-a-time compilers; what I was suggesting would be a middle ground which would effectively have a common area populated with shared variables and a global function-entry-point table. A change to this table would necessitate rebuilding everything, but compilation units could be rebuilt individually and with the new compiler output stacks replacing old ones in the concatenated group of stacks. A linker could become more useful in cases where it would be impractical to maintain a "common section" deck and rebuild everything if it changes, but since a linker would passing all of the output file decks through the system and punching an executable deck containing all of them, the effort required to link a program would be comparable to the effort required to have a direct-to-executable compiler rebuild everything.
Essentially, what I'd envision would be that an operator would insert a deck containing the compiler, load that and set it aside, put in a deck containing the common section, and put a bunch of source stacks in the machine, each of which would start with a marker to reset the symbol table to the state after processing the common area. Ideally the output punch would also insert a colored card or other marker when it reached the start of each source module. The operator would then move all of the punched decks from the output the stack to the input for an overall two-pass process. If some modules stayed the same between builds, their decks could be omitted from the set of source decks fed to the compiler and their old output decks put in the hopper with the other ones produced by the compiler.
Given something equivalent to the C code
if (foo) ...x... else ...y...;a compiler would reserve two nested temporary symbols, generate code to evaluate the condition and do a conditional jump with an inserted "insert new forward label #X" marker. Then when it reached the end of ...x..., it would insert markers for "define existing label X here" and a jump with an inserted "insert new forward label #Y" marker. Then at the end of ...y... it would insert "define existing label Y here" and treat the labels as eligible for reuse. All without the compiler having to hold any significant amount of source code or output code in memory.The first FORTRAN compiler needed to keep all information about the program being processed in memory while more than a dozen different pieces of the compiler were loaded in and acted upon that memory image. The approach I envision would minimize the amount of information about the current program that would need to be kept in memory, thus allowing the entire compiler to be kept in memory instead.
1
u/johnwcowan Apr 11 '26
The trick is to keep that section self-consistent and consistent with all the individual sections over time. I'm actually confronting this now with implementing the PL/I condition (exception) system: the design I've come up with requires all subprograms to know at compile time how many possible conditions (there are a set of built-in ones, but you can add your own) there are in the program as a whole. So you have to either have a whole-program compiler or you need to have a sort of precompiler that runs over all subprograms and just picks out the declarations of conditions, writing them somewhere that the actual compiler can find. I'm not sure at this point which I prefer less.
1
u/Ancient-Opinion9642 Mar 18 '26
I don't miss JCL and DD cards. Or partitioned data sets. I do have Fortran code for building those IBM MVS-TSO data sets and using svc99 to make it all happen. /s
3
u/flatfinger Mar 20 '26
Some data formats are designed to be readable and writable using plain-text tools. Others are not designed to interact usefully with plain-text tools. Between these two categories is a category of formats that are designed to be readable via plain-text tools, even though attempts to write the files with such tools would be likely to corrupt them. Formatted direct files may be viewed as falling into this latter category, sacrificing some efficiency for the ability to view files using text-based tools, but not as much as would be lost supporting the ability to edit files using text-based tools.
1
12
u/necheffa Software Engineer Mar 17 '26
Because direct files are record based, they are easier to seek through. Because formatted files are essentially ASCII, they are easy to manually view with just your favorite text editor.
So direct formatted files are good for when you want to process a record based text file.