Static Linking

From Freepascal Amiga wiki
Jump to: navigation, search

This topic is about static linking against AROS sdk libraries/objects with FPC compiler.

First let's take this moment here to inform that the acquired information and results could not have been obtained and created without the really fantastic help and support provided by AROS developer Deadwood. We can't thank him enough for his patience and shared knowledge. Also a big thank you to Chain-Q for helping us through this process by implementing new calling mechanism support in FPC for AROS.

introduction

When source-code is compiled by the compiler, there is a special program being used to accomplish the task of generating a workable executable for you. That program is called a linker.


Wikipedia explains what is a linker [1]:
In computing, a linker or link editor is a computer program that takes one or more object files generated by a compiler and combines them into a single executable file, library file, or another object file.


And on the topic of static linking Wikipedia writes [2]:
Static linking is the result of the linker copying all library routines used in the program into the executable image. This may require more disk space and memory than dynamic linking, but is more portable, since it does not require the presence of the library on the system where it is run.


The latter description being what we would like to accomplish with Free Pascal and AROS. The AROS SDK comes accompanied with quite some interesting available static libraries which we would like to use. Besides the SDK, the AROS archives also contains static libraries that were ported to AROS.


In case wondering, Wikipedia writes about a static library [3]:
In computer science, a static library or statically-linked library is a set of routines, external functions and variables which are resolved in a caller at compile-time and copied into a target application by a compiler, linker, or binder, producing an object file and a stand-alone executable.


Linking and Free Pascal

Free Pascal is just like any other compiler out there in that it is able to use an external linker such as from binutils provided linker (ld, or in the specific case of AROS it calls ld indirectly using the collect-aros binutil) in order to link against (static) objects in order to incorporate the linked code into the final produced executable.

An important thing to realize before any attempt is made to link against object files, is that the compiler is to be told where exactly it is able to locate those objects(-libraries) (in case you do not have them lined up side by side to your source-code).

  • In order to link against objects you can either use directive {$OBJECTPATH} or better configure fpc.cfg to specify the path using the option -Fo
  • In order to link against object libraries you can either use directive {$LIBRARYPATH} or better configure fpc.cfg to specify the path using option -Fl

Of course on both occasions it is also valid to provide these options providing them as command-line parameter


Some keywords with regards to linking and Free Pascal

  • directive {$LINK} [4]
  • directive {$LINKLIB} [5]
  • Errors of assembling/linking stage [6]


Some (Free Pascal) caveats:

  • The Free Pascal compiler is only able to show linker results when the main program source is compiled
  • The Free Pascal compiler will only link when at least one functionality of the static libraries is actually invoked inside your code. Just linking against the object(s) does not necessarily means the linker will link in all the requested objects.
  • When exported symbols are not 'used' in the eyes of the compiler, they will get optimized away, even when optimizations are turned off. This is a very important caveat to realize and it requires some trickery in order to force the compiler to do our bidding.

There are also some additional caveats depending on which linking method is being used, for that please refer to their corresponding paragraphs which describes those caveats.

Linking

In the process of experimenting with this topic different approaches were taken in order to achieve results.

It is always a bit of a difficult spot to give a name to something that was recently accomplished as the mind is still full of impressions focused on accomplishing a single goal, thereby losing oversight of the bigger picture (so in case reader has a better name for them, please feel free to suggest). So, at first idea i describe the different approaches:

  • Method 1: controlled linking with FPC
  • Method 2: 'Abuse' AROS' default startup/linking system

Linking Method 1

For a practical example see static MikMod - method 1

Linking Method 2

For a practical example see static MikMod - method 2

  • Step 1: Link in basic required objects
For this approach at least these 3 default items should be linked to your project:
  • libarosc.a
  • libautoinit.a
  • startup.o
These 3 link objects are the bare minimum required to resolve the symbols that are needed to get your chosen library linked in correctly in order to be used inside your Pascal project.
{$LINKLIB libarosc.a}
{$LINKLIB libautoinit.a}
{$LINK startup.o}
While libarosc takes care of resolving the clib functions, libautoinit supplies initialization routines (also symbols needed by libarosc) while startup.o takes care of providing the actual startup code and resolves the last bits of symbols needed by libarosc and libautoinit.
On the Pascal side of things we would need to implement functionality that initializes things as AROS would do. More on this later.
  • Step 2: Link in library of choice
If the above 3 objects are linked in successfully then we are left with the most important part namely linking in the static library of your choice.
This is done in a similar way:
{$LINKLIB NameOfYourPreferredLibray.a}
... where you replace the part "NameOfYourPreferredLibray.a" with the filename of the library that you actually wish to link to.
When done so successfully, you are able to use the functionality from this library to be used from Pascal side.
Note that sometimes a static library needs other objects to be able to resolve all missing symbols, in which case additional objects would need to be linked in as well. Which objects that are exactly depends completely on the library that is linked against and would require some proper research.
In case you are familiar with c-coding on AROS you are probably already aware which objects are used by the different available static libraries.
  • Step 3: Declaring Pascal headers for the linked library
Although said in the previous step that you could now use the functionality from the static library, we can only make such thing happen when all types, constants and functions have been declared in a so called Pascal wrapper.
That is also reason it is easier to link against the library of your choice in an existing header file that is already available for your library. In worse case scenario such header file does not exist and in which case you would have to convert from original sdk of that library.
It is a bit out of scope to make here a full example, therefor please see here for an example for libmikmod.
Be sure to at least read the chapter "Linking Issues" of the Free Pascal manual. There you are able to read about keywords/modifiers such as "external", "cvar" and "name" which more or less are used to import and export variables and procedures/functions.
  • Step 4: Mimic startup code
In order to get the initialization process used by AROS working for use with Free Pascal we are obligated to mimic some of that behaviour, otherwise we would be linking but none of the linked in objects would then be properly initialized (and so things would crash on usage).
The code we need to mimic originates from the file startup.c (it is indeed the original source-file that created the startup.o object that we linked against).
The two functions that we need to call are:
  • __startup_entries_init;
  • __startup_entries_next;
But, in order to be able to do so we also need some additional code:
  • Export and intialize SysBase variable
  • Export and intialize DOSBase variable
  • import function __startup_entries_init()
  • import function ___startup_entries_next()
  • Create a wrapper function __startup_entries_next() that invokes ___startup_entries_next()
If that's implemented then we are almost there, albeit some additional tuning for Free Pascal is required (more on that later).
Let's start with implementing the basic things first (and remember that exported variables mingle with c-code and that c-code is case sensitive), and let's use the main program file for this case in order to do so
var
  // Import variables SysBase and DOSBase
  SysBase         : Pointer; cvar; export;
  DOSBase         : Pointer; cvar; export;

  // Import the two startup entries functions
  procedure __startup_entries_init; cdecl; external;
  procedure ___startup_entries_next(SystemBase: Pointer); cdecl; external;

// startup_entries_next() wrapper function
procedure __startup_entries_next;
begin
  ___startup_entries_next(SysBase);
end;


// AROSC Initialization routine
procedure AROSC_Init;
begin
  // Intialialize SysBase variable
  SysBase := AOS_ExecBase;

  // Initialize DOSBase variable
  DOSBase := AOS_DosBase;

  // Call startup_entries_init()
  __startup_entries_init;

  // Call startup_entries_next()
  __startup_entries_next;
end;

begin
  WriteLn('Hello');
  AROSC_Init;
  WriteLn('Goodbye');
end.
Under no circumstances run this code yet, or at least, not with calling the function __startup_entries_next() yet (you can comment it out to see the results sofar if you like).
Now, proceed to the following step
  • Step 5: Let's say hello to Pascal
In case you didn't listen and did call __startup_entries_next from the previous step, then you might have seen that there is some looping going on, and for sure no additional Pascal code is fired (except for the looping parts). That is because the startup code we call from c does not have a clue yet about which code to call after it's done initializing. So let's take care of that here.
There is a 'magic' variable that needs to be exported and which contains the address of a startup function. A c-startup/entry function to be exact.
Because this function requires to explicit have a valid format we first create a type for this function.
Type
  TAROS_Startup_Function = function(argc: integer; argv: PPChar): integer; cdecl;
In case you are familiar with c programming then this function declaration is not real magic, but simply how a main entry code point function is declared.
ArgC is a value that corresponds to the number of parameters supplied, and the argv parameter is a list of pointers to 0-terminated strings that contains the individual parameter strings.
The last bit of information that might be important to realize is the return value. This values is normally returned back to shell and when not zero indicates an error.
Then we 'export' a variable that hold a pointer to such a function:
var
  __main_function_ptr : TAROS_Startup_Function = nil; cvar; export;
Note how we initialize the variable at compile time with the value nil. Remember that way above was told that not explicitly using a declaration means it is not actually exported (at least that is true for some symbols) ? Initializing this variable makes sure its symbol get's exported.
Now, you can run things with the call to function __startup_entries_next() enabled but, for sure that will gives us an access violation as the startup code now tries to jump to address zero. So we never will forget to set this address properly ;-)
So, let's fix this by creating a new function and have the exported variable point to that:
function AROSC_Startup_Entry(argc: integer; argv: PPChar): integer; cdecl;
begin
  DebugLn('ENTER - AROSC_Startup_Entry()');

  result := 0;

  DebugLn('LEAVE - AROSC_Startup_Entry()');
end;
We've added some debug feedback there so that we are able to check if the function is actually called ;-)
and:
// AROSC Initialization routine
procedure AROSC_Init;
begin
  // Intialialize SysBase variable
  SysBase := AOS_ExecBase;

  // Initialize DOSBase variable
  DOSBase := AOS_DosBase;

  // Initialize pointer to startup routine
  __main_function_ptr := @AROSC_Startup_Entry;

  // Call startup_entries_init()
  __startup_entries_init;

  // Call startup_entries_next()
  __startup_entries_next;
end;
Unfortunately, running this code shows there is still an issue somewhere. The code crashes on program exit. But please test and see that whatever you place inside (or call from inside) function AROSC_Startup_Entry is being executed as intended.
  • Step 6: Fine-tuning Options
If you've taken a look at the original startup.c source-code you might have noticed some odd symbols being declared. They are even described in the source-code itself.
Well, as it is the case, the c startup code is still performing some task that a) Pascal does not need and b) would require us to do additional initialization.
In order to 'disable' some of that task that the startup code is performing we need to export some symbols to turn this off.
Add the following lines to your code:
var
  __nocommandline   : Integer; cvar; export;
  __nostdiowin      : integer; cvar; export;
  __nowbsupport     : Integer; cvar; export;
For the exact explanation and workings of these symbols please see the file startup.c
Can we now then finally execute our code without issues ? Well, try and see for yourself ;-)
Just note though. we are not 100% home safe yet (albeit we are almost) and, some things can be improved. Head on to the next step please.
  • Step 7: Loose Ends
First let's start with the bad news. As soon as you halt your Pascal code somewhere, things can go from bad to worse in a very rapid pace.
Normally inside your pascal code you would allocate memory, initialize objects/classes and more of suchs things. Alas, the moment you halt your program all allocated memory is not freed up in a neatly manner. This is because halt bypasses the normal exit flow. Whatever you do inside your pascal code, you would need to take care of having a graceful exit flow that takes care of things as they are suppose to be.
So in case you think you _must_ make a call to halt then make sure you have added an proper exit procedure using AddExitProc() before actually making a call to the halt function.
In a similar way, using code from the linked c-library simply can't happen when things are not initialized yet, and for sure can't happen when we've already went through the normal exit flow. This means that using c link libraries from unit initialization and/or unit finalization is not possible. If you would do so, the code would simply crash.
ToDo: more to write about caveats and further optimizing things. check-in sources for individual steps as well as mikmod library example using described method.