diff options
Diffstat (limited to 'Configurations/README-design.md')
| -rw-r--r-- | Configurations/README-design.md | 604 |
1 files changed, 604 insertions, 0 deletions
diff --git a/Configurations/README-design.md b/Configurations/README-design.md new file mode 100644 index 000000000000..ef21a3ae2840 --- /dev/null +++ b/Configurations/README-design.md @@ -0,0 +1,604 @@ +Design document for the unified scheme data +=========================================== + +How are things connected? +------------------------- + +The unified scheme takes all its data from the `build.info` files seen +throughout the source tree. These files hold the minimum information +needed to build end product files from diverse sources. See the +section on `build.info` files below. + +From the information in `build.info` files, `Configure` builds up an +information database as a hash table called `%unified_info`, which is +stored in configdata.pm, found at the top of the build tree (which may +or may not be the same as the source tree). + +[`Configurations/common.tmpl`](common.tmpl) uses the data from `%unified_info` to +generate the rules for building end product files as well as +intermediary files with the help of a few functions found in the +build-file templates. See the section on build-file templates further +down for more information. + +build.info files +---------------- + +As mentioned earlier, `build.info` files are meant to hold the minimum +information needed to build output files, and therefore only (with a +few possible exceptions [1]) have information about end products (such +as scripts, library files and programs) and source files (such as C +files, C header files, assembler files, etc). Intermediate files such +as object files are rarely directly referred to in `build.info` files (and +when they are, it's always with the file name extension `.o`), they are +inferred by `Configure`. By the same rule of minimalism, end product +file name extensions (such as `.so`, `.a`, `.exe`, etc) are never mentioned +in `build.info`. Their file name extensions will be inferred by the +build-file templates, adapted for the platform they are meant for (see +sections on `%unified_info` and build-file templates further down). + +The variables `PROGRAMS`, `LIBS`, `MODULES` and `SCRIPTS` are used to declare +end products. There are variants for them with `_NO_INST` as suffix +(`PROGRAM_NO_INST` etc) to specify end products that shouldn't get installed. + +The variables `SOURCE`, `DEPEND`, `INCLUDE` and `DEFINE` are indexed by a +produced file, and their values are the source used to produce that +particular produced file, extra dependencies, include directories +needed, or C macros to be defined. + +All their values in all the `build.info` throughout the source tree are +collected together and form a set of programs, libraries, modules and +scripts to be produced, source files, dependencies, etc etc etc. + +Let's have a pretend example, a very limited contraption of OpenSSL, +composed of the program `apps/openssl`, the libraries `libssl` and +`libcrypto`, an module `engines/ossltest` and their sources and +dependencies. + + # build.info + LIBS=libcrypto libssl + INCLUDE[libcrypto]=include + INCLUDE[libssl]=include + DEPEND[libssl]=libcrypto + +This is the top directory `build.info` file, and it tells us that two +libraries are to be built, the include directory `include/` shall be +used throughout when building anything that will end up in each +library, and that the library `libssl` depend on the library +`libcrypto` to function properly. + + # apps/build.info + PROGRAMS=openssl + SOURCE[openssl]=openssl.c + INCLUDE[openssl]=.. ../include + DEPEND[openssl]=../libssl + +This is the `build.info` file in `apps/`, one may notice that all file +paths mentioned are relative to the directory the `build.info` file is +located in. This one tells us that there's a program to be built +called `apps/openss` (the file name extension will depend on the +platform and is therefore not mentioned in the `build.info` file). It's +built from one source file, `apps/openssl.c`, and building it requires +the use of `.` and `include/` include directories (both are declared +from the point of view of the `apps/` directory), and that the program +depends on the library `libssl` to function properly. + + # crypto/build.info + LIBS=../libcrypto + SOURCE[../libcrypto]=aes.c evp.c cversion.c + DEPEND[cversion.o]=buildinf.h + + GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" + DEPEND[buildinf.h]=../Makefile + DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm + +This is the `build.info` file in `crypto/`, and it tells us a little more +about what's needed to produce `libcrypto`. LIBS is used again to +declare that `libcrypto` is to be produced. This declaration is +really unnecessary as it's already mentioned in the top `build.info` +file, but can make the info file easier to understand. This is to +show that duplicate information isn't an issue. + +This `build.info` file informs us that `libcrypto` is built from a few +source files, `crypto/aes.c`, `crypto/evp.c` and `crypto/cversion.c`. +It also shows us that building the object file inferred from +`crypto/cversion.c` depends on `crypto/buildinf.h`. Finally, it +also shows the possibility to declare how some files are generated +using some script, in this case a perl script, and how such scripts +can be declared to depend on other files, in this case a perl module. + +Two things are worth an extra note: + +`DEPEND[cversion.o]` mentions an object file. DEPEND indexes is the +only location where it's valid to mention them + + # ssl/build.info + LIBS=../libssl + SOURCE[../libssl]=tls.c + +This is the build.info file in `ssl/`, and it tells us that the +library `libssl` is built from the source file `ssl/tls.c`. + + # engines/build.info + MODULES=dasync + SOURCE[dasync]=e_dasync.c + DEPEND[dasync]=../libcrypto + INCLUDE[dasync]=../include + + MODULES_NO_INST=ossltest + SOURCE[ossltest]=e_ossltest.c + DEPEND[ossltest]=../libcrypto.a + INCLUDE[ossltest]=../include + +This is the `build.info` file in `engines/`, telling us that two modules +called `engines/dasync` and `engines/ossltest` shall be built, that +`dasync`'s source is `engines/e_dasync.c` and `ossltest`'s source is +`engines/e_ossltest.c` and that the include directory `include/` may +be used when building anything that will be part of these modules. +Also, both modules depend on the library `libcrypto` to function +properly. `ossltest` is explicitly linked with the static variant of +the library `libcrypto`. Finally, only `dasync` is being installed, as +`ossltest` is only for internal testing. + +When `Configure` digests these `build.info` files, the accumulated +information comes down to this: + + LIBS=libcrypto libssl + SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c + DEPEND[crypto/cversion.o]=crypto/buildinf.h + INCLUDE[libcrypto]=include + SOURCE[libssl]=ssl/tls.c + INCLUDE[libssl]=include + DEPEND[libssl]=libcrypto + + PROGRAMS=apps/openssl + SOURCE[apps/openssl]=apps/openssl.c + INCLUDE[apps/openssl]=. include + DEPEND[apps/openssl]=libssl + + MODULES=engines/dasync + SOURCE[engines/dasync]=engines/e_dasync.c + DEPEND[engines/dasync]=libcrypto + INCLUDE[engines/dasync]=include + + MODULES_NO_INST=engines/ossltest + SOURCE[engines/ossltest]=engines/e_ossltest.c + DEPEND[engines/ossltest]=libcrypto.a + INCLUDE[engines/ossltest]=include + + GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)" + DEPEND[crypto/buildinf.h]=Makefile + DEPEND[util/mkbuildinf.pl]=util/Foo.pm + +A few notes worth mentioning: + +`LIBS` may be used to declare routine libraries only. + +`PROGRAMS` may be used to declare programs only. + +`MODULES` may be used to declare modules only. + +The indexes for `SOURCE` must only be end product files, such as +libraries, programs or modules. The values of `SOURCE` variables must +only be source files (possibly generated). + +`INCLUDE` and `DEPEND` shows a relationship between different files +(usually produced files) or between files and directories, such as a +program depending on a library, or between an object file and some +extra source file. + +When `Configure` processes the `build.info` files, it will take it as +truth without question, and will therefore perform very few checks. +If the build tree is separate from the source tree, it will assume +that all built files and up in the build directory and that all source +files are to be found in the source tree, if they can be found there. +`Configure` will assume that source files that can't be found in the +source tree (such as `crypto/bildinf.h` in the example above) are +generated and will be found in the build tree. + +The `%unified_info` database +---------------------------- + +The information in all the `build.info` get digested by `Configure` and +collected into the `%unified_info` database, divided into the following +indexes: + + depends => a hash table containing 'file' => [ 'dependency' ... ] + pairs. These are directly inferred from the DEPEND + variables in build.info files. + + modules => a list of modules. These are directly inferred from + the MODULES variable in build.info files. + + generate => a hash table containing 'file' => [ 'generator' ... ] + pairs. These are directly inferred from the GENERATE + variables in build.info files. + + includes => a hash table containing 'file' => [ 'include' ... ] + pairs. These are directly inferred from the INCLUDE + variables in build.info files. + + install => a hash table containing 'type' => [ 'file' ... ] pairs. + The types are 'programs', 'libraries', 'modules' and + 'scripts', and the array of files list the files of + that type that should be installed. + + libraries => a list of libraries. These are directly inferred from + the LIBS variable in build.info files. + + programs => a list of programs. These are directly inferred from + the PROGRAMS variable in build.info files. + + scripts => a list of scripts. There are directly inferred from + the SCRIPTS variable in build.info files. + + sources => a hash table containing 'file' => [ 'sourcefile' ... ] + pairs. These are indirectly inferred from the SOURCE + variables in build.info files. Object files are + mentioned in this hash table, with source files from + SOURCE variables, and AS source files for programs and + libraries. + + shared_sources => + a hash table just like 'sources', but only as source + files (object files) for building shared libraries. + +As an example, here is how the `build.info` files example from the +section above would be digested into a `%unified_info` table: + + our %unified_info = ( + "depends" => + { + "apps/openssl" => + [ + "libssl", + ], + "crypto/buildinf.h" => + [ + "Makefile", + ], + "crypto/cversion.o" => + [ + "crypto/buildinf.h", + ], + "engines/dasync" => + [ + "libcrypto", + ], + "engines/ossltest" => + [ + "libcrypto.a", + ], + "libssl" => + [ + "libcrypto", + ], + "util/mkbuildinf.pl" => + [ + "util/Foo.pm", + ], + }, + "modules" => + [ + "engines/dasync", + "engines/ossltest", + ], + "generate" => + { + "crypto/buildinf.h" => + [ + "util/mkbuildinf.pl", + "\"\$(CC)", + "\$(CFLAGS)\"", + "\"$(PLATFORM)\"", + ], + }, + "includes" => + { + "apps/openssl" => + [ + ".", + "include", + ], + "engines/ossltest" => + [ + "include" + ], + "libcrypto" => + [ + "include", + ], + "libssl" => + [ + "include", + ], + "util/mkbuildinf.pl" => + [ + "util", + ], + } + "install" => + { + "modules" => + [ + "engines/dasync", + ], + "libraries" => + [ + "libcrypto", + "libssl", + ], + "programs" => + [ + "apps/openssl", + ], + }, + "libraries" => + [ + "libcrypto", + "libssl", + ], + "programs" => + [ + "apps/openssl", + ], + "sources" => + { + "apps/openssl" => + [ + "apps/openssl.o", + ], + "apps/openssl.o" => + [ + "apps/openssl.c", + ], + "crypto/aes.o" => + [ + "crypto/aes.c", + ], + "crypto/cversion.o" => + [ + "crypto/cversion.c", + ], + "crypto/evp.o" => + [ + "crypto/evp.c", + ], + "engines/e_dasync.o" => + [ + "engines/e_dasync.c", + ], + "engines/dasync" => + [ + "engines/e_dasync.o", + ], + "engines/e_ossltest.o" => + [ + "engines/e_ossltest.c", + ], + "engines/ossltest" => + [ + "engines/e_ossltest.o", + ], + "libcrypto" => + [ + "crypto/aes.c", + "crypto/cversion.c", + "crypto/evp.c", + ], + "libssl" => + [ + "ssl/tls.c", + ], + "ssl/tls.o" => + [ + "ssl/tls.c", + ], + }, + ); + +As can be seen, everything in `%unified_info` is fairly simple suggest +of information. Still, it tells us that to build all programs, we +must build `apps/openssl`, and to build the latter, we will need to +build all its sources (`apps/openssl.o` in this case) and all the +other things it depends on (such as `libssl`). All those dependencies +need to be built as well, using the same logic, so to build `libssl`, +we need to build `ssl/tls.o` as well as `libcrypto`, and to build the +latter... + +Build-file templates +-------------------- + +Build-file templates are essentially build-files (such as `Makefile` on +Unix) with perl code fragments mixed in. Those perl code fragment +will generate all the configuration dependent data, including all the +rules needed to build end product files and intermediary files alike. +At a minimum, there must be a perl code fragment that defines a set of +functions that are used to generates specific build-file rules, to +build static libraries from object files, to build shared libraries +from static libraries, to programs from object files and libraries, +etc. + + generatesrc - function that produces build file lines to generate + a source file from some input. + + It's called like this: + + generatesrc(src => "PATH/TO/tobegenerated", + generator => [ "generatingfile", ... ] + generator_incs => [ "INCL/PATH", ... ] + generator_deps => [ "dep1", ... ] + incs => [ "INCL/PATH", ... ], + deps => [ "dep1", ... ], + intent => one of "libs", "dso", "bin" ); + + 'src' has the name of the file to be generated. + 'generator' is the command or part of command to + generate the file, of which the first item is + expected to be the file to generate from. + generatesrc() is expected to analyse and figure out + exactly how to apply that file and how to capture + the result. 'generator_incs' and 'generator_deps' + are include directories and files that the generator + file itself depends on. 'incs' and 'deps' are + include directories and files that are used if $(CC) + is used as an intermediary step when generating the + end product (the file indicated by 'src'). 'intent' + indicates what the generated file is going to be + used for. + + src2obj - function that produces build file lines to build an + object file from source files and associated data. + + It's called like this: + + src2obj(obj => "PATH/TO/objectfile", + srcs => [ "PATH/TO/sourcefile", ... ], + deps => [ "dep1", ... ], + incs => [ "INCL/PATH", ... ] + intent => one of "lib", "dso", "bin" ); + + 'obj' has the intended object file with `.o` + extension, src2obj() is expected to change it to + something more suitable for the platform. + 'srcs' has the list of source files to build the + object file, with the first item being the source + file that directly corresponds to the object file. + 'deps' is a list of explicit dependencies. 'incs' + is a list of include file directories. Finally, + 'intent' indicates what this object file is going + to be used for. + + obj2lib - function that produces build file lines to build a + static library file ("libfoo.a" in Unix terms) from + object files. + + called like this: + + obj2lib(lib => "PATH/TO/libfile", + objs => [ "PATH/TO/objectfile", ... ]); + + 'lib' has the intended library file name *without* + extension, obj2lib is expected to add that. 'objs' + has the list of object files to build this library. + + libobj2shlib - backward compatibility function that's used the + same way as obj2shlib (described next), and was + expected to build the shared library from the + corresponding static library when that was suitable. + NOTE: building a shared library from a static + library is now DEPRECATED, as they no longer share + object files. Attempting to do this will fail. + + obj2shlib - function that produces build file lines to build a + shareable object library file ("libfoo.so" in Unix + terms) from the corresponding object files. + + called like this: + + obj2shlib(shlib => "PATH/TO/shlibfile", + lib => "PATH/TO/libfile", + objs => [ "PATH/TO/objectfile", ... ], + deps => [ "PATH/TO/otherlibfile", ... ]); + + 'lib' has the base (static) library file name + *without* extension. This is useful in case + supporting files are needed (such as import + libraries on Windows). + 'shlib' has the corresponding shared library name + *without* extension. 'deps' has the list of other + libraries (also *without* extension) this library + needs to be linked with. 'objs' has the list of + object files to build this library. + + obj2dso - function that produces build file lines to build a + dynamic shared object file from object files. + + called like this: + + obj2dso(lib => "PATH/TO/libfile", + objs => [ "PATH/TO/objectfile", ... ], + deps => [ "PATH/TO/otherlibfile", + ... ]); + + This is almost the same as obj2shlib, but the + intent is to build a shareable library that can be + loaded in runtime (a "plugin"...). + + obj2bin - function that produces build file lines to build an + executable file from object files. + + called like this: + + obj2bin(bin => "PATH/TO/binfile", + objs => [ "PATH/TO/objectfile", ... ], + deps => [ "PATH/TO/libfile", ... ]); + + 'bin' has the intended executable file name + *without* extension, obj2bin is expected to add + that. 'objs' has the list of object files to build + this library. 'deps' has the list of library files + (also *without* extension) that the programs needs + to be linked with. + + in2script - function that produces build file lines to build a + script file from some input. + + called like this: + + in2script(script => "PATH/TO/scriptfile", + sources => [ "PATH/TO/infile", ... ]); + + 'script' has the intended script file name. + 'sources' has the list of source files to build the + resulting script from. + +Along with the build-file templates is the driving template +[`Configurations/common.tmpl`](common.tmpl), which looks through all the +information in `%unified_info` and generates all the rulesets to build libraries, +programs and all intermediate files, using the rule generating +functions defined in the build-file template. + +As an example with the smaller `build.info` set we've seen as an +example, producing the rules to build `libcrypto` would result in the +following calls: + + # Note: obj2shlib will only be called if shared libraries are + # to be produced. + # Note 2: obj2shlib must convert the '.o' extension to whatever + # is suitable on the local platform. + obj2shlib(shlib => "libcrypto", + objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ], + deps => [ ]); + + obj2lib(lib => "libcrypto" + objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]); + + src2obj(obj => "crypto/aes.o" + srcs => [ "crypto/aes.c" ], + deps => [ ], + incs => [ "include" ], + intent => "lib"); + + src2obj(obj => "crypto/evp.o" + srcs => [ "crypto/evp.c" ], + deps => [ ], + incs => [ "include" ], + intent => "lib"); + + src2obj(obj => "crypto/cversion.o" + srcs => [ "crypto/cversion.c" ], + deps => [ "crypto/buildinf.h" ], + incs => [ "include" ], + intent => "lib"); + + generatesrc(src => "crypto/buildinf.h", + generator => [ "util/mkbuildinf.pl", "\"$(CC)", + "$(CFLAGS)\"", "\"$(PLATFORM)\"" ], + generator_incs => [ "util" ], + generator_deps => [ "util/Foo.pm" ], + incs => [ ], + deps => [ ], + intent => "lib"); + +The returned strings from all those calls are then concatenated +together and written to the resulting build-file. |