openpkg/README@4a34d7a82eab (annotated)
openpkg/README
Tue, 29 Mar 2011 20:04:34 +0200
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Tue, 29 Mar 2011 20:04:34 +0200
- changeset 334
- 4a34d7a82eab
- child 428
- f880f219c566
- permissions
- -rw-r--r--
Rework package yet again, correcting and introducing new buildconf logic:
Conditionally disable bootstrap stage comparison correctly, correct
english grammar, better find system as(1) and ld(1), indotruce detailed
optimization option messages, more completely guess cpu types, allow
profiled bootstrapping without a preinstalled GCC because many other
compilers have long since implemented 64-bit arithmetic, instruct make
to build sequentially (not in sparallel) when building a profiled
bootstrap as GCC online documents recommend, and generally improve
comment blocks.
The single most important correction in this changeset relates to the
GCC changed optimization policy since at least GCC 4.5, in which -march
is always passed and not always correctly guessed. In the case of this
package, allowing GCC to guess the architecture leads to wild build
errors at various subcomponents (zlib, libgcc, libiberty...) and
bootstrap stages. It seems quite platform specific, and the safest
approach to correcting this seems to be explicitly always specifying the
-march argument when bootstrapping GCC. Because the best choice 'native'
is not available when bootstrapping using a foreign (non GCC) compiler,
a guess is made according to rpmmacros l_platform in that case.
It is questionable as to whether these recent optimization changes
on the part of GCC or this package are compatible with each other,
or if either are complete or correct at all. At least applying these
corrections allows this package to build again in most cases test.
| michael@13 | 1 | |
| michael@13 | 2 | OpenPKG Boostrap Package Source Tree |
| michael@13 | 3 | ==================================== |
| michael@13 | 4 | |
| michael@13 | 5 | This is the source code tree for the OpenPKG bootstrap package. What |
| michael@13 | 6 | you find here is rather complex and tricky stuff, so don't be confused |
| michael@13 | 7 | if you don't understand everything immediately. |
| michael@13 | 8 | |
| michael@13 | 9 | The Files |
| michael@13 | 10 | --------- |
| michael@13 | 11 | |
| michael@13 | 12 | README .................. this file ;-) |
| michael@13 | 13 | HISTORY ................. change history tracking of this package |
| michael@13 | 14 | |
| michael@13 | 15 | openpkg.spec ............ the regular build procedure |
| michael@13 | 16 | openpkg.boot ............ the bootstrapping procedure |
| michael@13 | 17 | |
| michael@13 | 18 | gzip-*.tar .............. untouched distribution tarball of the GNU gzip tool |
| michael@13 | 19 | make-*.tar.gz ........... untouched distribution tarball of the GNU make tool |
| michael@13 | 20 | patch-*.tar.gz .......... untouched distribution tarball of the GNU patch tool |
| michael@13 | 21 | bash-*.tar.gz ........... untouched distribution tarball of the GNU bash tool |
| michael@13 | 22 | tar-*.tar.gz ............ untouched distribution tarball of the GNU tar tool |
| michael@13 | 23 | uuid-*.tar.gz ........... untouched distribution tarball of the OSSP uuid tool |
| michael@13 | 24 | curl-*.tar.gz ........... untouched distribution tarball of the cURL tool |
| michael@13 | 25 | bzip2-*.tar.gz .......... untouched distribution tarball of the BZIP2 library |
| michael@13 | 26 | zlib-*.tar.gz ........... untouched distribution tarball of the ZLIB library |
| michael@13 | 27 | beecrypt-*.tar.gz ....... untouched distribution tarball of the BeeCrypt library |
| michael@13 | 28 | rpm-*.tar.gz ............ untouched distribution tarball of the RPM tool |
| michael@13 | 29 | config-*.tar.gz ......... untouched distribution tarball of the GNU config scripts |
| michael@13 | 30 | openpkg-registry-*.tar.gz untouched distribution tarball of the OpenPKG Registry |
| michael@13 | 31 | openpkg-tools-*.tar.gz .. untouched distribution tarball of the OpenPKG Tool Chain |
| michael@13 | 32 | openssl-*.tar.gz ........ untouched distribution tarball of the OpenSSL toolkit |
| michael@13 | 33 | perl-*-mini.tar.gz ...... stripped down distribution tarball of the Perl tool |
| michael@13 | 34 | |
| michael@13 | 35 | make.patch .............. patch for GNU make |
| michael@13 | 36 | bash.patch .............. patch for GNU bash |
| michael@13 | 37 | tar.patch ............... patch for GNU tar |
| michael@13 | 38 | beecrypt.patch .......... patch for BeeCrypt |
| michael@13 | 39 | openssl.patch ........... patch for OpenSSL |
| michael@13 | 40 | perl.patch .............. patch for Perl |
| michael@13 | 41 | gzip.c .................. replacement file for GNU gzip |
| michael@13 | 42 | rpm.patch.bugfix ........ patch for RPM (bugfixing parts) |
| michael@13 | 43 | rpm.patch.feature ....... patch for RPM (new features parts) |
| michael@13 | 44 | rpm.patch.porting ....... patch for RPM (portability enhancement parts) |
| michael@13 | 45 | rpm.patch.regen ......... patch for RPM (re-generated files parts) |
| michael@13 | 46 | |
| michael@13 | 47 | rpmpopt ................. replacements/extensions for RPM's POPT configuration |
| michael@13 | 48 | rpmmacros ............... replacements/extensions for RPM's macros |
| michael@13 | 49 | rpmrc ................... replacements for RPM's run-command configuration |
| michael@13 | 50 | |
| michael@13 | 51 | root.README ............. the source for installed <prefix>/README |
| michael@13 | 52 | local.README ............ the source for installed <prefix>/local/README |
| michael@13 | 53 | dot.bashrc .............. the source for installed <prefix>/.bashrc |
| michael@13 | 54 | dot.bash_login .......... the source for installed <prefix>/.bash_login |
| michael@13 | 55 | dot.lsyncrc ............. the source for installed <prefix>/local/.lsyncrc |
| michael@13 | 56 | |
| michael@13 | 57 | openpkg.c ............... the OpenPKG frontend (set-uid wrapper) |
| michael@13 | 58 | openpkg.sh .............. the OpenPKG frontend (main script) |
| michael@13 | 59 | openpkg.pod ............. the OpenPKG frontend manual page (source) |
| michael@13 | 60 | openpkg.1 ............... the OpenPKG frontend manual page (pre-generated output) |
| michael@13 | 61 | |
| michael@13 | 62 | rc ...................... the OpenPKG run-command handling script |
| michael@13 | 63 | rc.func ................. the OpenPKG run-command function definitions |
| michael@13 | 64 | rc.conf ................. the OpenPKG run-command configuration template |
| michael@13 | 65 | rc.openpkg .............. the OpenPKG run-command script for the bootstrap package |
| michael@13 | 66 | rc.pod .................. the OpenPKG run-command script manual page (source) |
| michael@13 | 67 | rc.8 .................... the OpenPKG run-command script manual page (pre-generated output) |
| michael@13 | 68 | |
| michael@13 | 69 | release.sh .............. the OpenPKG release information utility script |
| michael@13 | 70 | release.pod ............. the OpenPKG release information utility manual page (source) |
| michael@13 | 71 | release.8 ............... the OpenPKG release information utility manual page (pre-generated output) |
| michael@13 | 72 | |
| michael@13 | 73 | uuid.sh ................. the OpenPKG UUID management utility script |
| michael@13 | 74 | uuid.pod ................ the OpenPKG UUID management utility manual page (source) |
| michael@13 | 75 | uuid.8 .................. the OpenPKG UUID management utility manual page (pre-generated output) |
| michael@13 | 76 | |
| michael@13 | 77 | rpmdb ................... the OpenPKG RPM database management utility |
| michael@13 | 78 | |
| michael@13 | 79 | rpmtool ................. the rpmtool tool (program source) |
| michael@13 | 80 | rpmtool.8 ............... the rpmtool tool (manual page) |
| michael@13 | 81 | rpmtool.pod ............. the rpmtool tool (manual page source) |
| michael@13 | 82 | shtool .................. the portable shell tool (GNU shtool) |
| michael@13 | 83 | |
| michael@13 | 84 | rpm-config.sh ........... the RPM C API helper script |
| michael@13 | 85 | rpm-config.pod .......... the RPM C API helper manual page (source) |
| michael@13 | 86 | rpm-config.8 ............ the RPM C API helper manual page (pre-generated output) |
| michael@13 | 87 | |
| michael@13 | 88 | lsync ................... the lsync tool (program source) |
| michael@13 | 89 | lsync.8 ................. the lsync tool (manual page) |
| michael@13 | 90 | lsync.pod ............... the lsync tool (manual page source) |
| michael@13 | 91 | |
| michael@13 | 92 | aux.usrgrp.sh ........... user/group name/id determination script |
| michael@13 | 93 | aux.prereq.sh ........... prerequisite checking script |
| michael@13 | 94 | aux.wrapsrc.sh .......... wrapper script for generating openpkg-V-R.src.sh |
| michael@13 | 95 | aux.wrapbin.sh .......... wrapper script for generating openpkg-V-R.P-L.sh |
| michael@13 | 96 | |
| michael@13 | 97 | pod2man.sh .............. helper script for pre-generating manual page outputs |
| michael@13 | 98 | man.sh .................. helper script for "openpkg man" command |
| michael@13 | 99 | install.sh .............. helper script for "openpkg install" command |
| michael@13 | 100 | |
| michael@13 | 101 | openpkg.org.pgp ......... the OpenPGP public key "OpenPKG <openpkg@openpkg.org>" |
| michael@13 | 102 | openpkg.com.pgp ......... the OpenPGP public key "OpenPKG GmbH <openpkg@openpkg.com>" |
| michael@13 | 103 | openpkg.net.pgp ......... the OpenPGP public key "OpenPKG Foundation e.V. <openpkg@openpkg.net>" |
| michael@13 | 104 | |
| michael@13 | 105 | The Bootstrapping Procedure |
| michael@13 | 106 | --------------------------- |
| michael@13 | 107 | |
| michael@13 | 108 | The complexity of this OpenPKG RPM package results from the fact that |
| michael@13 | 109 | we force us to treat this bootstrapping package equal to every other |
| michael@13 | 110 | regular OpenPKG RPM package. First, this implies that the packaging |
| michael@13 | 111 | tool RPM is packaged with itself as an OpenPKG RPM package (means: its |
| michael@13 | 112 | build procedure is a real RPM .spec file and it can be installed and |
| michael@13 | 113 | upgraded through a binary or source RPM). Second, RPM is installed |
| michael@13 | 114 | into the same filesystem hierarchy as all other packages. Third, RPM |
| michael@13 | 115 | manages its own files. The reason for this approach should be obvious: |
| michael@13 | 116 | 100% consistency for the whole OpenPKG software packaging facility! |
| michael@13 | 117 | |
| michael@13 | 118 | The drawback is that this package requires a very tricky bootstrapping |
| michael@13 | 119 | procedure which had cost a lot of time to figure out and establish. If |
| michael@13 | 120 | you ever wanted to know the gory details, here they are... |
| michael@13 | 121 | |
| michael@13 | 122 | The first step was that we wrote the regular openpkg.spec file for |
| michael@13 | 123 | building the bootstrap package with OpenPKG RPM under the assumption |
| michael@13 | 124 | that OpenPKG RPM is already available. This way we can provide OpenPKG |
| michael@13 | 125 | RPM as an RPM package. Just remains the problem how we actually |
| michael@13 | 126 | bootstrap in case where OpenPKG RPM is still not available, i.e., |
| michael@13 | 127 | when we reach a new platform and have to build the package from |
| michael@13 | 128 | scratch. Here the "openpkg.boot" script comes into play. It executes |
| michael@13 | 129 | the "openpkg.spec" build procedure very similar to the way the real |
| michael@13 | 130 | OpenPKG RPM would do ("openpkg rpm -bb"). That is, "openpkg.boot" |
| michael@13 | 131 | partly emulates OpenPKG RPM -- just enough that "openpkg.spec" works. |
| michael@13 | 132 | As a result, "openpkg.spec" cannot use any fancy OpenPKG RPM features |
| michael@13 | 133 | or other things before "openpkg.boot" is able to emulate it, of |
| michael@13 | 134 | course. |
| michael@13 | 135 | |
| michael@13 | 136 | After "openpkg.boot" executed the "%prep", "%build" and "%install" |
| michael@13 | 137 | scripts of "openpkg.spec", there is a fresh version of the target |
| michael@13 | 138 | filesystem hierarchy staying under a temporary "build root". The |
| michael@13 | 139 | "openpkg.boot" script then creates a very special temporary "openpkg |
| michael@13 | 140 | rpm" command which allows the installed "openpkg rpm" command inside |
| michael@13 | 141 | the "build root" to work (although it is built for the final target |
| michael@13 | 142 | filesystem path). Then the $RPM_BOOT variable is set and the package |
| michael@13 | 143 | is _again_ build via "openpkg.spec" -- but this time with the real |
| michael@13 | 144 | OpenPKG RPM. To avoid unneccessary re-compilation, the "openpkg.spec" |
| michael@13 | 145 | skips "%prep", "%build" and "%install" sections if $RPM_BOOT is |
| michael@13 | 146 | defined. So, on this second build phase, only the "%files" section is |
| michael@13 | 147 | executed, i.e., a binary OpenPKG RPM package "openpkg-V-R.P-T.rpm" |
| michael@13 | 148 | is rolled from the files in the "build root". Additionally, a source |
| michael@13 | 149 | OpenPKG RPM package "openpkg-V-R.src.rpm" is rolled for consistency |
| michael@13 | 150 | reasons. |
| michael@13 | 151 | |
| michael@13 | 152 | Finally, we override the installation in the "build root" |
| michael@13 | 153 | again by installing the now rolled binary OpenPKG RPM package |
| michael@13 | 154 | "openpkg-V-R.P-T.rpm" by using the real OpenPKG RPM. This way |
| michael@13 | 155 | we achieve that OpenPKG RPM is remembered as a real OpenPKG RPM |
| michael@13 | 156 | package in the RPM database. We just have to make sure the package |
| michael@13 | 157 | is still relocated to the "build root" while installing. For this |
| michael@13 | 158 | we could use "--prefix=$RPM_BUILD_ROOT%{l_prefix}", but this would |
| michael@13 | 159 | create an incorrect file list for the package "openpkg" in the RPM |
| michael@13 | 160 | database. Instead we use the tricky "--justdb" option for "openpkg |
| michael@13 | 161 | rpm" which means "openpkg rpm" behaves as it would install into the |
| michael@13 | 162 | real location, but does not actually install anything. But as a |
| michael@13 | 163 | side-effect, the database inside the "build root" is now correct. |
| michael@13 | 164 | |
| michael@13 | 165 | After this procedure, the "build root" contains the target filesystem |
| michael@13 | 166 | hierarchy with OpenPKG RPM installed with itself. What is now just |
| michael@13 | 167 | remaining is to roll a bootstrap package "openpkg" with this stuff for |
| michael@13 | 168 | initial installation without OpenPKG RPM. For this the "build root" |
| michael@13 | 169 | is packed into a "tarball", compressed, again wrapped into another |
| michael@13 | 170 | tarball together with the uncompression tools ("bzip2" and "tar"), |
| michael@13 | 171 | and finally wrapped into a self-extracting shell script by appending |
| michael@13 | 172 | "aux.wrapbin.sh" (padded to 64KB for easier unpacking of the attached |
| michael@13 | 173 | tarball) to its front. |
| michael@13 | 174 | |
| michael@13 | 175 | The result is the binary bootstrap script "openpkg-V-R.P-T.sh" which |
| michael@13 | 176 | can be used to install the target hierarchy from scratch without any |
| michael@13 | 177 | pre-installed OpenPKG RPM. Nevetheless, the installed target hierarchy |
| michael@13 | 178 | looks _exactly_ as it would have been installed with OpenPKG RPM. |
| michael@13 | 179 | If one later wants to upgrade this hierarchy one can just use the |
| michael@13 | 180 | generated (or a newer) "openpkg-V-R.P-T.rpm". |
| michael@13 | 181 | |
| michael@13 | 182 | To allow one to easily repeat this from-source bootstrapping procedure |
| michael@13 | 183 | on other machines, one can run "./openpkg.boot -s" which rolls |
| michael@13 | 184 | a "openpkg-V-R.src.sh" script which is a self-extracting script |
| michael@13 | 185 | containing an attached tarball of the sources of this directory. This |
| michael@13 | 186 | script contains the same contents like "openpkg-V-R.src.rpm", but |
| michael@13 | 187 | is intended for running the described bootstrapping procedure from |
| michael@13 | 188 | scratch without any OpenPKG RPM. |
| michael@13 | 189 |
