[lfs-dev] About packaging LFS - How it works for me 8)

Marius Tolzmann tolzmann at molgen.mpg.de
Tue May 22 17:00:52 PDT 2012


Hi..

I was pointed to the "Once More: Package Management" discussion 
yesterday. So I just joined this list.

First I want to thank you for LFS. It's great. I love it. And i made a 
few more people love it, too. The same is true for BLFS.

I will start with many words about my LFS background 8)

I am using (B)LFS (as a reference) since (i think) 2004. After we had 
some serious issues adjusting a red hat linux to our needs I started a 
linux from scratch to finally have a linux that follow my rules 8)

I really learned a lot from this documentation - even if it wasn't very 
easy all the time.

When using an LFS based distribution in a production environment you 
have to do some kind of minimalistig package management. what we do and 
what we have done isn't really package management it is more about 
package tracking - to answer the important questions like which file 
belongs to which package? just to be able to install and remove a binary 
package after creating it following the (B)LFS Book(s).

Two years ago we were forced to rebuild the whole system - because it 
seemed to be easier to rebuild than to upgrade and solve the dependency 
hell.
The way we maintained our LFS based system between 2004 and 2012 worked 
wuite well, but was not really perfect: To document how a package was 
build we just scripted the steps to build it and executed the script, 
did some binary package creation and just kept the directory of the 
build as documentation. Our documentation was quite bad these days.
So when starting to rebuild our LFS based system I decided to do it 
better this time. I tried to find some package management software which 
was very easy to use, is highly configurable and helps me to document 
the build - just to make updating and rebuilding packages easier for us 
- Long Story short: I havn't found a package manager.

But lazy as I am, I started scripting some code together to help us to 
maintain the system. The goal was easy: 90% of the software is configure 
&& make && make install. some special options here - some patches there 
- some post install issues in some packages.
All those special options don't change much from one version of a 
package to another. So Reusing build-scripts was a main goal.
Since we are LFS based we want to stay as close to (B)LFS as possible.
So just cut & paste the LFS recipies would be great for us. After taking 
some time to think about our needs we started to implement something. 
The idea was to first have a solution for the easiest case:
configure && make && make install - no options, perfect software with 
perfect defaults - so we implemented something for the perfect world. 
And step by step we were forced to face reality and add more flexibility.

After Two years we have managed to have a working LFS based system - 
build with our own package system - which is used in production on many 
big servers and many workstations now.

So much for the background and our ideas. We just released our first 
official version of our tool to manage packages in our LFS based 
environment. I will not call this tool a package manager because it 
lacks some important features to call it so but i think it may be 
interesting for some of you to know it exists. It is not perfect - even 
it tries to be perfect so hardly - and it may have some (minor) bugs but 
it works (at least for us).

The name of our tool is BEE - an acronym for "bau et einfach" (which is 
german for "just build it" but it may also be an acronym for "build 
extremely easy".

What BEE can do:

In it's easiest form it can build a binary package directly from an URL 
pointing to a source tarball. And that is already the main feature. The 
user just has to know the URL. We tried, but we haven't found an easier 
way to create a binary package out of source tarballs.

bee binary packages are tarballs.. no fancy format..

Yes, this works for autoconf based sources and for cmake based sources 
and for perl modules and ... it just works.. it autodetects a lot of stuff.

And yes, you still have the possibility to change every single step in 
the build process. All BEE does is to create an initial (so called) 
bee-file which provides a basic structure of a generic build process.
(In fact it creates a script (bash) which has a lot of comments and just 
sets one variable: the SRCURL).

You can just add some shell functions to overwrite/expand the 
autodetected defaults with your own idea of how to configure the 
package, build the package, patch it or whatever.

Short: you can influence nearly every step bee does.

So it is easy to create bee-files for LFS packages that are in my 
opinion quiet close to what is described in the LFS book.

We already started a small proof-of-concept project where we provide 
some bee-files for the LFS chapter05 toolchain (the other chapters will 
follow when we find some time)

Why i don't want to call it package management:
You can install the binary packages you can update them or remove
them from your installation. Which is in fact some kind of management.

We don't provide support for remote repositories where you can get 
binary packages from. It is more intended to work as follows: You create 
a bee-file, you build your package, and you install it.
Local build, local package repository (some directory), local installation.

We don't provide config-file management (yet). We have some (cool) ideas 
how to solve this issue without loosing our main feature:
URL -> PKG.

But we don't provide any automated dependency resolution. And we don't 
plan to do this in the future.

But we plan to provide some information which can be automatically 
generated at buildtime about what packages the current package was built 
against. How the new package may break already installed packages.

We just don't have a solution how to fix the dependency hell without 
getting crazy. In my opinion, everything which needs to be done manually 
when trying to provide information about how to resolve dependencies 
will fail.

And we also don't support build-time-dependencies at all. Because there 
is nothing a test for the availability of build-time-dependencies can 
tell you what a failing configure can't tell you better. (And even 
worse: if a configure does not fail, it does not mean that all build 
time dependencies are satisfied - the make may fail later 8) )
Perhaps nobody will ever solve this - we don't try.

Ok... this is a very large first post.. 8) sorry for that.

just some quick bee examples:

the bee file for LFS chapter 5.4: binutils pass1 [1]:
   (comments and some other stuff removed, original file in [2])

-- bee-file: tc-binutils_pass1-2.22.0-0.bee -- cut --
#!/usr/bin/env beesh
SRCURL[0]="http://ftp.gnu.org/gnu/binutils/binutils-${PKGVERSION}.tar.bz2"

mee_configure() {
     start_cmd ${S}/configure \
         --target=${LFS_TGT} \
         --prefix=${PREFIX} \
         --disable-nls \
         --disable-werror
}

mee_install_pre() {
     if [ "$(uname -m)" = "x86_64" ] ; then
         start_cmd mkdir -pv ${D}/${PREFIX}/lib
         start_cmd ln -sv lib ${D}/${PREFIX}/lib64
     fi
}
-- EOF --

bee-file explained:

beesh: the intepreter of the bee-file - just a collection of bash magic

$PKGVERSION is extracted automatically, which makes updates very easy:
        cp pkg-1.0-0.bee pkg-2.0-0.bee
        ./pkg-2.0-0.bee (you execute bee files to build the package)

$S  variable to point to the sourcedir ($B would be build dir)

$D  variable to point to DESTDIR (bee depends on DESTDIR installations)

$PREFIX prefix to use.. defaults to /usr can be set in bee-config or
         bee-files

$LFS_TGT you know about that 8)

that's all..

As I said, we just released v1.0 to the public (today) - it is used by 
us for over 2 years now - every day.

We don't want to provide a package "manager" for LFS - it's more to tell 
somebody that BEE exists now.. 8))

I think, we will provide a LFS hint soon. Documentation is still rare 
for bee.

We will also present BEE at the linuxtag2012 tomorrow in Berlin (oh it's 
today..)

To test it, download it from our (new) website [3] or from github [4].
bee-lfs (our proof of concept project) can also be found on github [5].

I forgot to mention an important feature: you can use bee as a user to 
install software in your home-diretory - it's like package "management" 
in userland - if you don't have root access, e.g. at your university or 
at work... - easy install , easy update and easy rebuild (if the base 
system changes and you have to rebuild stuff against new library versions)

I will stop here 8) please be nice and don't blame us for inventing yet 
another package system..

best,

   marius


[1] 
http://www.linuxfromscratch.org/lfs/view/7.1/chapter05/binutils-pass1.html
[2] 
https://github.com/bee/bee-lfs/blob/lfs-7.1/chapter05/tc-binutils_pass1-2.22-0.bee
[3] http://beezinga.org/download.html
[4] https://github.com/bee/bee
[5] https://github.com/bee/bee-lfs



More information about the lfs-dev mailing list