BTS

Issue523

Title grml-debootstrap: provide more options
Priority feature Status resolved
Superseder Nosy List mika, suntong
Assigned To mika Topics grml-debootstrap

Created on 2008-09-08.13:45:40 by mika, last changed 2010-02-28.01:26:26 by suntong.

Messages
msg3076 (view) Author: suntong Date: 2010-02-28.01:26:26
I should have closed it long before.
msg3024 (view) Author: mika Date: 2010-02-27.23:13:19
Ping, any news?
msg2402 (view) Author: mika Date: 2009-08-05.09:17:33
* Tong Sun <bts@bts.grml.org> [20090805 08:32]:

> I've come up with something new since last request was made, and might
> find some time later this month to dig into it.

> I'll send in the live-initramfs patch again then as well.

Awesome, thanks!

regards,
-mika-
msg2400 (view) Author: suntong Date: 2009-08-05.06:32:27
I've come up with something new since last request was made, and might
find some time later this month to dig into it.

I'll send in the live-initramfs patch again then as well.
msg2393 (view) Author: mika Date: 2009-08-05.02:09:20
Hi Tong Sun!

I don't have any time to work on this issue. Do you have any plans to check out 
what still might be interesting or is it fine for you if I close this bugreport?

regards,
-mika-
msg1733 (view) Author: mika Date: 2008-12-12.08:38:31
* Tong Sun <bts@bts.grml.org> [20081212 05:48]:

> > Looks like we would have to split package lists of grml-live
> > (/etc/grml/fai/config/package_config/*) into smaller parts to follow
> > this approach.

> > But: I won't have the time for implementing this feature in grml-live.
> > Tong, are you basically interested in taking care of it?

> hmm... less likely at the moment. I tried to use grml-live for several
> times, but in the end gave it up. I now build my live system squashfs
> solely using grml-debootstrap. I still haven't got around to try my
> plan yet, but I believe once I succeeded, it is just a matter of two
> or three extra lines of code in the wiki (that uses aufs to capture
> the changes) before calling grml-debootstrap.

> Due to the time constrain, I can only do small fixes/enhancements at
> the moment, not big architectural restructure. Nonetheless, I still
> thinking building the live system using grml-debootstrap *might* have
> a less steep learning curve, with the help of clear instruction in
> wiki -- which is what I planned to write if I succeed.

Alright, thanks. Please keep me up2date if there are any news. :)

regards,
-mika-
msg1732 (view) Author: suntong Date: 2008-12-12.04:48:09
On Thu, Dec 11, 2008 at 3:28 AM, Michael Prokop <bts@bts.grml.org> wrote:
>
> Michael Prokop <mika@grml.org> added the comment:
>
> * Tong Sun <bts@bts.grml.org> [20081211 05:55]:
>
>> > If anyone is interested, here is how Slax is build, module by module,
>> > using aufs to capture the changes...
>
> Looks like we would have to split package lists of grml-live
> (/etc/grml/fai/config/package_config/*) into smaller parts to follow
> this approach.
>
> But: I won't have the time for implementing this feature in grml-live.
> Tong, are you basically interested in taking care of it?

hmm... less likely at the moment. I tried to use grml-live for several
times, but in the end gave it up. I now build my live system squashfs
solely using grml-debootstrap. I still haven't got around to try my
plan yet, but I believe once I succeeded, it is just a matter of two
or three extra lines of code in the wiki (that uses aufs to capture
the changes) before calling grml-debootstrap.

Due to the time constrain, I can only do small fixes/enhancements at
the moment, not big architectural restructure. Nonetheless, I still
thinking building the live system using grml-debootstrap *might* have
a less steep learning curve, with the help of clear instruction in
wiki -- which is what I planned to write if I succeed.

cheers
msg1731 (view) Author: mika Date: 2008-12-11.08:28:03
* Tong Sun <bts@bts.grml.org> [20081211 05:55]:

> > If anyone is interested, here is how Slax is build, module by module,
> > using aufs to capture the changes...

> > http://lontra.org/pub/Linux/slax/SLAX-6.x/rc4/src/slax-build-6.0.0.tar.gz

> Ah, forgot this, Slax build script from slax-build-6.0.4

> http://live-developers.svn.sourceforge.net/viewvc/live-developers/sys/SlaxMini/trunk/build?view=log

Thanks for the hint.

Looks like we would have to split package lists of grml-live
(/etc/grml/fai/config/package_config/*) into smaller parts to follow
this approach.

But: I won't have the time for implementing this feature in grml-live.
Tong, are you basically interested in taking care of it?

regards,
-mika-
msg1730 (view) Author: mika Date: 2008-12-11.08:21:33
* Tong Sun <bts@bts.grml.org> [20081211 05:37]:

> > BTW, are you using 256K block size when creating squashfs modules?
> > According to Tomas from Slax, this is the best setting to get smaller
> > module size while maintaining good speed.

> Quoting from Tomas:

> * Here is an overview what great results can be achieved using bigger blocks:
>   http://blog.slax.org/2007/11/02/mksquashfs-lzma-block-sizes-comparison/

> * And here is described how it affects the speed of the system:
>   http://blog.slax.org/2007/11/08/rc7-done-but-offline/

> It appears on my computer that block size 256KB is the best compromise
> between speed and size (the ISO is 13MB smaller! and boot takes only 4
> seconds longer),

Thanks for information, Tong. I'm aware of Thomas' squashfs
comparisons. But 4 seconds longer for booting is just too much for
my taste. :-( We at grml are searching for ways how to speed up our
booting sequence. ;)

regards,
-mika-
msg1729 (view) Author: suntong Date: 2008-12-11.04:55:50
On Wed, Dec 10, 2008 at 11:48 PM, Tong Sun <suntong001@gmail.com> wrote:

> If anyone is interested, here is how Slax is build, module by module,
> using aufs to capture the changes...
>
> http://lontra.org/pub/Linux/slax/SLAX-6.x/rc4/src/slax-build-6.0.0.tar.gz

Ah, forgot this, Slax build script from slax-build-6.0.4

http://live-developers.svn.sourceforge.net/viewvc/live-developers/sys/SlaxMini/trunk/build?view=log
msg1728 (view) Author: suntong Date: 2008-12-11.04:49:01
On Sat, Nov 29, 2008 at 10:30 AM, Tong Sun <suntong001@gmail.com> wrote:
> separated module. The next step would be install grml-full on top of
> grml-media, and capture the changes using aufs, but haven't get time
> to go further yet. I'll definitely report back if I make any
> successful progress on it.

If anyone is interested, here is how Slax is build, module by module,
using aufs to capture the changes...

http://lontra.org/pub/Linux/slax/SLAX-6.x/rc4/src/slax-build-6.0.0.tar.gz
msg1727 (view) Author: suntong Date: 2008-12-11.04:37:05
On Fri, Dec 5, 2008 at 3:18 PM, Tong Sun <suntong001@gmail.com> wrote:
> BTW, are you using 256K block size when creating squashfs modules?
> According to Tomas from Slax, this is the best setting to get smaller
> module size while maintaining good speed.

Quoting from Tomas:

* Here is an overview what great results can be achieved using bigger blocks:
  http://blog.slax.org/2007/11/02/mksquashfs-lzma-block-sizes-comparison/

* And here is described how it affects the speed of the system:
  http://blog.slax.org/2007/11/08/rc7-done-but-offline/

It appears on my computer that block size 256KB is the best compromise
between speed and size (the ISO is 13MB smaller! and boot takes only 4
seconds longer),

Ref:

http://lontra.org/pub/Linux/slax/SLAX-6.x/rc7/README
msg1715 (view) Author: suntong Date: 2008-12-05.20:19:07
On Sat, Nov 29, 2008 at 10:30 AM, Tong Sun <suntong001@gmail.com>

>> And regarding the "layered module architecture": I noticed that your're playing
>> with the module layout using live-initramfs. If you have any information that
>> might help us in providing this feature or are interested in working on that
>> I'd love to hear from you. :)
>
> So far, I have just been putting the modules that I build for slax on
> top of grml. It works perfectly, but they are all my home, text or
> script files. I.e., no debian packages involved in the layered modules
> yet. I have something in mind how to go about the debian packages.
> I've successfully split debian packages info dir in var into a
> separated module.

I have

 tmp -> var/tmp/

in my root. And I split var & tmp into different layered module from
the base. This give me the freedom so that when later grml-full is
installed on top of it, I need just to replace the var module in the
grml-full iso, while keeping the grml-media-base as is. This is how I
created the split modules.

 cd $os_mnt
 mksquashfs . ../001-base.squashfs -noappend -b 256K -lzmadic 256K -e var

 mksquashfs var tmp ../002-vars.squashfs -noappend -b 256K -lzmadic 256K

You can exclude more in base and include more in var as you wish.

BTW, are you using 256K block size when creating squashfs modules?
According to Tomas from Slax, this is the best setting to get smaller
module size while maintaining good speed.

> The next step would be install grml-full on top of
> grml-media, and capture the changes using aufs, but haven't get time
> to go further yet. I'll definitely report back if I make any
> successful progress on it.

Another option is just to use

  dpkg-deb -X $dpkg extract_path

to extract a bunch of small tools into 'extract_path', then create
squashfs modules out of it, using the 'live-module -c' that I just
mentioned in my previous email.

This method has been proofed working fine. I put more than 10 Debian
tools together this way and use then in Slax. :-)

cheers
msg1703 (view) Author: suntong Date: 2008-11-29.15:30:42
On Thu, Nov 27, 2008 at 5:44 AM, Michael Prokop <bts@bts.grml.org> wrote:

> I like the base_tgz approach of grml-debootstrap, if you'd like to implement
> that you've my ACK. :)
>
> Regarding --base_sys: is this still valid with current grml-debootstrap?

The --base_sys is no longer necessary. The newly introduced option
--nodebootstrap, ie "Skip debootstrap, only do configuration to the
target", together with the --target option, means the same thing to
it.

The --base_tgz was use to explain the --base_sys option more easily
for me. After thinking over it, I think we'd better not to add it
because it brings more trouble than the little help:

- should we wipe the --target entirely before expanding the tgz file,
or preserver it.
- if user choose not to wipe, but want to overwrite it with files from
the tgz file, how about those left-overs?
- how about .tar.bzip2 compressed files?
- how about other archive files e.g., dar...

It is better leave the user the freedom to prepare the --target
directory the way they want, then invoke grml-debootstrap with the
--nodebootstrap option.

So I think we can close this now.

> And regarding the "layered module architecture": I noticed that your're playing
> with the module layout using live-initramfs. If you have any information that
> might help us in providing this feature or are interested in working on that
> I'd love to hear from you. :)

So far, I have just been putting the modules that I build for slax on
top of grml. It works perfectly, but they are all my home, text or
script files. I.e., no debian packages involved in the layered modules
yet. I have something in mind how to go about the debian packages.
I've successfully split debian packages info dir in var into a
separated module. The next step would be install grml-full on top of
grml-media, and capture the changes using aufs, but haven't get time
to go further yet. I'll definitely report back if I make any
successful progress on it.

cheers
msg1681 (view) Author: mika Date: 2008-11-27.10:44:54
Hi Tong,

I like the base_tgz approach of grml-debootstrap, if you'd like to implement 
that you've my ACK. :)

Regarding --base_sys: is this still valid with current grml-debootstrap?

And regarding the "layered module architecture": I noticed that your're playing 
with the module layout using live-initramfs. If you have any information that 
might help us in providing this feature or are interested in working on that 
I'd love to hear from you. :) 

thanks && regards,
-mika-
msg1472 (view) Author: mika Date: 2008-09-08.13:45:39
Hi Mika:

What do you think of the following enhancement to grml-debootstrap? If you
ok with it, I'll start implementing *after* you release the new grml, for I
don't want to bother you when you are busy releasing new grml.

Add the following options to grml-debootstrap:

--base_tgz <tgz_file>

use the tgz file as the base system for grml-debootstrap, instead of
invoking debootstrap to build one. This allows people to utilize existing
base systems such as the grml daily-built one, or allow people to manually
fine tuning a base system before handling it over to grml-debootstrap.

Further note, without altering the grml-debootstrap configuration structure,
it is possible to use the following trick for grml-debootstrap to work on
different tgz files, i.e. different chroot base systems.

Before calling grml-debootstrap,

 mount -o bind /path/to/a/customized/debootstrap /etc/debootstrap

I.e., by bind mounting different configuration files, it is possible to
use grml-debootstrap to work on different base systems.

-b, --base_sys

Denote that the mountpoint specified by --mntpoint actually contains a
expanded base system as described above. Apart from eliminating the step of
expanding the tgz file, it enables building a modular live system which can
achieve slax's layered module architecture.

Further note on layered module architecture, instead of releasing different
sized live system differently, the slax derivatives' modular way is that
larger live systems are based on smaller ones. For example, here's a
sample module layout (ripped off from
http://forums.wolvix.org/index.php/topic,583.0.html):

* 01_kernel (Guess what's in here)

* 02_core (Core tools, like bash, and the basic GNU utilities)

* 03_ base_app (CLI applications like MC, Elvis, htop and Multimedia tools like 
ogg,
+normalize.)

* 04_base_lib (Basic libraries used by many apps. Mostly audio related.)

* 05_base_net (CLI network tools like mutt, elinks, bind, traceroute, etc.)

-- The grml small edition starts and ends here.

* 06_x11 (Xorg and fonts.)

* 07_common_desk (Xfce plus goodies, Fluxbox plus "goodies", icons, themes, 
wallpapers, etc.
+This one I'll break up a bit more, to separate Flux and Xfce.)

* 08_common_lib (All libs and requirements for the common applications. A lot 
of Gnome,
+graphics and multimedia libraries and bindings. Xfce has some deps here, guess 
I need to
+sort them out.)

* 09_common_app (All the GUI applications, network apps, graphics apps, office 
tools,
+multimedia apps, you name it. Guess this might be broken up a bit too)

-- grml media ends here.

* 10_more_lib (All the libraries and requirements for the apps in Hunter)

* 11_more_app (All additional apps for Hunter, not found in Cub plus large
things like Java and Samba. Perhaps I could make the two last separate
modules.)

-- grml large ends here.

This is just an example showing how Wolvix does module layout. The modular
approach has many advantages, For example, all files which belongs to
Xwindow are packed in xwindow module, KOffice related stuff is in koffice
module, etc. If you work with KOffice, you usually need only files from
KOffice and nothing else; and hence all files from that part of the
filesystem are separated from the rest of it, your CD drive has to seek only
in a 10 MB area, which might be fully cached by Linux kernel. This
significantly improves the speed. Another great advantage is that, I never
need to remaster slax any more, putting my extra stuff in is just as simple
as putting in several modules. My live-usb can always contains the latest
tools/docs of my own, without going through the remastering process.

I know grml is different but the spirit is the same. For example, we
can put forensic analysis, and network-penetration/intruder-detection tools
into different modules, and load them only when necessary.

The trick to build such layered modules in grml is to use Aufs to build a
virtual base system under mountpoint. For example, to build grml media on top
of grml small, use Aufs to combine the readonly grml small base system and
a writable empty directory as the mountpoint directory, then after using
grml-debootstrap to install/config packages for grml media, pack only
the changed files into the grml media module.

Please comment on my plannings. Mike, if it is feasible and make sense,
and after *everything* is agreed upon, I can start coding and send you
the patch set via email.

thanks

tong
History
Date User Action Args
2010-02-28 01:26:26suntongsetstatus: need-eg -> resolved
nosy: mika, suntong
messages: + msg3076
2010-02-27 23:13:19mikasetstatus: needs-progress -> need-eg
nosy: mika, suntong
messages: + msg3024
2009-08-05 09:17:35mikasetnosy: mika, suntong
messages: + msg2402
2009-08-05 06:32:29suntongsetnosy: mika, suntong
messages: + msg2400
2009-08-05 02:09:20mikasetstatus: need-eg -> needs-progress
nosy: mika, suntong
messages: + msg2393
2008-12-12 08:38:33mikasetnosy: mika, suntong
messages: + msg1733
2008-12-12 04:48:13suntongsetnosy: mika, suntong
messages: + msg1732
2008-12-11 08:28:04mikasetnosy: mika, suntong
messages: + msg1731
2008-12-11 08:21:36mikasetnosy: mika, suntong
messages: + msg1730
2008-12-11 04:55:50suntongsetnosy: mika, suntong
messages: + msg1729
2008-12-11 04:49:02suntongsetnosy: mika, suntong
messages: + msg1728
2008-12-11 04:37:07suntongsetnosy: mika, suntong
messages: + msg1727
2008-12-05 20:19:10suntongsetnosy: mika, suntong
messages: + msg1715
2008-11-29 15:30:45suntongsetnosy: mika, suntong
messages: + msg1703
2008-11-27 10:44:55mikasetstatus: unread -> need-eg
nosy: mika, suntong
messages: + msg1681
2008-10-17 03:09:06suntongsettopic: + grml-debootstrap
nosy: + suntong
2008-09-08 13:45:40mikacreate