FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Java

 
 
LinkBack Thread Tools
 
Old 10-24-2011, 05:07 PM
Will Woods
 
Default Merging new image building code (treebuilder)

The stars have aligned! The time is upon us! Iš! Iš! Treebuilder Fhtagn!

I'm planning to merge treebuilder into (or perhaps "over") lorax's
"master" branch today or tomorrow.

In the next days I'll throw together some Feature pages and/or a roadmap
on lorax's Trac instance so we have a clearer idea of what's changed and
what's still coming, but here's the New Features:

- Faster startup[1]!
F15: 25s F16 TC: 25s Treebuilder F16: 15s
- Lower memory use[2]!
F15: 541MB F16 TC: 320MB Treebuilder F16: 195MB
- Normal dracut initrd (21MB! No 'loader'[3]!)
- Netboot for PPC works again[4]!
- New, expanded template language makes customizing images easy[5]!

I've tried to make sure that any bugfixes that went into master got
pulled into treebuilder, but it's possible that I've missed some. So if
you've fixed bugs in lorax recently (or say the past 6 months) it'd
really help if you could check the treebuilder logs to see if the fix
got ported.

Please let me know if you've got any questions about (or problems with)
the plan[7]!

-w

[appendix: some details on the changes in treebuilder]

# Images are built like Live images. This means:
* runtime is LiveOS/squashfs.img
* initramfs is built by dracut
- we do include some special dracut modules (e.g. livenet)
* booting requires a correct root=live:XXX arg
- written into the bootloader config by lorax
- "root=live:/dev/sr0" will work in a pinch
- with livenet, "root=live:http://.../squashfs.img" is valid
(pass "ip=XXX ..." to set up the network first)

# New, expanded template language
- Nearly all the image building happens in the templates
- New commands to help simplify/clarify the build process:
install, mkdir, replace, append, hardlink, symlink,
copy, copyif, move, moveif, remove, chmod, gconfset,
installpkg, removepkg, removefrom, run_pkg_transaction,
treeinfo, installkernel, installinitrd, runcmd
- Lorax image build happens in 4 stages:
install, postinstall, cleanup, and (per-arch) image build.
- More code should move into the templates over time.

# Data/config files moved out of anaconda and into lorax
- These should actually all move out of lorax and into a separate
subpackage, but we need to rework some bits of lorax first.

[1] Time to "Starting Anaconda [...]" message in a KVM guest with 768MB
RAM. Host system is a MacBookPro7,1 running x86_64 F16. Results were the
same for three test runs of each version.
[2] Difference of "MemTotal" and "MemFree" lines in /proc/meminfo after
running "echo 3 > /proc/sys/vm/drop_caches; sleep 1", after anaconda has
started its GUI. Variance between test runs was negligible (~0.1%).
[3] 'loader' is still in the anaconda runtime, but we're working on
replacing it.
[4] netbooting with treebuilder images currently requires different boot
arguments than previous versions of Fedora/RHEL
[5] Well, OK, this is a matter of opinion; some people think shell is
simple and clear enough for any task. (Those people are wrong[5a].)
[5a] This is also a matter of opinion, but if you really want to know
more about What I Think Is Wrong With Bash I'd be happy to explain my
position - using colorful and inventive curse words and possibly obscene
hand gestures - after you buy me a beer.
[7] Special note for future readers: this would have been the time to
bring up the problem(s) you're now having with the plan. Y'know, for
future reference. Which I guess is just "reference" for you, since
you're in the future. How's it going, by the way? Do you guys have
jetpacks yet or what?

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-24-2011, 07:59 PM
John Reiser
 
Default Merging new image building code (treebuilder)

> # New, expanded template language
> - Nearly all the image building happens in the templates
> - New commands to help simplify/clarify the build process:
> install, mkdir, replace, append, hardlink, symlink,
> copy, copyif, move, moveif, remove, chmod, gconfset,
> installpkg, removepkg, removefrom, run_pkg_transaction,
> treeinfo, installkernel, installinitrd, runcmd
[snip]
> - More code should move into the templates over time.

I had some problems with the new templating during the development
of treebuilder branch. I could not figure out what was happening,
documentation was sparse, and the result was just "wait."

If the template language cannot detect and report errors (or
"policy violations") by line number, or cannot be debugged/
traced/breakpointed/etc., then I consider that to be a
significant disadvantage.

What is the current status of new templates with regard to
detecting and reporting errors, debugging, and other aspects
of Usability, Reliability, and Supportability?

--

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-24-2011, 09:58 PM
Will Woods
 
Default Merging new image building code (treebuilder)

On Mon, 2011-10-24 at 12:59 -0700, John Reiser wrote:
> > # New, expanded template language
> > - Nearly all the image building happens in the templates
> > - New commands to help simplify/clarify the build process:
> > install, mkdir, replace, append, hardlink, symlink,
> > copy, copyif, move, moveif, remove, chmod, gconfset,
> > installpkg, removepkg, removefrom, run_pkg_transaction,
> > treeinfo, installkernel, installinitrd, runcmd
> [snip]
> > - More code should move into the templates over time.
>
> I had some problems with the new templating during the development
> of treebuilder branch. I could not figure out what was happening,
> documentation was sparse, and the result was just "wait."

After you asked last time, I improved the exception handling and added a
bunch of docstrings for the template commands. (See e.g. commits
318c843e and 473f01c6.)

Exceptions currently give you the exact command and error on the
console, like this:

template command error in x86.tmpl:
install HERP DERP
IOError: nothing matching /srv/[...]/installroot/HERP in /

The full traceback, with associated line number, is written to the log:

: template line 10: install HERP DERP
: template command error in x86.tmpl:
: install HERP DERP
: IOError: nothing matching /srv/[...]/installroot/HERP in /
: Traceback (most recent call last):
: File "[...]/ltmpl.py", line 175, in install
: for src in rglob(self._in(srcglob), fatal=True):
: File "[...]/ltmpl.py", line 93, in rglob
: raise IOError, "nothing matching %s in %s" % (pathname, root)
: IOError: nothing matching /srv/[...]/installroot/HERP in /

I also added comments to the templates themselves to try to explain
what's going on in there. Oh, and there's a 'log' command, so you can
make the templates write info into the logs.

Templates do not stop on errors, but you can enable that behavior by
setting 'fatalerrors' to True in treebuilder.py.

Also, if you wanted to end a template early, a simple <% return %> would
do. You might want to read the Mako documentation to better understand
how mako templates work: http://www.makotemplates.org/

I'll note here that the PPC secondary-arch team is currently using
treebuilder to make their images and it's working pretty well for them.

> If the template language cannot detect and report errors (or
> "policy violations") by line number,

As above, you get the exact template line number - although that's the
*parsed* template, which can be longer than the input (due to looping
and things like that). Still, it really shouldn't be that hard to figure
out the source of the problem, given the exact command being executed.

If there's something specific that you couldn't understand, feel free to
ask, but "I had some problems and I couldn't figure out what was going
on" isn't something I can act on to improve.

> or cannot be debugged/traced/breakpointed/etc.,

I don't consider the lack of debuggers or breakpoints a show-stopper
(you ever set a breakpoint on a loop in bash?) but in theory you could
use the Python debugger on Mako modules; you might try asking the Mako
developers about it if you really must pursue that further.

> then I consider that to be a significant disadvantage.

Compared to what? Previous versions of lorax also use mako to generate
the templates; I just added more commands, consolidated all the command
parsing in one python class (LoraxTemplateRunner() in ltmpl.py), and
added docstrings (with examples) for every command available.

I think, on the balance, the advantages outweigh the perceived
disadvantages.

> What is the current status of new templates with regard to
> detecting and reporting errors, debugging, and other aspects
> of Usability, Reliability, and Supportability?

I hope I've explained the current state of these things above. If anyone
has further questions, or ideas for specific improvements, I'd really
like to hear them.

-w

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 

Thread Tools




All times are GMT. The time now is 07:20 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org