Overriding install prefix in build systems

Following up with reviewing capabilities of various build systems in previous post, recently I looked how to build all packages with custom --prefix and friends values. This is continuation of Optware for Android project, where I'd like to achieve ability to install on non-rooted device.

Here are the results:

• I was surprised at first, but OpenWRT has --prefix hardcoded as /usr when calling configure. Fortunately, that call is in common file, so it would be easy to introduce configurable variable instead. However, it turns out that install routines, which are individual for each package, have /usr hardcoded either, so each recipe needs to be patched still, which is problem for maintainability and/or upstreaming.
• Looking at Optware, situation is even worse - there, there's even less common infrastructure, and even configure call is hardcoded in each script with --prefix=/opt.
• Buildroot, being prototype for OpenWRT, has the same hardcode at /usr.
• I obviously turned to OpenEmbedded at that point (well, I tried Yocto this time) - it for sure always have had nice configurable variables for prefix and all other layout variables (--sysconfdir, etc.). Turned out, while variables are there, OE/Yocto is not ready out of the box to full configurability of both prefix and base_prefix. There were few issues seen:
• Compiler search path issues. Some aspects of search paths (mostly related to shlib linking) appear to be not stored by gcc when configured during cross-compiler, and require command-line overrides. OE passes --sysroot on each invocation, but I had to add few -rpath-link directives to get it right. While it took some time to figure the need for -rpath-link, I'm still not sure I figured where to put them (well, per upstreamability criteria).
• Some packages still hardcode paths, especially related to base_prefix. Sometimes it's a bit harder than trivial to patch (like with busybox).
• Ordering issues. Some packages implicitly assume that base_prefix and prefix are different, and filter filesets during packaging by file prefix. This doesn't work if base_prefix==prefix, so requires reshuffling filtering patters and/or package order (fortunately, OE has well-defined package splitting order).

Critical review of build systems

Last time I looked at possible alternatives to OpenEmbedded, I mostly looked at how many packages they support, how many architectures/OSes, and how easy is to add overrides. Of course, OpenEmbedded outperforms any other generally available build system in these respects.

But still, while the idea of grand unified build system is very cool, maybe it would be easier to add/extend support for some non-Linux OS to some other build system? (As OE is targetted at Linux.) Let's see.

GUB
This software tries to steal "grand unified" motto, though so far it has 200+ packages, but supports Windows. Recipes are structured as Python classes, overrides are handled like:

class Ctypes__mingw (Ctypes):

 One thing which immediately catches eye is "def patch (self)". Yeah, Python doesn't let people play with curlies, but they'll go out fo their ways with braces. Recipes-classes do allow deal of declarativeness, for example, in specifying patches, and yet there're stuff like:

    def patch (self):
        self.system ('''
cd %(srcdir)s && patch -p1 < %(patchdir)s/busybox-mkconfigs.patch
''')

Summary: Violates standard codestyle for target language (PEP8), breaks declarative vs imperative metadata separation.

mingw-cross-env
Targets mingw32 (only). Doesn't support DLLs. Uses .mk for recipes. Both recipes and other files (patches, etc.) lie in one flat directory. Recipes look clean, but one target and no sharedlibs is trivial case on its own.

Smallscreen mod for GtkFileChooser

OpenEmbedded have had smallscreen patch for GtkFileSelection widget for quite some time now, but unfortunately it doesn't affect usability of the most software, simply because many programs now use GtkFileChooser widget instead. GTK+ 2.12 manual also following about the matter: "GtkFileSelection has been superseded by the newer GtkFileChooser family of widgets."

Let's look how standard GtkFileChooserDialog looks on QVGA portrait device (240x320):



Note that "Places" pane on left cannot be shrinked beyond the size shown, which doesn't leave room for any useful display of filenames, so the dialog is clearly unusable.

So, I've took time to patch GTK+ 2.10.14 (this is the version used by Angstrom 2007.12, and my primary focus is to improve its usability on the supported devices) to make the widget be more PDA-friendly. I started with "data-driven" changes (meaning changing numbers and strings, but note code constructs):

• Reduced Gnome Human Interface Guidelines mandated spacing and borders of 6 and 12 pixels to 2. It's nice to have standards (even though Gnome doesn't offer us opportunity to choose from ;-) ), but on small screens, there's simply not enough space to spend pixel-wealth on gaps.
• Removed labels from Places pane's buttons, leaving only icons.
• Made Places pane shrinkable.
  

Even these changes made big difference, but stopping there would be to leave the dialog in barely usable state, and I'm really hope for a Linux GUI environment which is comfortable to use.

So, adaptive relayout was required, based on "PDA" mode being active. I first started with defining PDA mode as being portrait screen orientation, but thinking further, the problem, at least for GtkFileChooser, can be defined as follows: there's just too little of horizontal space available, and there should be efforts applied to use it to show as much of user's filenames' length as possible. Based on this note, PDA mode was defined as having a screen width below some threshold. This threshold was set as absolute value, but is not really arbitrary, and tied to real-world available screen sizes. I found that standard dialog looks pretty well in VGA landscape orientation, but still leaves too little space for filenames
in portrait orientation. So, threshold was set to 490 (to account for fudge factor ;-) ).

Now, with the above note on horizontal space scarcity, and keeping in mind that screen area at all is scarce, for vertical space should be saved too, following dynamic layout changes are done to dialog when being in PDA mode (i.e. run on the screen <490 pixels width):

• Vertical split pane is used for Places vs Filelist
• Buttons are packed to the left instead of bottom of Places
• Actually, only "Add" button was left, as "Remove" action can be easily accessed from Places list context menu, and most importantly, intuitively can be expected there. ("Add to shortcuts" choice is available for context menu on Filelist items, but who'd look for it there, without knowing it is?). This allows for more visually pleasing layout when Places pane is minimized.
• Dialog itself is sized to take all available width (originally, GTK+ sizes it to take no more than 3/4 of screen width, if possible, or packed requested size otherwise).
  

All in all, that's how the dialog looks now:


A refreshing change, with pretty long filenames being easily visible.

I've tested the patch pretty well, plus code changes are pretty small, so I intend to commit it to OE soon, and RFC its addition to Angstrom stable branch after it gets more review and testing.

There're still few things to consider and do to make it even more usable on small, PDA-like screens:

• Catch and process screen geometry change events - I'm pretty sure this is not done by GTK+ currently, and on dynamic screen device like PDA could make parts of dialog go offscreen when switching from landscape to portrait and vice-versa.
• Possibly, adjust screen width threshold based on more testing (original value was based on review of big-gaps dialogs, maybe smaller gaps affect that).
• Alternatively, gaps can be calculated on the fly, so original Gnome HIG-mandated values would be used on big screens, and small values only in PDA mode.
• Default "File system" choice in Places is superflous, it can be accessed with the icon from the path buttons at the top of dialogs. Unfortunately, screenshots above have it scrolled away. And having "File system" is Places causes exactly same problem: it takes space and calls for scrolling, so we'd better skip it there altogether, letting user to access own choices easier.
• Think about dialog height. Currently, as seen from above screenshot, it leaves bunch of screen area unused, and (apparently) due to Matchbox window management constraints, it cannot be resized by user. We could fit 1-2 more filenames if sized the dialog near-fullscreen. But that would look OK for QVGA, but already artificial for VGA.
  

Finally, it would be fun to have a look at GTK+ 2.12. As they say, it has stuff contributed by Nokia based on N770/N800 work, so I keep my fingers crossed that I didn't do monkey's job! ;-) Nonetheless, the focus is current Angstrom release, as I wrote above.

*Real* C#/Java support in OE

I finally found time to read Cliff Brake's excellent article summarizing his work on adding realy-working C#/.NET support to OE. Well written, it gives info why C# is useful choice for embedded systems, compares it to C and Python (great choices!) and details how to use it in OE. What of course would be *really* interesting to know is bug/feature (in the sense of performance, memory/disk footprint, compatibility with existing standards, etc.) comparison of current open-source C# and Java implementations, to be able to make choice for some new, independent project. Well, that's not going to be that easy so far.

One point where .NET has win over Java is "by default" portability/device coverage. It's pretty to write simple applications which run both on full-fledged .NET and CompactFramework, which means it's easy to develop small utils on desktop, and have them run without extensive debugging on PocketPC device. Back in the days I still bothered with trying to make PocketPC do what I need, that was a big win! Example of real-worldly usable app written in such way is here. Of course, back then, it was very hard to get full benefit of that opportunity, due to the way vendor provided toolset for .NET development - overloaded with GUIs and unimportant features, it possibly helped someone with a good favorite editor, but made integration work a great chore. Reading thru Cliff's article, one may once again rejoice that we have OpenEmbedded, which handles task of integrating 3rd-party dependencies into the project and complete deployable solution a breeze.


Returning back to Java, we are going to have fresh Java support too, thanks to the work of
Robert Schuster. Hope it will land soon in OE mainline, so the application developers/integrators for Angstrom, OpenMoko, etc. will have unprecedented choice of tools available at their fingertips!

Back from hiatus

Having spent August on vacation and later working with my GSoC student, Maria Zabolotnaya, I now gradually come back to speed on embedded stuff, even though I likely won't have as much time to work on this as before.

The first aim is to get OE build again, which is "of course" broken after being away for some time and then coming back ;-). Then, process all the backlog of HH.org kernel patches and cleanup tasks. Further aims will be to automate process as much as possible (for example, I finally want to setup local Continuous Integration for OE builds, like I did for HH.org kernel), and upgrade HH.org kernel to 2.6.22 (though to that time 2.6.23 likely will be out).

Finally out of the list of pressing OE issues

With the last changes to properly support modprobe 2.6 for kernel module configuration directive, I can't remember other pressing issues with OE which would prevent its proper and flexible usage for distro engineering. Among the other infrastructure changes done within the last half-year with I was involved are:

• Support for building efficient initrd/initramfs images
  
• Flexible and extensible initramfs to boot rootfs from different media
• Adding support for machine screen properties
• Image building speed improvements
• Alleviating maintenance effort for kernel porting projects with lots of supported devices (using defconfigman)

With all these and other changes, era of grunt work on OE should be over, and finally more effort can be dedicated to more sensible work - Angstrom distro engineering and software/package porting and maintenance.

Modular initramfs for OE

While all new initramfs-related features are yet need to be described, they are already put to work. Alex Osborne of Hack'n'Dev created a modular initramfs system to serve all the device booting needs. We discussed design principles and came to following tentative picture:

• initramfs in question handles just the task of mounting rootfs and booting from it.
• Doing so, it tries to follow both existing kernel behavior and interface, like command line parameters. For example, NFS boot uses standard root=/dev/nfs and nfsroot= parameters. Where extended feature is added, it tries to mimic existing parameters. For example, loop device boot uses root=/dev/loop and looproot=.
• Support for different boot methods is implemented by modules with more or less well defined interface. It's possible to construct initramfs images with the given feature by combining modules.
• Generally, new and extra features are expected to be added using modules, to keep design clean.

It is committed by now. Both loop and NFS boot modules are tested to work. I hope that Alex will describe loop module command line syntax in more detail in his blog.

WANTED: modules to boot via WiFi, with WEP and WAP support, and to boot via GPRS ;-D

Initrd and initramfs with OE: snippet 1

One of the former weak areas of OpenEmbedded was creation of small, fine-tuned, (hand)optimized initrd and initramfs images. However, during few last months, the situation vastly improved, and by now OE offers all tools needed to create initrd/initramfs per your purposed and requirements. I would like to describe them, and as that would be quite a bit of contents, going to do it describing features one by one, in "snippets".

The first is:

Building initramfs with OE and everything you wanted to know about initramfs but didn't bother to figure out yourself.

IMAGE_FSTYPES = "cpio.gz"

Yes, that's all what you need to create an initramfs image, voila. After that, you can use resulting image instead of conventional initrd (in ext2 or some other format). For the kernel to be able to boot it, you need to have CONFIG_BLK_DEV_INITRD=y in its config (contrary to its name, this option actually means "support external initrd *or* initramfs images"). As initramfs support is unconditionally builtin into kernel, you don't need any filesystem enabled for initramfs support.

(Note that above is written for standalone initramfs image which is used in initrd-like manner. Support for initramfs which are linked into the kernel image so far is not provided OE, but would be pretty easy to add. Still, it's of more interest to vendors which work on specific products than OpenSource distros. By the way, mentioned config option is not required for builtin initramfs.)

Note that execution of initramfs starts with /init. If you want to rebuild initrd (or otherwise conventional rootfs image) as initramfs, you'll need to pass following parameter on kernel command line to have it started properly: "rdinit=/sbin/init".

Final note is that contrary to popular belief, initramfs does not completely covers and supersedes initrd functionally. Initramfs has relatively big memory requirements, which precludes usage of big initramfs images on low-memory systems. Specifically, initramfs is compressed, and needs to be uncompressed during startup, which already limits its size to at most half of the available RAM. But actually, ramfs, which is used as backing store for initramfs content, is pretty inefficient for such type of content (static), and that imposes even more limits. Also, my experiments show that kernel has poor error handling while uncompressing initramfs, and out of memory condition makes it just hang. So, when you need to fill memory up with lots of content, nothing would replace an initrd with builtin compression like cramfs or squashfs. Granted, that's rather adhoc need.

Finally a kernel hackers' initrd image in OE!

When I only started with Linux-on-PDA hacking, I, as people usually do, combed internet for any kernel/initrd images which were marked as being for the PDA model I had at that time. In such way I picked up few initrd's, and after practically comparing among themselves, selected one for standard hacking use, like kernel testing. A bit later I augmented it by installing some missing stuff (like mc), and seeing that many people are not so thorough and searching the net, put it up for others' use. That's basically what I've been using since them and suggesting to others who needed environment to test some kernel feature with minimal userspace involvement.

Of course, that situation was rather ridiculous - using some random binary-only image instead of OE-built is not perfect at all, and I was just hoping that someone else would go for that. Well, with the latest changes in OE to facilitate creating efficient initrd/initramfs images, I finally got to this myself. So, here's now such image, codenamed 'devimage'. Consequently, recipe to build it as a complete image is called 'devimage-image.bb'.

Its purpose defined as: provide environment which by default runs only minimal userspace (busybox sh and dropbear now, to support different connection methods), but offers all needed "kernel userspace" utilities from udev to specific utils for ALSA, IrDA, Bluetooth, WiFi, etc.

At the current stage it's not as complete as that, so help with filling it right is welcome (caveat: size should be kept in mind).

It's all firefighting, right?

Well, time for a nice "depressive" post ;-).

So, at the beginning of week I decided to port GParted. Was more or less easy, just a fix for an obvious thinko from the author regarding C++ std namespace ;-). However, it failed on linking stage with infamous rpl_malloc() and friends not resolved for libparted. Looking thru cofigure log, that's of course result of crosscompiling assumptions, but OE has defines for the corresponding tests in autoconf site files, so what was actually happening is shipped parted 1.8.7's configure not loading these site files.

It also turned out that autoreconf'ing of this parted version doesn't work, as it requires autoconf 2.61 and friends as minimal, while OE has 2.59. Trying autoreconfing 2.61 (which is DEFAULT_PREFERENCE=-1 in OE) lead to the same issue with site files loading. At this point I submitted bugs on this, and raised issue on #oe. Actually I didn't even expect immediate big attention, but it caught developers' eye, and with the help of XorA and hrw, to the end of the day we had a solution. (And the root of issue was bug in autoconf 2.61. They used "shell programming patterns" (cough) you really don't want to use in your scripts, and the result is proverbial - they outslied themselves).

Well, on the droop of day, GParted was built. I run in QEMU and was surprised that UI was pretty barely usable on QVGA. Run on a real device to try a real card brought a gore however. First of all, GParted didn't detected my SD card at all. I even went looking at GParted source for how it scans devices, but then run parted on it. "Overlapping partitions now supported" was abort message. Overlapping partitions? Me? No! Wrong call! Fortunately, running fdisk showed instead of the senseless bold accusation the fact that my camera indeed tried to dig weird holes in parttable, apparently confused with ext2 partition there ;-)

Well, finally GParted was run, bliss. But just going thru menus once made them no longer accessible. Already trained on such stuff with LiveRamdisks work, I knew it's out of memory. Running ps aux showed some unusual suspects. Killing them (btw, GParted's RSS was "merely" 15Mb comparing to those 6Mb's. Ah, it's all shared libraries, ok, ok, I shut up.), I finally had the thing live. Now closer look look showed it doesn't recognize EXT2 partition (at least FAT it did). The same in parted. Have no idea is it camera hosed it or it's just proverbial GNU software quality. It also turned out that the only operation I was able to do is to change boot flag for a partition ;-). I tried to format unrecognized partition as EXT2 for example, and it failed after shuffling a card for a bit. Getting to HTML report it laid in the home dir surfaced "sh: nice: Command not found". Oh, oops, dear app ;-).

Finally, I wanted to make a screenshot, but gpe-scap failed to write it, complaining that it can't do UTF-8 -> ISO-8859-1 conversion for PNG CREATOR chunk (soft laugh).

---

So, what are the conclusions? X is not an ultimate solution, but merely an option. Lure to run large body of existing, and updated, software is still a lure, as besides this opportunity X brings bloat, which on resource-constrained system vice-versa inhibits ability to run all that wealth of soft well.

Also, all those kernel-hacking games are really old. And I hope that everyone on the block understands by now that what they get once kernel is well and polished is havoc in userspace (this applies to new players like Neo1973 too). So, it's time (yes, finally now, if not long ago)
to leave machine-level feudalism behind, and bring all Linux PDA/phone communities together, to work on common aims. And OE is well established community crossroad project, so implementation-wise, that would mean making sure that any more or less developed device is available in OE, and that any newcomer for a questions "Is X supported?" or "Where can I find stuff for X?" could be answered "Use OpenEmbedded." Yes, right that - use OpenEmbedded, period. Not even "You can use OpenEmbedded, just copy one to another one, and tweak this and that".

First steps at making OPIE fit well with OE filesystem structure

One of the requirement set for OPIE to be supported for Angstrom was compliance with standard GNU/Linux installation location ($prefix and friends). There was bug in OE long enough for this, with few developers considering how to proceed with this.

I finally found time to test some ideas from it, and initial conversion from /opt/QtPalmtop to /usr went pretty smooth requiring besides rebuild few fixes to packages where the former path was harcoded in the literal form. Resulting opie-image booted in QEMU quite well, so I went with committing it, bumping snaphost date for OPIE 1.2.3-pre to 20070702 at the same time.

Hope other developers will help with testing it, and then the conversion needs to be fixed by moving data and plugin to appropriate locations (now they live at places like /usr/pics). This would take a bit more effort exactly because of this dichotomy - data vs plugins, with first should be living under /usr/share, and the latter (ideally) under /usr/lib. I wonder, if a solution like making QTDIR, etc. environment variables to accept list of paths is good enough (well, it is generic from system design point of view, but adds extra runtime overhead).

Running "user" QEMU emulation on Fedora Core 5

I have few boxes around with different OSes/distros and try to use them in reasonable manner. For a some time I contemplated setting up continuous integration for OpenEmbedded, and the box suiting that purpose runs FC5 here.

Well, soon after running a test build on it, I hit issue that OE's binary locale generation, done by running target localedef under QEMU "user" emulation (qemu-arm) errors out there, and on closer inspection, just segfaults. Taking some time to run it thru gdb and strace, I found that it happens after mmap2() syscall, and in seemingly random places in code. More close look at the stuff showed that address parameter to faulty mmap() kind of correlates with VDSO location reported by gdb. Gotcha! So, somehow guest process manages to overwrite VDSO as used on FC5 system.

Now it took some time to figure out what's happening, but it turned out that FC5 applies whole bunch of patches (mostly labelled "execshield" and "xen") affecting VDSO handling to 2.6.15 kernel, when final set of VDSO randomization patches, etc. was not yet in mainline. In other words, FC developer took their own try on solving VDSO issues, with the outcome described above.

Well, it took only couple hours more to rebuild kernel without those patches and teaching it to pickup original initrd (maybe I should have rebuilt it instead, I just didn't do that before). And now I have pristine and full-feature OE builds on FC5 system, voila! ;-)

Blog start and older news

I always treated blogs (and generally diaries) skeptically, but well, all tools have their uses. So, I'm going to start blogging on my Embedded Linux (well, probably not only embedded) endeavors.

And first of all, older news, which I kept in my Handhelds.org wiki page.

2005-11-13: Subscribed to the h4100-port mailing list.

2005-11-29: Confirmed by bisecting that PXA suspend/resume broke in 2.6.13 comparing to 2.6.12. Turned out ot was fixed in 2.6.15

2005-11-29: Subscribed to the kernel-discuss mailing list.

2006-02-03: Proposal for asic3_mmc cross-machine refactor.

2006-02-05: Faced strange issue with ASIC3 suspend/resume, took few months to resolve ;-)

2006-05-30: Posted patch for h4000 ASIC3 suspend/resume problem, turned out to be as "simple" as initializing MSCx PXA registers after resume, what Linux kernel doesn't do (leaving that to bootloader). Anyway, this was second big issue I solved, and after that I felt "force" ;-).

2006-06-03: Started to work on final version asic3_mmc cross-machine refactor, based on the previous experiments.

2006-06-19: Final version of asic3_mmc cross-machine refactor posted.

2006-06-21: Granted write access to !HandHelds.org Kernel CVS repository.

2006-08-20: Posted RFC for common button map for all HH.org PDA-type machines.

2006-08-24: Granted write access to OpenEmbedded Monotone repository.

2006-09-20: Unofficial Familiar 0.8.4 release for h4000.

2006-10-03: Received donation of h3900 device to support elaboration of 2.6 kernel port for it and advance userspace/distrubutions support. Contributor: Mr. Alex Deucher. Thank you!

2006-10-11: Posted patch for common button map for all HH.org PDA-type machines.

2006-10-18: Received donation of hx4700 device to support elaboration of 2.6 kernel port for it and advance userspace/distrubutions support. Contributor: preferred to stay anonymous, but many thanks!

2006-11-09: Upgraded HH.org kernel CVS to 2.6.18.

2006-11-14: Finished implementation of asic3_keys driver and making sure it works well with gpio_keys.

2006-11-21: With having common linux-handhelds-2.6_*.bb and consistent names for all iPaq machines, first phase of cleaning up HH.orh-maintained machines in OE.dev is complete.

2006-12-02: Upgraded HH.org kernel CVS to 2.6.19.

2006-12-19: Angstrom upgraded to 2.6.16-hh8 kernel with common buttonmap support.

2006-12-22: h3900 reached full "C" status, with partial "B" status, in other words, basic PDA functionality works!

2006-12-28: h3900 2.6 port added to OE.dev, requested support by Angstrom.

2007-01-04: Initial release of [http://sourceforge.net/projects/defconfigman/ defconfigman]

2007-01-10: Started to revamp OPIE support in OE.dev, starting with cleaning up qte patchset.

2007-01-18: Received new contribution: h3600 and palmtt2 devices. Thanks! Now I have access to PXA250, PXA255, PXA270, StrongArm, OMAP devices.

2007-01-20: asus620 port updated in CVS. Thanks to Vincent Benony, asus620 maintainer.

2007-01-21: defconfigman kernel's sound tested to work with GPE. Both h3900 and h4000 in OE.dev now have defconfigs generated by defconfigman. h4000 updated to use modules for sound.

2007-02-16: Upgraded HH.org kernel CVS to 2.6.20.

2007-03-05: Continuous Integration system for kernel tree set up, http://handhelds.org/~pfalcon/busyb/buildlogs/

2007-03-17: Proposal to use Defconfigman-generated configs for kernel CVS, to improve consistency and CI coverage.

2007-04-11: GPIODEV API RFC posted to LKML/LAKML

2007-04-26: All 30 ports in HH.org CVS are buildable now (for 2.6.20)!

2007-05-01: SoC base drivers RFC posted to LKML.

2007-05-07: Upgraded HH.org kernel CVS to 2.6.21.

2007-05-11: ipkg speedup patches committed to OE.dev

2007-05-15: Mystery of PCMCIA ACX is solved, first WiFi ping on h4000!