Finally out of the list of pressing OE issues

With the last changes to properly support modprobe 2.6 for kernel module configuration directive, I can't remember other pressing issues with OE which would prevent its proper and flexible usage for distro engineering. Among the other infrastructure changes done within the last half-year with I was involved are:

• Support for building efficient initrd/initramfs images
  
• Flexible and extensible initramfs to boot rootfs from different media
• Adding support for machine screen properties
• Image building speed improvements
• Alleviating maintenance effort for kernel porting projects with lots of supported devices (using defconfigman)

With all these and other changes, era of grunt work on OE should be over, and finally more effort can be dedicated to more sensible work - Angstrom distro engineering and software/package porting and maintenance.

Modular initramfs for OE

While all new initramfs-related features are yet need to be described, they are already put to work. Alex Osborne of Hack'n'Dev created a modular initramfs system to serve all the device booting needs. We discussed design principles and came to following tentative picture:

• initramfs in question handles just the task of mounting rootfs and booting from it.
• Doing so, it tries to follow both existing kernel behavior and interface, like command line parameters. For example, NFS boot uses standard root=/dev/nfs and nfsroot= parameters. Where extended feature is added, it tries to mimic existing parameters. For example, loop device boot uses root=/dev/loop and looproot=.
• Support for different boot methods is implemented by modules with more or less well defined interface. It's possible to construct initramfs images with the given feature by combining modules.
• Generally, new and extra features are expected to be added using modules, to keep design clean.

It is committed by now. Both loop and NFS boot modules are tested to work. I hope that Alex will describe loop module command line syntax in more detail in his blog.

WANTED: modules to boot via WiFi, with WEP and WAP support, and to boot via GPRS ;-D

Initrd and initramfs with OE: snippet 1

One of the former weak areas of OpenEmbedded was creation of small, fine-tuned, (hand)optimized initrd and initramfs images. However, during few last months, the situation vastly improved, and by now OE offers all tools needed to create initrd/initramfs per your purposed and requirements. I would like to describe them, and as that would be quite a bit of contents, going to do it describing features one by one, in "snippets".

The first is:

Building initramfs with OE and everything you wanted to know about initramfs but didn't bother to figure out yourself.

IMAGE_FSTYPES = "cpio.gz"

Yes, that's all what you need to create an initramfs image, voila. After that, you can use resulting image instead of conventional initrd (in ext2 or some other format). For the kernel to be able to boot it, you need to have CONFIG_BLK_DEV_INITRD=y in its config (contrary to its name, this option actually means "support external initrd *or* initramfs images"). As initramfs support is unconditionally builtin into kernel, you don't need any filesystem enabled for initramfs support.

(Note that above is written for standalone initramfs image which is used in initrd-like manner. Support for initramfs which are linked into the kernel image so far is not provided OE, but would be pretty easy to add. Still, it's of more interest to vendors which work on specific products than OpenSource distros. By the way, mentioned config option is not required for builtin initramfs.)

Note that execution of initramfs starts with /init. If you want to rebuild initrd (or otherwise conventional rootfs image) as initramfs, you'll need to pass following parameter on kernel command line to have it started properly: "rdinit=/sbin/init".

Final note is that contrary to popular belief, initramfs does not completely covers and supersedes initrd functionally. Initramfs has relatively big memory requirements, which precludes usage of big initramfs images on low-memory systems. Specifically, initramfs is compressed, and needs to be uncompressed during startup, which already limits its size to at most half of the available RAM. But actually, ramfs, which is used as backing store for initramfs content, is pretty inefficient for such type of content (static), and that imposes even more limits. Also, my experiments show that kernel has poor error handling while uncompressing initramfs, and out of memory condition makes it just hang. So, when you need to fill memory up with lots of content, nothing would replace an initrd with builtin compression like cramfs or squashfs. Granted, that's rather adhoc need.

Finally a kernel hackers' initrd image in OE!

When I only started with Linux-on-PDA hacking, I, as people usually do, combed internet for any kernel/initrd images which were marked as being for the PDA model I had at that time. In such way I picked up few initrd's, and after practically comparing among themselves, selected one for standard hacking use, like kernel testing. A bit later I augmented it by installing some missing stuff (like mc), and seeing that many people are not so thorough and searching the net, put it up for others' use. That's basically what I've been using since them and suggesting to others who needed environment to test some kernel feature with minimal userspace involvement.

Of course, that situation was rather ridiculous - using some random binary-only image instead of OE-built is not perfect at all, and I was just hoping that someone else would go for that. Well, with the latest changes in OE to facilitate creating efficient initrd/initramfs images, I finally got to this myself. So, here's now such image, codenamed 'devimage'. Consequently, recipe to build it as a complete image is called 'devimage-image.bb'.

Its purpose defined as: provide environment which by default runs only minimal userspace (busybox sh and dropbear now, to support different connection methods), but offers all needed "kernel userspace" utilities from udev to specific utils for ALSA, IrDA, Bluetooth, WiFi, etc.

At the current stage it's not as complete as that, so help with filling it right is welcome (caveat: size should be kept in mind).

It's all firefighting, right?

Well, time for a nice "depressive" post ;-).

So, at the beginning of week I decided to port GParted. Was more or less easy, just a fix for an obvious thinko from the author regarding C++ std namespace ;-). However, it failed on linking stage with infamous rpl_malloc() and friends not resolved for libparted. Looking thru cofigure log, that's of course result of crosscompiling assumptions, but OE has defines for the corresponding tests in autoconf site files, so what was actually happening is shipped parted 1.8.7's configure not loading these site files.

It also turned out that autoreconf'ing of this parted version doesn't work, as it requires autoconf 2.61 and friends as minimal, while OE has 2.59. Trying autoreconfing 2.61 (which is DEFAULT_PREFERENCE=-1 in OE) lead to the same issue with site files loading. At this point I submitted bugs on this, and raised issue on #oe. Actually I didn't even expect immediate big attention, but it caught developers' eye, and with the help of XorA and hrw, to the end of the day we had a solution. (And the root of issue was bug in autoconf 2.61. They used "shell programming patterns" (cough) you really don't want to use in your scripts, and the result is proverbial - they outslied themselves).

Well, on the droop of day, GParted was built. I run in QEMU and was surprised that UI was pretty barely usable on QVGA. Run on a real device to try a real card brought a gore however. First of all, GParted didn't detected my SD card at all. I even went looking at GParted source for how it scans devices, but then run parted on it. "Overlapping partitions now supported" was abort message. Overlapping partitions? Me? No! Wrong call! Fortunately, running fdisk showed instead of the senseless bold accusation the fact that my camera indeed tried to dig weird holes in parttable, apparently confused with ext2 partition there ;-)

Well, finally GParted was run, bliss. But just going thru menus once made them no longer accessible. Already trained on such stuff with LiveRamdisks work, I knew it's out of memory. Running ps aux showed some unusual suspects. Killing them (btw, GParted's RSS was "merely" 15Mb comparing to those 6Mb's. Ah, it's all shared libraries, ok, ok, I shut up.), I finally had the thing live. Now closer look look showed it doesn't recognize EXT2 partition (at least FAT it did). The same in parted. Have no idea is it camera hosed it or it's just proverbial GNU software quality. It also turned out that the only operation I was able to do is to change boot flag for a partition ;-). I tried to format unrecognized partition as EXT2 for example, and it failed after shuffling a card for a bit. Getting to HTML report it laid in the home dir surfaced "sh: nice: Command not found". Oh, oops, dear app ;-).

Finally, I wanted to make a screenshot, but gpe-scap failed to write it, complaining that it can't do UTF-8 -> ISO-8859-1 conversion for PNG CREATOR chunk (soft laugh).

---

So, what are the conclusions? X is not an ultimate solution, but merely an option. Lure to run large body of existing, and updated, software is still a lure, as besides this opportunity X brings bloat, which on resource-constrained system vice-versa inhibits ability to run all that wealth of soft well.

Also, all those kernel-hacking games are really old. And I hope that everyone on the block understands by now that what they get once kernel is well and polished is havoc in userspace (this applies to new players like Neo1973 too). So, it's time (yes, finally now, if not long ago)
to leave machine-level feudalism behind, and bring all Linux PDA/phone communities together, to work on common aims. And OE is well established community crossroad project, so implementation-wise, that would mean making sure that any more or less developed device is available in OE, and that any newcomer for a questions "Is X supported?" or "Where can I find stuff for X?" could be answered "Use OpenEmbedded." Yes, right that - use OpenEmbedded, period. Not even "You can use OpenEmbedded, just copy one to another one, and tweak this and that".

Linux installers bumbo-jumbo

On Monday, I decided, as a kind of change-activity rest, to review the situation on Linux installers market. I had closer look at what big players use, as well as googled and searched SF.net. Well, a generic, truly modular, lighweight and elaborated installer doesn't seem to exist. So, if we speak reuse, then considering an installer of specific distro is apparently the way to go. And big players are:

• RedHat systems with Anaconda, written in Python (PyGTK) but with bunch of C code too. Seemed like nice design, I even checked out CVS repo, and tried to make it show me some dialog without complete build. Nah, written in Python doesn't automatically mean it's written nice. Building it also didn't work - it depends on bunch of RH-authored libs, which don't even packaged in Gentoo for example.
  
• Debian with Debian Installer. This is apparently largely written in C, as one of the design goals was to support all architecture set of Debian, in particular lowy ARMs. The default version is text-mode, but recently(?) they got also GUI version (GTK+, and they build it against DirectFB). It is also said to be modular and reusing other Debian components, like debconf, and indeed, looking at the SVN repo from bird's view in browser was scary enough that I didn't proceed to specifics.
• Ubuntu's Express-Espresso-Ubiquity. Reuse at run. Uses Debian Installer as the backend, with (apparently Python) frontend.
• Gentoo Installer. This one will be finished when it will be finished, plus also I managed to spot bunch of UML diagrams but not the language it's written in.

Not very bright results. And thinking of it, the only thing in installer which calls for reusing some existing implementation is disk partitioning (and yes, PDAs do have secondary storage, and most users want to install Linux there (even though most of those most may not yet know that)). The rest of installer's work is pretty trivial and could be coded in Python in a day or two.

Well, ground-level solution is just present users with generic GUI partitioning app, and let them do partitioning themselves, right? So, I decided to quickly try how GParted runs on a common PDA, and that's all subject for another post.

Driver for ACX100/PCMCIA

Thanks to Fabrice Crohas, htcblueangel maintainer, who have been tackling at acx_cs driver since we were able to achieve first WiFi ping on h4000 and htcbluenagle with the help of devmem2, the PCMCIA support for ACX100 WiFi driver now works! It took quite some time to figure out what issue with PCMCIA was and make it work after first slavemem-connected device (hx4700) had first ping, but finally we're there - ACX is support on the great majority of devices which have it (though it is still has something in store, as couple of newest device have it connected via SDIO).

Working WiFi also means that all hardware on h4000 is supported now, and it has A- support status per HH.org classificaion, with the only missing feature being a native flash bootloader for Linux. Of course, not everything works great yet, or even fully tested, like flash write at all (but it is known to work for h6300 which has the same DiskOnChip and uses the same driver). ACX itself for one needs lot of work to make it support suspend/resume properly and PM in general.

But all those are just things to test/bugs to fix, and not white spots in hardware information/support. We're past age of "Linux will never be able to support that device completely" for another bunch of devices, and the future is bright! ;-)

First steps at making OPIE fit well with OE filesystem structure

One of the requirement set for OPIE to be supported for Angstrom was compliance with standard GNU/Linux installation location ($prefix and friends). There was bug in OE long enough for this, with few developers considering how to proceed with this.

I finally found time to test some ideas from it, and initial conversion from /opt/QtPalmtop to /usr went pretty smooth requiring besides rebuild few fixes to packages where the former path was harcoded in the literal form. Resulting opie-image booted in QEMU quite well, so I went with committing it, bumping snaphost date for OPIE 1.2.3-pre to 20070702 at the same time.

Hope other developers will help with testing it, and then the conversion needs to be fixed by moving data and plugin to appropriate locations (now they live at places like /usr/pics). This would take a bit more effort exactly because of this dichotomy - data vs plugins, with first should be living under /usr/share, and the latter (ideally) under /usr/lib. I wonder, if a solution like making QTDIR, etc. environment variables to accept list of paths is good enough (well, it is generic from system design point of view, but adds extra runtime overhead).