More Angstrom LiveRamdisks

Last week, I spent some time to elaborate and automate LiveRamdisks production with the aim to mass-produce them from now on. Taking into account previous agreement with Angstrom maintainers to host them at the main Angstrom download area, this means that eventually there will be a user-friendly way to try a recent Angstrom snapshot on any supported PocketPC/WindowsMobile device.

With these new scripts, Angstrom x11 LiveRamdisks for the following devices were generated and uploaded:

• h2200
• h3900
• h4000
• h5000
  
• hx4700
  

For h4000 and hx4700, also OPIE LiveRamdisks were uploaded.

Current problem with this is that uploading images in different formats (and LiveRamdisk is just another such format of course) means trafficking the same data over the net again and again, and I don't have such a big pipe. So, instead, I created a set of image conversion tools which would allow to upload master image format (apparently, tar.bz2) and convert it to other formats and make LiveRamdisk out of it right on the Angstrom hosting service

Right now, I'm building current Angstrom snapshosts, and going to try these tools on them...

Back from hiatus

Having spent August on vacation and later working with my GSoC student, Maria Zabolotnaya, I now gradually come back to speed on embedded stuff, even though I likely won't have as much time to work on this as before.

The first aim is to get OE build again, which is "of course" broken after being away for some time and then coming back ;-). Then, process all the backlog of HH.org kernel patches and cleanup tasks. Further aims will be to automate process as much as possible (for example, I finally want to setup local Continuous Integration for OE builds, like I did for HH.org kernel), and upgrade HH.org kernel to 2.6.22 (though to that time 2.6.23 likely will be out).

Finally out of the list of pressing OE issues

With the last changes to properly support modprobe 2.6 for kernel module configuration directive, I can't remember other pressing issues with OE which would prevent its proper and flexible usage for distro engineering. Among the other infrastructure changes done within the last half-year with I was involved are:

• Support for building efficient initrd/initramfs images
  
• Flexible and extensible initramfs to boot rootfs from different media
• Adding support for machine screen properties
• Image building speed improvements
• Alleviating maintenance effort for kernel porting projects with lots of supported devices (using defconfigman)

With all these and other changes, era of grunt work on OE should be over, and finally more effort can be dedicated to more sensible work - Angstrom distro engineering and software/package porting and maintenance.

Modular initramfs for OE

While all new initramfs-related features are yet need to be described, they are already put to work. Alex Osborne of Hack'n'Dev created a modular initramfs system to serve all the device booting needs. We discussed design principles and came to following tentative picture:

• initramfs in question handles just the task of mounting rootfs and booting from it.
• Doing so, it tries to follow both existing kernel behavior and interface, like command line parameters. For example, NFS boot uses standard root=/dev/nfs and nfsroot= parameters. Where extended feature is added, it tries to mimic existing parameters. For example, loop device boot uses root=/dev/loop and looproot=.
• Support for different boot methods is implemented by modules with more or less well defined interface. It's possible to construct initramfs images with the given feature by combining modules.
• Generally, new and extra features are expected to be added using modules, to keep design clean.

It is committed by now. Both loop and NFS boot modules are tested to work. I hope that Alex will describe loop module command line syntax in more detail in his blog.

WANTED: modules to boot via WiFi, with WEP and WAP support, and to boot via GPRS ;-D

Initrd and initramfs with OE: snippet 1

One of the former weak areas of OpenEmbedded was creation of small, fine-tuned, (hand)optimized initrd and initramfs images. However, during few last months, the situation vastly improved, and by now OE offers all tools needed to create initrd/initramfs per your purposed and requirements. I would like to describe them, and as that would be quite a bit of contents, going to do it describing features one by one, in "snippets".

The first is:

Building initramfs with OE and everything you wanted to know about initramfs but didn't bother to figure out yourself.

IMAGE_FSTYPES = "cpio.gz"

Yes, that's all what you need to create an initramfs image, voila. After that, you can use resulting image instead of conventional initrd (in ext2 or some other format). For the kernel to be able to boot it, you need to have CONFIG_BLK_DEV_INITRD=y in its config (contrary to its name, this option actually means "support external initrd *or* initramfs images"). As initramfs support is unconditionally builtin into kernel, you don't need any filesystem enabled for initramfs support.

(Note that above is written for standalone initramfs image which is used in initrd-like manner. Support for initramfs which are linked into the kernel image so far is not provided OE, but would be pretty easy to add. Still, it's of more interest to vendors which work on specific products than OpenSource distros. By the way, mentioned config option is not required for builtin initramfs.)

Note that execution of initramfs starts with /init. If you want to rebuild initrd (or otherwise conventional rootfs image) as initramfs, you'll need to pass following parameter on kernel command line to have it started properly: "rdinit=/sbin/init".

Final note is that contrary to popular belief, initramfs does not completely covers and supersedes initrd functionally. Initramfs has relatively big memory requirements, which precludes usage of big initramfs images on low-memory systems. Specifically, initramfs is compressed, and needs to be uncompressed during startup, which already limits its size to at most half of the available RAM. But actually, ramfs, which is used as backing store for initramfs content, is pretty inefficient for such type of content (static), and that imposes even more limits. Also, my experiments show that kernel has poor error handling while uncompressing initramfs, and out of memory condition makes it just hang. So, when you need to fill memory up with lots of content, nothing would replace an initrd with builtin compression like cramfs or squashfs. Granted, that's rather adhoc need.

Finally a kernel hackers' initrd image in OE!

When I only started with Linux-on-PDA hacking, I, as people usually do, combed internet for any kernel/initrd images which were marked as being for the PDA model I had at that time. In such way I picked up few initrd's, and after practically comparing among themselves, selected one for standard hacking use, like kernel testing. A bit later I augmented it by installing some missing stuff (like mc), and seeing that many people are not so thorough and searching the net, put it up for others' use. That's basically what I've been using since them and suggesting to others who needed environment to test some kernel feature with minimal userspace involvement.

Of course, that situation was rather ridiculous - using some random binary-only image instead of OE-built is not perfect at all, and I was just hoping that someone else would go for that. Well, with the latest changes in OE to facilitate creating efficient initrd/initramfs images, I finally got to this myself. So, here's now such image, codenamed 'devimage'. Consequently, recipe to build it as a complete image is called 'devimage-image.bb'.

Its purpose defined as: provide environment which by default runs only minimal userspace (busybox sh and dropbear now, to support different connection methods), but offers all needed "kernel userspace" utilities from udev to specific utils for ALSA, IrDA, Bluetooth, WiFi, etc.

At the current stage it's not as complete as that, so help with filling it right is welcome (caveat: size should be kept in mind).

It's all firefighting, right?

Well, time for a nice "depressive" post ;-).

So, at the beginning of week I decided to port GParted. Was more or less easy, just a fix for an obvious thinko from the author regarding C++ std namespace ;-). However, it failed on linking stage with infamous rpl_malloc() and friends not resolved for libparted. Looking thru cofigure log, that's of course result of crosscompiling assumptions, but OE has defines for the corresponding tests in autoconf site files, so what was actually happening is shipped parted 1.8.7's configure not loading these site files.

It also turned out that autoreconf'ing of this parted version doesn't work, as it requires autoconf 2.61 and friends as minimal, while OE has 2.59. Trying autoreconfing 2.61 (which is DEFAULT_PREFERENCE=-1 in OE) lead to the same issue with site files loading. At this point I submitted bugs on this, and raised issue on #oe. Actually I didn't even expect immediate big attention, but it caught developers' eye, and with the help of XorA and hrw, to the end of the day we had a solution. (And the root of issue was bug in autoconf 2.61. They used "shell programming patterns" (cough) you really don't want to use in your scripts, and the result is proverbial - they outslied themselves).

Well, on the droop of day, GParted was built. I run in QEMU and was surprised that UI was pretty barely usable on QVGA. Run on a real device to try a real card brought a gore however. First of all, GParted didn't detected my SD card at all. I even went looking at GParted source for how it scans devices, but then run parted on it. "Overlapping partitions now supported" was abort message. Overlapping partitions? Me? No! Wrong call! Fortunately, running fdisk showed instead of the senseless bold accusation the fact that my camera indeed tried to dig weird holes in parttable, apparently confused with ext2 partition there ;-)

Well, finally GParted was run, bliss. But just going thru menus once made them no longer accessible. Already trained on such stuff with LiveRamdisks work, I knew it's out of memory. Running ps aux showed some unusual suspects. Killing them (btw, GParted's RSS was "merely" 15Mb comparing to those 6Mb's. Ah, it's all shared libraries, ok, ok, I shut up.), I finally had the thing live. Now closer look look showed it doesn't recognize EXT2 partition (at least FAT it did). The same in parted. Have no idea is it camera hosed it or it's just proverbial GNU software quality. It also turned out that the only operation I was able to do is to change boot flag for a partition ;-). I tried to format unrecognized partition as EXT2 for example, and it failed after shuffling a card for a bit. Getting to HTML report it laid in the home dir surfaced "sh: nice: Command not found". Oh, oops, dear app ;-).

Finally, I wanted to make a screenshot, but gpe-scap failed to write it, complaining that it can't do UTF-8 -> ISO-8859-1 conversion for PNG CREATOR chunk (soft laugh).

---

So, what are the conclusions? X is not an ultimate solution, but merely an option. Lure to run large body of existing, and updated, software is still a lure, as besides this opportunity X brings bloat, which on resource-constrained system vice-versa inhibits ability to run all that wealth of soft well.

Also, all those kernel-hacking games are really old. And I hope that everyone on the block understands by now that what they get once kernel is well and polished is havoc in userspace (this applies to new players like Neo1973 too). So, it's time (yes, finally now, if not long ago)
to leave machine-level feudalism behind, and bring all Linux PDA/phone communities together, to work on common aims. And OE is well established community crossroad project, so implementation-wise, that would mean making sure that any more or less developed device is available in OE, and that any newcomer for a questions "Is X supported?" or "Where can I find stuff for X?" could be answered "Use OpenEmbedded." Yes, right that - use OpenEmbedded, period. Not even "You can use OpenEmbedded, just copy one to another one, and tweak this and that".