Sunday, April 5, 2015

Interfacing user code with MTK MAUI OS – Simcom EmbeddedAT vs Mediatek Labs LinkIt ONE approach

[This post is dedicated to Fernvale project]

(Intro skipped for now)

EmbeddedAT defines API in terms of function pointers. Locations of these function pointers are provided in symbol file (.sym) supplied to ARM RealView linker during user app linking. On OS firmware side, these function pointers are packed together in a section which then appear towards beginning of Flash, and whose address are thus relatively stable even if firmware is updated. Flash map is specifically:


COREAPI is the API function pointer table (thunk table, vtable). “app” is user application.

LinkIt ONE uses more elaborated scheme which emulates protected-OS syscall approach.

First of all, these “syscalls” happen via message passing between user and OS tasks (aka threads).
Here's typical call from user-facing API function:

LTask.remoteCall(linkit_sms_delete_handler, (void*)_msgId);

remoteCall is implemented as (some code skipped):

void _LTaskClass::remoteCall(remote_call_ptr func, void* userdata)
m_msg.remote_func = (remote_call_ptr)func;
m_msg.userdata = userdata;
m_msg.signal = m_signal;
sendMsg(VM_MSG_ARDUINO_CALL, &m_msg);

So, it packs reference to remote function and its args into a message and sends it to a remote execution task, then wait it to signal completion. This all happens synchronously, protected by a mutex. Note that remote function is a real function identified by address, not a syscall number. It is defined as:

boolean linkit_sms_delete_handler(void* userdata)
VMUINT16 msg_id = (VMUINT16)((VMUINT32)userdata);
vm_sms_delete_msg(msg_id, linkit_sms_delete_callback, userdata);
return false;

I.e. it calls similarly named vm_* function. We saw another vm_* function already - vm_signal_wait. Let's see how that is implemented. The source code is actually not provided, that function is precompled in libmtk.a:vmthread.o. But looking at that object file, there's very little magic: it defines bunch of vm_* functions, and bunch of corresponding _vm_* variables, and has one outside function call: vm_get_sym_entry(), it also has strings matching vm_* function names. One can speculate that each of vm_* functions defined there, take a corresponding string, pass to vm_get_sym_entry() to look up address of that symbol, then call that address, caching it in _vm vars for next time.

Grepping source, vm_get_sym_entry turns out to be not a function proper, but a function pointer. And it gets initialized from a value passed to function:

void gcc_entry(unsigned int entry, unsigned int init_array_start, unsigned int count)
vm_get_sym_entry = (vm_get_sym_entry_t)entry;

And gcc_entry is marked as an ELF entry point. So, the overall structure is: user app is compiled to an ELF object (note that actual storage format may be different). When initialized, its entry point gets called with reference to vm_get_sym_entry() API function. Using that function, all other API functions can be looked up. Wrapped in few more layers, we arrive to Arduino API which sketches call.

Actual application binary storage format appears to be “vxp”, which is actually ELF with some stuff (“tag”) appended at the end by:

tools/mtk/PackTag.exe sketch_apr05a.cpp.elf

Saturday, October 19, 2013

Announcing libperipha and cortex-uni-startup projects

More and more ARM Cortex based boards appear for use by OpenSource/OpenHardware/Maker communities, and
frequently asked question in response to a new announcement is often "Is this supported by open-source toolchain?". Many people know that gcc supports ARM, including Cortex-M, very well, and many projects now lean towards using as the package of their choice.

But compiler is only part of the toolchain - there should be also at
least basic hardware definition headers which would allow to access peripherals of particular chip. There was big step in that direction done by ARM Ltd, itself, by releasing CMSIS library for the Cortex-M CPU cores under open-source license, though that didn't go without controversy, see comments to that post and followup.

That's great step, but again, it's only part of the story - while it allows to access Cortex-M core registers which are common across all MCUs, there're more to MCU than just CPU - it's all the peripheral blocks. Even basic things as GPIO and timers are vendor- and model-specific, and periphery is what differentiates one vendor's  offering from another's.

And other vendors don't rush to follow ARM's example, so few vendor-specific CMSIS packages are under OpenSource license (one great example of the contrary is Energy Micro EFM32 CMSIS - it's great because they open-sourced it even before ARM). One  good approach to use OpenSource-friendly MCUs for your projects, and everyone is encouraged to do so. Unfortunately, that may not be practical or even possible (I wish I could by EFM32 chips in my part of the world).

So, the idea is to provide open-source MCU headers for use in OpenSource projects - until all vendors get smart to provide them. That's how libperipha project was born. Note that it's not limited to ARM Cortex-M MCUs, though support for them was a primary motivation.

There's final, small, but just as important piece which construes MCU toolchain - startup code. Without it, your application simply won't run. Another project, cortex-uni-startup (unified Cortex startup) was created to cover that. As excellent article by Ivan Sergeev explains, Cortex-M was specifically re-engineered to allow startup code to be written without any single line of assembly! cortex-uni-startup explands on this to provide easily reusable Cortex-M core startup code (usable on any Cortex-M device, but without support for peripheral-specific interrupts), then extends it to cover interrupts of particular MCU model.

These two projects are separate to cover differences in scope (cortex-uni-startup is Cortex-M specific, libperipha is unbound), licensing (cortex-uni-startup core is public domain, libperipha is BSD and others), and usage (libperipha is gross database which you probably will just want to use, then hack, cortex-uni-startup is nice learning tool of beauty of Cortex-M architecture, which you welcome to glance over before using - you won't be overwhelmed, as it's small).

Last final note - while these projects are conceived to be comprehensive, I don't have an aim to sit and add each and every possible bit of every MCU there. Instead, it is intended to be easy do add missing pieces to it on demand - right while you're working on some applied project and find such missing pieces. Then, if people find this to be indeed so, and project overall useful and contribute, then they indeed will acquire comprehensive coverage of MCUs. So, have a look, give it a try, shared feedback, spread the word, contribute patches, and overall, enjoy ;-).

Sunday, October 6, 2013

Growing GPL/FOSS Loyalty Among Chinese ARM SoC Vendors

There are visible changes in how Chinese ARM SoC vendors deal with Open Source license compliance (words) and Open Source software at all (spirit) during this year. Not only western resellers of devices based on Chinese SoCs release GPL compliance packages, but many popular SoC companies themselves established GPL compliance/loyalty/support pages. Few go beyond just adhering to word of GPL by releasing kernel source blobs (by blobs I means large tarballs devoid of any development history and support files), but also establish git repositories, release Android trees, set up public bugtrackers.

Of course, they are still not exactly GPL compliant - scrutiny of any releases would likely show missing parts, binary objects for drivers, etc. But there's definitely a difference between plucking out WiFi driver out of flash image you have on you device and downloading "official binary" from a SoC vendors. It's also a good start, and there's a hope the situation with complete source availability will improve. And as everyone watching the area knows, the industry at the whole needs to improve - Western companies may comply with GPL by providing source for kernel shims, but that doesn't help users to support vendors' hardware if they just move binary blobs to userspace.

The changes discussed didn't come by themselves and at once. Lot of people paved the road to this via, via various leaks, etc. But critical change came with emergence of Open Hardware startups. They are led by people who strongly believe in Open paradigm, but also to do their ODM-like business, they need to provide sufficient documentation/sources to the customers. So they have little choice but request that from upstream SoC vendors, in redistributable form. So, thanks Rhombus-Tech, Olimex, Cubitech!

Anyway, here're the links to Chinese companies' GPL loyalty pages I know so far. Additions and corrections are welcome.
And as extra, community projects to support some of vendor SoCs from above list:

Monday, August 12, 2013

Braindead X11 Clipboard Exposed

I've been using Linux with X11 as GUI system as my primary and only work system for like 6 years now. It's great, much better than another world - asymptotically.  But there're few issues which sustain - an  being hated - for years. One of them is how clipboard in X11 works. It works illogically, unpredictably, and non-reliably. Based on many years' background black-box analysis, there appears to be more than one clipboard buffer, and some apps apparently take clipboard with themselves when they quit.

It finally came a right moment to both be hit by it and have time to investigate. This blog post shares the same sentiment as I above, and confirms suspected design braindeadness.

To quote:

Three selections are defined in the ICCCM: CLIPBOARD, PRIMARY, and SECONDARY, each of which behaves like a clipboard in its own right:
  • CLIPBOARD: traditionally used when text is copied and pasted from the edit menu, or via the Ctrl+C and Ctrl+V shortcuts in applications that support them.
  • PRIMARY: traditionally used when a mouse selection is made, and pasted with middle-click or Shift-Insert.
  • SECONDARY: ill-defined secondary selection. Most applications don't use it.

Be gone soon, X11, be replaced by something sane.

Wednesday, July 17, 2013

Ubuntu Spyware

On few occasions I noticed that some "zeitgeist" daemon running on my Ubuntu 12.04 system takes a lot of CPU cycles, so it had to be killed. Today, clearing off my disk, I also noticed that it eats noticeable chunk of it (100MB+). I proceeded to investigate what Zeitgeist is, finally. Here some links and comments:

1. Tutorial on GNOME Activity Journal, Zeitgeist and Sezen describes it as a universal user's interaction logging system.
2. I was never asked if I want to install or enable such system and/or logging during Ubuntu install/upgrade.
3. It turned out that daemon was installed and running automatically, but frontend app (gnome-activity-journal) wasn't installed.

So, this thing collected my activity without my consent and behind my back, and clearly not for the purpose of letting me use that data. This all pretty matches the definition of trojan spyware. In addition to that it also eats system resources and makes system running slow.

Few people share the concern that Zeitgeist is exactly spyware and not some productivity tool, future of Linux user interface and other marketing bullshit from Gnome3 and Canonical:
How to remove Zeitgeist:

Overall situation with spyware on Ubuntu, from Richard Stallman: .

Friday, June 28, 2013

Links for June 2013

Sunday, June 9, 2013

Buffalo SkyStation CWR-GN150S Teardown

Walking last few hours thru Hong Kong streets and molls during last visit, I was looking for some impulsive shopping, and bought micro-router having following labels: Buffalo SkyStation CWR-GN150S, Hornington NE-GN150S, etc. What lured me was its really small size (5.5x4.5x1.7cm) and 2 Ethernet ports. It was also couple of bucks cheaper than TL-WR703N. I was thinking where's the catch, and later got it - it doesn't have USB host port (micro USB it has is only for power).

All manual for the router is in Japanese/Chinese, same for firmware, no English firmware on Internets, actually, barely any English pages at all. Teardown was even more disappointing: it has only 16MB of RAM (bye, OpenWRT), and turned out it uses RTL8196C SoC, infamous for being of MIPS-Lexra architecture, which is MIPS subarch crippled due to patent trolling between MIPS vendors. There's barely any support for this subarch from gcc and OpenSource projects, I found just one brave guy who tries to change the situation:
 Well, I added hacking on all this stuff to my (very long) queue. In the meantime, just posting this to let other folks to google it up.