Out of Hanwell

August 16, 2008

Khoros: A Lua Operating System

Filed under: Development, Linux, My Projects — Tags: , , , , — Matthias Miller @ 4:06 pm

Over the past year or two, I’ve been tinkering with my own operating system, which I’ve dubbed Khoros. The name comes from the Greek word χορός, meaning dance, from which we get our English words chorus and choreography. Of course, I chose this name because of its obvious play on words.

My first task was simply getting a bootable system. Rather than write my own bootloader, I used Grub. I based my work off Brandon Friesen’s excellent tutorial. I initially used VMware for virtualization, but since then, I’ve since switched VirtualBox and have had much better success.

As much as possible, the kernel is written in Lua. (Yes, Microsoft’s Singularity project caught my attention.) I am not particularly gung-ho over Lua as a language, but I picked it for its exceptionally small footprint, and it opens interesting possibilities for sandboxing. For example, my C kernel interfaces high-level interfaces to allocate and manipulate memory. However, the Lua scripts cannot specifically address memory. They can only manipulate memory block within the allocated bounds. This allows very powerful but safe low-level programming for my drivers.

With the past several weeks, I finished up basic multi-process support using Lua’s coroutines and debug hooks. (I am not using traditional process boundaries, so everything is running in kernel mode.) Processes can communicate with each other using a very simple IPC mechanism.

Today, my operating system boots into a simple Lua shell, allowing me to run arbitrary Lua code from the command line. This is no small feat, but it is admittedly of limited use. I am trying to decide how to proceed.

A networking stack could be quite beneficial, since it would allow me to program it remotely or to do something interesting like serve websites. However, this involves learning about the PCI bus, the NIC, ethernet, TCP/IP, DHCP, and many more details of which I am certainly unaware.

Alternatively, I could implement rudimentary file system support. For example, I could provide a way to save Lua objects to disk. However, I am lost to what interesting piece this might lead me to.

I blog so infrequently that I’ve probably lost most of my readership, but to all of you out there, what do you think I should develop next?

Advertisements

18 Comments »

  1. Even if you loose all your anonymous readers you can get new ones – casual readers – who catch an interest in your writings. Here I caught interest because of the report on “creating a new operating system”.

    I think that there is room for another operating system today even if Linux is very scalable, Free/OpenBSD is stable and secure and Microsoft XP-Vista has a lot of proprietary drivers.

    So I caught interest and then you set me off studying Lua, which is a very nice concept. Now I am going to find out whether it is another Perl (whic evolves to muddy syntax?) or gawk, which is beautiful and fast but may be a little limited in its interface to other languages(?).

    Go on! Tell more,
    regards/Donald Axel
    http://d-axel.dk/
    http://dax2.wordpress.com/

    Comment by dax2 — August 17, 2008 @ 3:50 am

  2. Interesting concept. I’ve often considered doing my own OS, possibly based on SLang, as well, but lack of time and/or motivation has prevented it.

    Lua is an interesting choice of languages, since AFAIK it was designed strictly as an embedded scripting language. So my guess is you have a very small core that just starts up Lua, from which you do all your real work?

    @dax2, AFAIK (I haven’t worked with it at all, only know what little I’ve read), Lua is quite an elegant language which doesn’t devolve like Perl. But it was designed to be a scripting language to be embedded in applications. The biggest use of I know of is in the MySQL Proxy project, which uses Lua to allow load balancing, modification of queries, issuing of extra queries, detailed query logging, and much much more. Which means it’s gotta be fast.

    BTW, WRT regular readers – I’m one. Through the magic of RSS I’m still here 🙂

    Michael

    Comment by Michael Johnson — August 19, 2008 @ 1:23 pm

  3. Michael, you’re right. The C core interfaces a “c_kernel” object with functions such as in8, out8, etc. (For example, to reboot: “c_kernel.out8(100, 254)”.) The C core then launches a core Lua script that is responsible for loading the process/task manager and all necessary processes, such as tty, shell, etc.

    Each process is essentially a script that is loaded into a coroutine that is sandboxed via setfenv.

    -Matthias

    Comment by Matthias Miller — August 19, 2008 @ 1:54 pm

  4. Yes Lua is interesting and I too am ambivalent, Lua is superb and horrible such as poor documentation omitting vital information. A lot of needless frustration results.
    (just gone through hell over something undocumented, for-goodness-sake don’t just say it returns a thing, what exactly does it return because it doesn’t do what is implied. Turned out to be broken, missing release resource and a different not so recommended way does work but only if other things not mentioned are done, logical and easy once you know) The Lua code is very good, just hard to use.

    ANSI C was a very good move and some libs are portable too.

    Bringing up say IBM clone might not be so hard and there are examples of amateur OS.

    My historic background is embedded systems where in effect your code is all there is and might be a whole language. Never done this with massive systems though.

    Lua is large, not small in the embedded context. It will however compile straight and just about run under DOS 16, done as an experiement and just needed careful compiler and linker use.

    There is an enormous amount I could say but leaving out a lot of justifications and reasons why…

    Either you accept Unix and do it all in Unix of some kind making Unix of some kind or accept reality and take the DOS route. Few Unix people ever understand what DOS is/was, wrong thinking.
    It’s actually very clever and powerful, done in a small space across a lot of hardware, and simple, KISS principle. It is a personal computer not a holy mainframe, live with it.

    One way is this.

    Find a simple old computer, good IBM clone with IDE and straghtforward graphics, well known network card. That is your working system.

    Bring that up using FreeDOS. You can also see if you can get wattcp working on the network card as a precursor for later use. (I could send you a copy of Lua which will run under DOS but is of very restricted capability, too litte stacks space, did run factorial.lua first time though, not useful only a proof of concept)

    Back in the 1980s I did several things which here give some clues.
    I ported a specialist Forth from CPM/80 to DOS. This was written (by me) in assembler. Made some minor changes and passed the source though an American Automation cross assembler (cost a lot of money). Because i86 and 8080/Z80 are very similar the result was a DOS executable.

    This ran first time complete with disk access. Understand, CPM/80 and DOS and to at least XP there are a common set of BIOS calls, (first 16 I think) which are identical. Even the DTA is in both (disk transfer area).

    Another thing I did was write the first 32 bit floating point Forth. This was in 68k assembler and aimed at embedded systems but ran on personal computers too.

    Essentially there is a relocatable chunk which can be loaded into memory by a loader, go. (or work out of eprom etc.) At a definite place is a jump table, these are the hooks to the system.
    Things like initialise, where is memory, readkey, write char, graphics… and so on, all basic and only a very few are needed to have a text system. The correct OS addresses are set there and off you go.
    Some very fundamental code has to be somewhere BIOS is part which is unique to that particular hardware, not a lot.

    The next stage I will mention is perhaps look at Open Watcom, ex full commerical and a very good toolchain. This is actively in serious use… as a hint can target
    DOS 16, DOS 32, DOS extender, QNX, Novell, Win 16, Win 32, Linux, OS/2

    Will compile Lua and libs.

    Lua is not yet really suitable for full embedded without a formal OS, got to get rid of the C OS libraries and calls are littered all over the source. Others are I suspect working on fixing this.

    Compiling a say DOS 32 Lua and maybe it would work very usefully from FreeDOS.

    Go to DOS extender and there is megabytes available and the start of crossing the barrier between real mode and protected mode to get at the BIOS calls for things like disk access.

    wattcp should allow getting up TCP/IP, just needs a new personality source in Luasocket, currently usocket.c and wsocket.c
    I guess it would work by crossing the barrier into wattcp running in real mode. There is an API of some kind, maybe you write it.

    Graphics? Can start under DOS and go upwards from there. Libs if you know where to look. Rest assured that Linux and BSD go through hell handling graphics hardware, often needing to but can’t cross the barrier into real mode. Use VESA, life is too short to invent the lot.
    Widget lib, try FLTK. Rumour it has been used under DOS. And see murgalua.

    Then work on eliminating FreeDOS. Easiest is just accept it as a core boot loader. FreeBSD for example uses a Forth as a boot loader. Debian uses some kind of embedded VM to boot. And so on.
    That Forth is embeddable and I guess boots in effect on BIOS.

    Layers tend to be everywhere. What else is X but a layer sitting on a command line system?

    Yes Watcom can run (gui) debug over a serial port or whatever, debugging the app on your new box.

    Currently the Watcom DOS distribution is without a maintainer and is probably broken. So it might be difficult to avoid win32. Doubt that it works well enough under Linux yet to be useful.

    I recently released a simple tool in C which helps in making static linked Lua, including Lua scripts. This might be useful before you have disk working (but feeding source via a terminal channel is old hat in the embedded world) Lua in fact uses this simple streaming of basic channels, io. does just that.

    We were writing dll for DOS way back so that is no big deal.

    Maybe I misunderstand.

    You got my email address.

    Tim

    Comment by Tim C — August 24, 2008 @ 4:54 pm

  5. The idea of writing an OS in Lua is great. I’m currently thinking about creating a project on a “higher level” than an OS. It would be a mixture of middleware and application. The first step would be to have the project working on a “host OS”. But having a kernel in Lua would make it possible to exchange data between both. A fascinating idea.
    I subscribed to your blog and would be happy about some updates on your project.
    Personally, I think I would implement some filesystem functionality to be able to read and write data. I’m still a big fan of the Apple Newtons way of storing data: in a database. Having a system that relies on DBs instead of filesystems.
    As Tim C mentioned above using some parts of C code from other projects could be a good idea. Lua is great for interfacing with C code, and FreeDOS http://www.freedos.org is the right place to look at.

    Just drop me a mail if you like.

    Christian

    Comment by Christian — October 7, 2008 @ 11:35 am

  6. This is interesting. I was just reading a blurb about the benefits of Lua for constrained platforms like cell phones, linked to from Lua’s about page. It sounds like your project might be just the thing for embedded platforms. The idea just came to me from the conjunction of your article and the aforementioned blurb. So I haven’t pondered it in depth, but you could dispense with a lot of the baggage associated with the PC hardware architecture.

    Also is your OS bootable on something like Virtualbox?

    Comment by Ken Yap — November 22, 2008 @ 3:08 pm

  7. Ken, yes, I’ve been using VirtualBox for development (except I haven’t gotten VirtualBox running on my Eee yet).

    Comment by Matthias Miller — November 25, 2008 @ 12:26 pm

  8. Hi,

    since I’m actually trying to do the same (more or less), I just wanted to aks: any news on this? Is there any source code anywhere available?
    Did you know eLua?

    Cheers,

    umaxx

    Comment by umaxx — May 24, 2009 @ 5:50 pm

    • No, I haven’t had time to work on this lately. I’ve got it in my private Subversion repository. Are you interested in working on it, or are you just wanting to take a look at the code?

      I wasn’t aware of eLua, although it looks like their philosophy’s a bit different from mine, since my goal here was to keep the amount of C code to an absolute minimum.

      -Matthias

      Comment by Matthias Miller — May 24, 2009 @ 6:52 pm

  9. Would be great, if you would put your project under the GPL and release the source code to the public,I would love to take a look at the code.

    Comment by Someone — July 4, 2009 @ 6:05 pm

  10. It’s odd, I wanted to make a kernel to learn more about how a system works, but I wanted to use something that was easy to write and work with. I chose Lua so maybe I could find new, interesting ways to do things with it. A few days later, I stumble upon this and I was intrigued. I’d very much love to see the source code to use as a learning resource.

    Comment by Daniel Ferguson, Jr — November 3, 2009 @ 11:10 pm

    • The source code of khoros can now browsed through trac under: http://www.khoros.org/devel/
      There is tag/miller which contains millers original source code.
      trunk/ contains current development status.

      New developers are welcome just drop a mail to info@khoros.org if you are interested.

      Comment by umaxx — November 4, 2009 @ 11:02 am

  11. It would really be nice to have a look at the source coed, I like Lua a lot, and to see an OS written in it would be great!

    Comment by Z — November 21, 2009 @ 7:55 pm

  12. *coed should be code

    Comment by Z — November 21, 2009 @ 7:56 pm

  13. Your URL to Brandon’s tutorial is now broken. It’s now at http://www.osdever.net/bkerndev/

    Comment by Adrian — December 27, 2009 @ 9:25 am

  14. more info under http://www.khoros.org

    Comment by umaxx — August 13, 2010 @ 10:12 am

  15. Hey man I think this is great. Lua is a nice, decently clean language, and fairly easy. I play minecraft, and am in the process of building a computer. The os is the hardest part of it, but since it based off the computer at homebrewcpu.com I want it to have a good operating system. Hand coding is virtually impossible, so to get around it I got a mod called computercraft, which uses lua. Ever since then I have been looking for an os that uses lua.
    Youre great man. Any way I could get a copy of a full lua kernel? Even really basic. When I publish the computer, you will receive full credit for os.

    Comment by Griffin — April 11, 2012 @ 2:05 pm

  16. Hey, I’m unsure if you still actively use this blog, but I was wondering if you still have the Khoros source? I’m currently getting in to Lua and would love to take a look. Thanks.

    Comment by Lewis Smith — February 11, 2014 @ 9:59 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: