Dynamically loading code

[Folknology]

Wonder if you guys could help me here as this requires much more expertise than I currently possess about the XS1 operation and compiler etc..

I would like to be able to dynamically load modules as part of the Amino Project, this would provide a number of advantages including better use of the limited SRAM and a more modular approach to problem solving.

When I say modules I mean code of course, right now the compiler compiles a single binary designed to run on the core/s. I would like to run management code that can dynamically load code modules conforming to an agreed lifecycle interface and unload them when complete or requested.

As such the lifecycle interface would likely contain the following rough sort of API:

configure(struct/arrays)
init()
start(context/channels)
stop()
terminate()

Configure passes in any configuration parameters, Init initialises the module, start runs the modules main thread with a given context and and at least 1 channel, stop of course asks the modules to stop its tasks and termination would be expect the module to free any used resources.

The question is how does one go about doing this as I have found zero documentation in the standard Xmos documents as to how this could be achieved. I have however found this interesting thread on Xlinkers which opens up some of the key issues albeit in a less structured topic.

http://www.xmoslinkers.org/forum/viewto ... ?f=3&t=163

So I would like to really like to get the ball rolling on this topic and see if we can produce a working example, I would therefore love your feedback, ideas and any examples around this concept..

[Berni]

Id imagine you could write some asm rutine that reads the code byte by byte and writes it to the correct RAM locations. and then just do a goto to its starting location. Altho for what you are doing you also need the code to be able to run from different locations than that single fixed one. In that case you would need to modify any gotos and other address sensitive commands to work correctly.

Meaby its possible to make a large variable array in XC and then use inline asm to direct the program counter in to that array.

I would love to see some of this working.

[lilltroll]

This document was recently updated:
http://www.xmos.com/system/files/xdkqs_1v1.pdf

Check out page 8/9, what about getting that loading code from XMOS ??

[Folknology]

HI Mika

I hadn't looked at the XDK but as you say it might hold some clues. I am not sure however if it is loading components/modules or rather resetting itself and reloading an entire program.

I will need to look more closely at the code
this is also interesting wondering if it also has anything similar inside:
http://archive.xmoslinkers.org/node/319

Keep coming up with ideas we will get there

regards
Al

[russf]

Just spotted this, and will comment, though it's possible you are already far down this road...

These are generalities...

It's likely you want to make 'position independent code', that branches to offsets from a base pointer. There should be the fewest possible absolute locations. If you achieve this, the code is 'relocatable'. The task of preparing code to run in an OS includes a 'fixup' pass, that adjusts absolute pointers, among other things.

I would consider creating variables by allocating slots in a table. If you made this a hashtable, you'd more easily be able to handle scoped variables, but this is already very deep into compiler/machine architecture.

If your code speaks to a shared resource (any absolute location, or shared data structure) then you'll have contention issues, so access to the environment should be mediated by a library that virtualizes all shared resources. This is also a Master's thesis ;)

Had you considered creating/using a 'little language' for this task? There are many interpreted languages, and one of the features in some of them is the ability to restrict access to shared resources. This is essential for any shared compute engine, and of course there are speed penalties, but it has a big upside in being much easier to debug than building a dynamic compiler/linker/loader.

Good luck,
--r

[monk_is_batman]

I actually just completed a system that sounds quite a bit like what you are talking about. To do it I would compile the code in the xde and then disassemble it with xobjdump and run it through a parser written in perl.

The key to relocating the code is including the constant pool and space for the data array. You can find the locations of both of these near the beginning of the disassembly, they load and set values for the the CP and the DP and from what i can tell about programs compiled in xde these do not change from the rest of the program.

So what I did to reproduce this was to save the entire section of code from the first function used to the last element in the data array to flash, then when I want to run it i load the entire thing from flash and copy it into some allocated space. Then there was some assembly required to set the CP and DP to their relative locations, and allocate space for the stack. Lastly as mentioned above you use a sort of goto to get to the code, what I did was spawn a thread and have that thread execute a BAU instruction.

Hopefully this information can help you figure out how you want to go about doing it.

[Folknology]

Thanks for the feedback monk_is_batman, I'm suitably impressed by your Xmos kung fu!

You have obviously done your homework and have come up with a great hack to show it's possible!

I was lead to believe the code was relocatable and you have proved it. It would be helpful if Xmos could come back on this thread and offer a more official method for doing this, perhaps a command line option on the build tools to create special elf files or something.

Anyone at Xmos listening to this thread, it seems like it's something the toolset could easily handle given monk_is_batman's manual labour? It also helps with the memory issues being able to dynamically load code modules from flash.

[monk_is_batman]

I can't speak for what XMOS will do but eventually I hope to have this into an easy process. This is going to be a part of my OS project for the XMOS and if it turns out well it will be released and have a nice automated makefile for use of making modules that can be stopped and started easily (installing perl may be required).

[lilltroll]

"dynamically load code modules from flash."

Just tell me when you are finished with the possibility to dynamically read modules and other data over a 4-pin 100 Mbit SDCard link! :ugeek: :ugeek: :ugeek:

[Berni]

btw you cant get that kind of speed in SPI mode, you need to figure out that 4bit bus thing to do it. Documentation is kind of sloppy on that interface but i guess it could be done. Altho you should be able to get 50Mbit/s using SPI on the so called high speed cards(x133 and so on) To get 50Mhz SPI you probably need to switch the i/o clock in the xn file and use the serializer feature of a pin. But be sure to check the registers what kind of card is it during initialization before switching to the high speed clock( Also the socket compatible MMC cards have yet again different clocking specs.)

Also one thing thats going to be slowing you down is the FAT file system as for every sector it has to check if its fragmented and where the fragment is located.