Feedback on build system xmake/xcommon etc

Technical questions regarding the XTC tools and programming with XMOS.
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Feedback on build system xmake/xcommon etc

Post by Ross »

Hi,

XMOS libraries and applications have typically used a build and dependency management system based on make and makefiles supplied in module_xcommon and provided with the xmos tools (xmos builds and ships an old version of make as xmake). There are some docs here: https://www.xmos.ai/documentation/XM-01 ... index.html

The dependency grabbing downloads from xmos.com directly. With a move to GitHub development these releases have not been maintained. Internal developers have also favoured cmake for a few reasons and designs have now got fragmented in terms of methodology. Some designs, such as USB Audio, have continued to use the old system while other projects, such as XCORE-VOICE use their own cmake based systems. The xcommon based system certainly does have its issues, dep grabbing only from xmos.com and maintainability being two high items high on my list.

Threads like the following show some disillusionment with the current arrangement and also some praise for the previously unified scheme: https://www.xmos.ai/documentation/XM-01 ... index.html.

My team has been tasked with again unifying the build and dep management systems and so I thought it would be prudent to ask for any input from users here. What did you like about the the old system? what don't you like? what feature would you like to see added/removed?

Thanks for your time.
User avatar
fabriceo
XCore Addict
Posts: 214
Joined: Mon Jan 08, 2018 4:14 pm

Post by fabriceo »

Hi Ross, good to hear this initiative.

when we start a new project based on an xmos application (say audio usb) we basically recreate a local repository with all the components provided and we replicate this as our own GitHub project. then we modify and add tons of things.

so from my perspective the problem of version and dependencies with xmos.com or xmos.ai or GitHub/xmos doesn't exist.
if you guys update a library then we will eventually and very carefully integrate the module(s) into our source code by our own.
because we may have created incompatibilities or because we just don't want to publish something not fully tested or validated.

in other words, we ll use your solution (xmake or cmake) as it is and then we'll work from there with our own journey.

what is important to me is that xmos continues creating libraries/module that would be compatible with xmake projects (made with all the makefile.xcomon features). This framework has demonstrated enough flexibility and robustness.
The only thing which is missing in the module_build_info is a variable that could be called at different steps of the build, especially the end, for example I have created the followings:

Code: Select all

#FINISH_All = $(call XFLASH_FACTORY,$(1)) 
FINISH_All = $(call XFLASH_UPGRADE,2,$(1))
#FINISH_All = @echo Ready to Run with XTag3 with xscope

XFLASH_UPGRADE = @echo Creating upgrade $(1) image $(2).bin && xflash --verbose --no-compression --factory-version 14.3 --upgrade $(1) $(2).xe -o $(2).bin
XFLASH_FACTORY = pwd && xflash --verbose --no-compression --boot-partition-size 0xC0000 --factory $(1).xe 
which are called by an additional line added in Makefile.common1 around line 1220 in the .xe role:

Code: Select all

$(BIN_DIR)/$(APP_NAME).xe : $(XMOSUPDATE) $(HEADER_WARNINGS) $(REAL_OBJ_FILES) $(call UNMANGLE,$(XN_SOURCE_FILES))  $(call UNMANGLE, $(XTA_SOURCE_FILES)) $(call UNMANGLE,$(LIB_FILES)) $(call UNMANGLE, $(MODULE_XSCOPE_SOURCE_FILES)) $(call UNMANGLE, $(APP_XSCOPE_SOURCE_FILES)) $(PCA_FILE) $(CURRENT_MAKEFILE) $(TARGET_DIR)/_obj.rsp | $(BIN_DIR)/
	@echo Creating $(notdir $@)
	$(AT)cd $(TARGET_DIR) && $(XCC) $(TARGET_FLAGS) $(XCC_MAP_FLAGS) $(XCC_EXTRA_MAP_FLAGS) $(LIB_FLAGS) @_obj.rsp $(call TO_OS,$(foreach x,$(XN_SOURCE_FILES),$(CMDQUOTE)$(call DOTDOT,$(call UNMANGLE_NO_ESCAPE,$x))$(CMDQUOTE))) $(call TO_OS,$(foreach x,$(XTA_SOURCE_FILES),$(CMDQUOTE)$(call UNMANGLE_NO_ESCAPE,$(call DOTDOT,$x))$(CMDQUOTE))) $(XSCOPE_ARGS) -o $(CMDQUOTE)$(call TO_OS,$(call DOTDOT,$@))$(CMDQUOTE)
	$(call FINISH_All,$(BIN_DIR)/$(APP_NAME)) $(call FINISH_$(CONFIG),$(BIN_DIR)/$(APP_NAME))
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Post by Ross »

Thanks very much for the feedback Fabriceo!
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Post by Ross »

Just to keep people in the loop, we've been working on an update to xcommon which more nicely handles dependencies from GitHub etc - the old system was limited to downloads from xmos.com. This means we can use the same systems internally for development and regression (we had an internal centralised "manifest" based system) as external users.

We've taken the opportunity to move things to cmake to make things easier to maintain. We'll be gradually be rolling this out over our libraries and reference designs, USB Audio being the first to migrate (though the existing Makefiles will be left in place). We will provide early access for evaluation purposes, it will eventually be added to xmos tools releases.
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Post by Ross »

For those that are interested the new system is called xcommon_cmake and can be found here: https://github.com/xmos/xcommon_cmake

We'll be documenting and rolling out over the next weeks.
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Post by Ross »

User avatar
vergil19
Member++
Posts: 22
Joined: Thu Jan 20, 2022 3:54 am

Post by vergil19 »

Hi Ross, it's great to hear this news. Migrating xmake to the more maintainable CMake is a wise move.

I appreciate the robustness of xmake and some features such as concurrent firmware compilation and relatively concise makefiles, which are exactly what we need.

However, the lack of usage instructions and ongoing maintenance for xmake is concerning: we've encountered several issues recently, as have our clients.

The good news is that these issues are likely to be improved in CMake: a larger community and continuous software updates. This is also very important for XMOS to unify its technology stack: microphone arrays and UAC audio.

Additionally, I hope that XTC compilation can be accelerated with multi-threading on the Windows platform, which is crucial for the 62% of Windows users (according to statcounter) :)
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Post by Ross »

Hi Virgil19,

Thanks for the feedback. Agree on the importance of unification - it's underway.

Regarding multi-threading on Windows, it's unlikely that Make based xcommon system will be fixed for this now. As you probably know, xmake is a (rather old now) build of GNU make but it has -j disabled on Windows to avoid some issues - I think with the xcommon makefiles.

Multi-threading with the xcommon-cmake system will be fine on Windows if you use GNU Make. The plan will be to update xmake to a later version of Make (4.4. at the moment) as it is convenient for users to have make supplied for all operating systems.

Of course, with CMake users can use whatever generator they want - ninja of nmae, for example, but XMOS really only wants to test, verify and guarantee one flow.

For anyone interested here is a presentation (with notes) on the new system: https://github.com/xmos/xcommon_cmake/r ... slides.pdf
User avatar
BritishCat
Member++
Posts: 21
Joined: Tue Oct 24, 2023 11:16 am

Post by BritishCat »

I'm looking at the update
sw_usb_audio on Git Hub. I can’t understand with what software and how to compile updated applications. The application folders contain both Makefile and CmakeLists. There are no libraries in the folder, should they be downloaded in advance or pulled from GitHub? Have you any new description?
User avatar
Ross
Verified
XCore Expert
Posts: 981
Joined: Thu Dec 10, 2009 9:20 pm
Location: Bristol, UK

Post by Ross »

We've not updated the USB audio documents yet as the xmake/xcommon flow is still the supported build method. For the moment both the xcommon and xcommon_cmake related files will remain in place - allowing for some migration period.

The instructions for building using xcommon_cmake should largely be covered by https://github.com/xmos/xcommon_cmake/r ... _cmake.pdf, but here are some brief ones:

Clone the sw_usb_audio repo:

Code: Select all

git clone git@github.com:xmos/sw_usb_audio.git
run cmake, this will fetch the dependencies and generate Makefiles

Code: Select all

cmake -G "Unix Makefiles" -B build 
(im on a Mac and Unix makefiles is the default so I only have to do cmake -B build)

Run the build

Code: Select all

cd build
xmake
You can use a different generator/builder if you like, but this is the flow we test.