Architect

My hack week project about building a library repository for Mono turned out to be more complex than expected. Actually, I didn't expect it to be simple, but I wanted to have something working (even if not complete) after that week. I finally spent the week doing research and implementing some of my ideas as a proof of concept.

My initial idea was about building a Library Management System, but after all the feedback I got and after looking at similar systems for other platforms, that idea evolved into building something more generic. Here is a summary of my conclusions.

The end goal is to simplify the development with Mono, improve productivity and code reuse. There are three main areas that we need to improve:

• Library management. Most of projects depend on third party libraries, and yet there isn't a common model for sharing them.
  
• Project dependencies. Dependency checking is traditionally done by configuration scripts using tools like pkg-config. However, this system falls short when trying to implement more advanced features, such as on-demand dependency downloading or better integration with IDEs.
• Development life cycle. The way applications are developed in Linux is very complex for many developers, specially those coming from Windows. There is an important gap between IDE based development and command line based development, and that's specially noticeable when a project has to be published, packaged and distributed.
  

The idea is then to create a better Mono development platform built on top of what we have now, but providing more advanced and better integrated development tools. This is what it would provide:

• A Library Management System, which would be used to share libraries in an on-line repository. It would provide tools for publishing, searching and downloading libraries. It would also define a common model for dealing with libraries in projects.
  
• A new model for project dependency checking, with support for on-demand downloading of dependencies. Projects would be able to define dependencies on libraries, compilers, code generators or any other kind of development tool.
• It would define a common development model which would work with independence of the development environment being used.

Right now I'm using the codename Architect to refer to this platform (subject to change). Following are some more details about this it.

Archtiecture

Architect will be based on Mono.Addins. What's interesting about Mono.Addins is that it allows defining an add-in ecosystem which is not limited to a single application, but which can be shared by many applications. There will be an Architect add-in ecosystem, and development tools will be able to plug into it in order to offer extension points, or to extend other tools.

That is, everything will be modeled as add-ins (in the more generic sense of the word): libraries, compilers and other development tools. A project would then depend on a set of add-ins.

Mono.Addins defines an add-in model with support for dependencies, and it has some support for on-demand dependency downloading that would be integrated in the IDE and other development tools.

Library Management System

Architect will provide tools for publishing, searching and downloading reusable libraries. Libraries will be published as add-ins, and as such they will be able to depend on other libraries or tools. Library add-ins will be able to include not only the library itself, but also other things like for example Monodoc documentation or designer editors (if the library implements a widget).

Reusable libraries may or may not come with source code. Mono binaries are portable, so we could build a library repository which contains only binary libraries and be able to reuse them from any system. However, when developing an open source project, the source code of the libraries will be needed soon or later. Architect should take it into account, and should make it easy to handle.

When developing an application, I don't really need the source code of the external libraries I use. The binary assemblies are enough to build the project. At development time, the most comfortable way of dealing with libraries would be to specify that my project needs for example Mono.Cecil.dll and get it automatically downloaded by Architect when I need to build the project. In this way I can avoid adding binaries or a private copy of the Mono.Cecil sources into SVN. Architect would take care of giving me what I need.

However, at packaging time, when creating a tarball of my project, I may need the source code of the library. It depends on whether the library is being individually distributed as a package or not.

There are two kinds of libraries. On one hand there are API stable libraries which are installed in the GAC and which are distributed in a package. For example, gtk#. On the other hand there are libraries which are not so stable, or which are small and which are not installed in the GAC. Mono.Cecil is an example of such library. To make things a bit more complex, some distros may decide to package some libraries which are not packaged by other distros.

So, when creating a tarball for a project, Architect should take into account the kind of libraries needed by the project. It should download the source code of Mono.Cecil and include it in the tarball. Well, not only include the source, but it should also take care of building the library using the included sources when the tarball is built. And that leads to an important question...

Should Architect provide its own Build System?

Defining a new build system with a new project model is not really necessary right now. However, Architect should provide a generic API and a generic command for building projects. It would be like a proxy command which would end calling the native build tool. That's required to support building of libraries downloaded from the repository in a generic way.

Add-in Management

Right now Mono.Addins has an add-in manager which can be used to download and install add-ins, and it defines a repository format for publishing add-ins. However, the add-in engine is independent from this add-in manager. The add-in engine only cares about add-ins existing in the file system, doesn't matter how those were installed.

So, what we could do is to make the add-in manager pluggable, so it could support additional distribution systems. For example, we could add support for apt. It means that there could be an apt repository with a deb package for each add-in, and the apt support add-in would be in charge of mapping add-ins to packages.

So for example, if we are trying to build an application that requires Mono.Cecil, MonoDevelop would request Architect.Mono.Cecil to be installed to the add-in manager, and the add-in manager would request this add-in to the apt support add-in, which would pull Architect.Mono.Cecil.deb from the repo.

However, I don't think that everything in the Architect repository can be published as a deb package. We might end having 1000's of libraries in the repository, many of them very small or rarely used. CPAN has over 11.000 modules. Publishing all of them as packages would be crazy. So, downloading of unpackaged add-ins should still be allowed.

What's interesting here is that Architect based applications always work in terms of 'add-in', and will be the duty of the add-in manager find the best way of locating and installing all required add-ins.

A Common Development Model

All the features provided by Architect will be available both through an API and from command line tools, so they can be used from different development environments. To illustrate how all this would work, I've written some use cases:

Creating a new project with MonoDevelop

• The user creates a new project of type Boo.
  
• The project will need Mono.Cecil, so the user selects it in the reference selector. If it is not installed the user can open the library manager, look for the package that provides the library, and install it.
• If the user wants it, MonoDevelop can generate a makefile for building the project.
  
• Commit the project to SVN (or wherever).

Checkout and build and existing project in MonoDevelop

• Get the code from SVN.
• Load the project in MonoDevelop.
• If Boo or Mono.Cecil is not installed, MonoDevelop will use Architect to download and install it.
• Build the project.

Checkout and build and existing project using a makefile:

• Get the code from SVN.
• Run the build configuration script, generated previously by MonoDevelop. This script will use Architect's command line tools to check and download dependencies if necessary, including the Boo compiler.
  
• Run the makefile, also generated by MD.
  

Create a tarball of the project:

• Run the package configuration script. Since Mono.Cecil is not a packaged library, the source has to be included in the package. The package configuration script uses Architect to get the source code of the library and include it in the tarball.
• Run make dist.
  

Build a tarball of the project

• Run the build script and the makefile, as usual.
• The makefile will use a tool provided by Architect to build the embedded Mono.Cecil sources.
  

Conclusion

Implementing all this will take quite a lot of work, but some of it is already done. Mono.Addins already has some support for add-in management and on-demand downloading of add-ins. MonoDevelop has some support for library management, and has a generic build api. I spent part of the hack week improving Mono.Addins and extracting code from MonoDevelop into an independent library. I also implemented some command line tools which could be used to automatically include source code from libraries in tarballs. I finally was able to more or less support the use cases I described above. However, I consider this code still a proof of concept.

In the following months that code may make it into SVN. All depends on the feedback I get and the release plan for MonoDevelop. One important first step would be to stabilize and freeze the Mono.Addins API, and publish it as an independent package. In this way we would have the basic infrastructure on top of which to start building the new platform.