Wednesday, October 27, 2010

NGit

One of the features we want to include in the next MonoDevelop release is integrated support of Git. I committed a first version of a git add-in some months ago which works by invoking the git command and parsing the results. This solution satisfied some basic needs (Mono was moving to git at that time), but it was done as a short term solution, since parsing git output is not very reliable, and has portability issues.

The next obvious step was to use GitSharp, a fully managed .NET library which implements most of git commands. This library is basically a port of JGit, a Java library for doing the same. I started replacing the git invocations by calls to GitSharp classes, and I could implement the basic functionality, but I found some limitations that were not easy to overcome. On one hand, GitSharp was a bit outdated, lacking some of the recent features and bug fixes done in JGit. Bringing GitSharp up to date with JGit would be a lot of work, since every JGit commit had to be manually translated from Java to C#. On the other hand, GitSharp depends on a set of cryptographic libraries (required for the ssh protocol support) which we can't easily include in MonoDevelop due to export regulations.

I then decided to try a different approach: the idea was to use Sharpen, a free Java to C# translator implemented by db4o, to automatically convert the JGit code to C#.

After some weeks of work, I have been able to generate a C# library (which I call NGit) with all the JGit code. Most of the work went into tunning and fixing Sharpen, and into implementing in C# some core Java classes which didn't have direct replacement in Mono. This was more complex and took more time than I expected, but I think the effort is worth it. The generation of the library is automatic, although not perfect since it requires some patches in the java code and some patches in the generated code, but the patches are small and easy to maintain. Keeping NGit in sync with JGit is very easy, since it is just a matter of pulling the java code and running the conversion (I already have the conversion process automated in a makefile).

I've also been able to convert and run the JGit unit tests, and got 90% of tests working. The failures of the remaining 10% are in general due to different behavior of Java vs C#, or JUnit vs NUnit, and are not real NGit bugs (although I still have to review some of them).

In the process, I also converted Jsch, which is the library used by JGit for the ssh communication. The new NSch library only has Mono.Security as external dependency, so it will be much easier to distribute for us.

So, we now have a fully managed git library with ssh support without any dependency external to Mono. The library has around 56000 lines of generated C# code (including the unit tests). I recently published the source code in github.

I'm now implementing the git add-in using NGit. Most of the commands are already implemented, although I'll have to do some testing before I push it to master. There are still some performance issues I'm tracking down, but things are looking good. I intend to submit my fixes to JGit. My fist patch has already been accepted and pushed, and conveniently ported to NGit.

Update: I'd like to clarify the relation of NGit with GitSharp. GitSharp is composed by two libraries: GitSharp.Core.dll and GitSharp.dll. GitSharp.Core.dll is a manual port of JGit. GitSharp.dll is a more high level and .NET friendly API that wraps GitSharp.Core.dll. NGit can be a replacement for GitSharp.Core.dll, but GitSharp.dll is still useful, since NGit may be a bit too low level and java-ish for some use cases.

25 comments:

Chris Aniszczyk (zx) said...

It's great to see you contributing to JGit! I hope you view as us a welcoming bunch! I and the rest of the JGit team look forward to working with you and spreading Git everywhere!

Anonymous said...

Please point out where/how to get Sharpen, and the license it is available under.

Joshua Poehls said...

I wonder where this leaves the GitSharp project. If GitSharp is a port of JGit, and NGit is now a 'better' port of JGit... interesting.

Lluis said...

The Sharpen code is available here:
https://source.db4o.com/db4o/trunk/sharpen

Some documentation:
http://developer.db4o.com/Documentation/Reference/db4o-7.12/java/reference/html/reference/sharpen.html

Sharpen's license is GPL.

Lluis said...

@Joshua: updated post with an answer.

Andrius Bentkus said...

So it takes basically one C# developer to generate as much C# code as 30 Java developers can code.

This is not fair :D

mdi said...

This is fascinating, basically there are X things to get excited about:

* MonoDevelop is getting Git support across all platforms.

* There is a new up-to-date Git binding for .NET based on the fantastic JGit work and we will be able to keep track of the changes the JGit team is doing.

* Sharpen has been tuned to help developers port Java libraries to C#, this I feel is a big story on its own.

pablo said...

Sounds awesome Lluis!

We're looking to implement interoperability between Plastic SCM and Git and I wonder if we could take advantage of this code too. I wonder if we can use the license at all.

My goal is to make Plastic replicate back and forth from Git.

I'd also like to put some effort on a MD add-on with Plastic, specially since we tend to use MD more and more internally.

Congrats!

pablo

www.plasticscm.com

Hylke Bons said...

I'm considering using NGit for my project (sparkleshare.org).

Currently I'm using GitSharp. It's really nice, but SSH seems to be broken. The API is very flexible and I've been able to port most logic in a clean way from wrapping commands. Also, there haven't been any commits to it in a month...

Hopefully NGit will get a nice API as well. I looked at some of the code and it looked pretty confusing. There also is no documentation...

So I guess I have to start complaining to the JGit devs instead :)

Chris Aniszczyk (zx) said...

Hylke, the API in JGit is currently in the 'org.eclipse.jgit.api' package. We have commands that mimic the git porcelain commands. We have started some documentation for JGit but not much...

http://wiki.eclipse.org/JGit/User_Guide

A lot of us that work on JGit also work on EGit which is the Eclipse tooling on top of JGit. Since Eclipse.org is moving to Git, we are placing priority on getting some great tooling out there for developers.

Anyways, we love to have feedback on the API or contributions. We have an extensive contributor guide for JGit and EGit:

http://wiki.eclipse.org/EGit/Contributor_Guide

All patches come in via a Gerrit instance at eclipse.org

http://egit.eclipse.org/

Anonymous said...

So that means NGit is BSD-licensed like JGit, since it is a derivative work?

Lluis said...

Yes, the license is the same as JGit: BSD.

Vagaus said...

Hi

Nice to see sharpen being used outside db4o :)

@Lluis

You mentioned fixes in Sharpen. I am curious, what issues have you found? Is it possible to send us a patch (so we can consider incorporating this changes into Sharpen) ?

@Anonymous

And we have a forum to discuss Sharpen issues also: http://developer.db4o.com/Forums/tabid/98/aff/7/Default.aspx

Best

Vagaus said...

For some reason the link for Sharpen forum gets truncated when posted in the comments.

Best

Anonymous said...

Not to undermine the excitement about MonoDevelop, NGit and Sharpen, but a port of Jsch is what caught my eye. This was done years ago for SharpSsh [1] (also hosted on code project). The project was not well maintained. Recently, someone started ssh.net [2] but my question is will nsch be maintained as well as ngit? If so, this should be shared with equal fanfare as it is very exciting. Will Nsch use Windows (FIPS certified) Crypto API on Windows?

[1] http://www.tamirgal.com/blog/page/SharpSSH.aspx
[2] http://sshnet.codeplex.com/

Nice work,

Vlad

Anonymous said...

Will NSch be packaged as a stand alone library for mono and .net, .net could really use a good open source up to date library(I think there are a couple of projects that have been abandoned.

Lluis said...

@Vagaus: JGit makes extensive use of generics, and Sharpen was having trouble some use cases. I also added some basic support for JUnit->NUnit conversion. It would be great if those changes could be merged to upstream Sharpen. I'll clean up the patches and I'll post them to the forum.

Lluis said...

@Anonymous: I'm also interested in having a good and up to date ssh library. Right now I'm busy polishing up MD's git support, but I intend to publish NSch as a standalone library in the future (in fact, it can already be used as a standalone library, all you need is NSch.dll and Sharpen.dll.

Anonymous said...

Can use NSch with both mono and .net currently?

Lluis said...

@Anonymous: I haven't tested the library on .NET, but it should just work. If you want to try, you'll have to get Mono.Security.dll from Mono.

jfb said...

Nice.

However I must say, your GitSharp export objections are wrong. Org.Mentalis is only needed for HMAC, which can be replaced easily by the System.Security.Cryptography variant (I have done this in a branch I made), and DiffieHellman.dll is in Mono.Security.dll, which for my version incidentally is SharpSSH's only dependency. It's about an 8 line patch to get SharpSSH to that state.

I'll have to take a look at this automatic converter you mention - it could be very useful.

jfb said...

I'm taking a look at this, and it seems more nicely encapsulated than GitSharp, so I will use it for my project and let you know how it goes. After two grueling days of altering GitSharp, I find its SSH is half-broken anyway, so I appreciate this port you've done.

However one comment on gen/cs.patch:

+ string path = Path.Combine (AppDomain.CurrentDomain.BaseDirectory, "resources");
+ path = Path.Combine (path, "global");
+ return new FilePath (Path.Combine (path, fileName));

Could you please use new FilePath(a, b) instead of Path.Combine(a, b)? That way Path.* is all within a single .cs file. If it's not too much trouble anyway -- it might be minor for you, but it would save me a ton of trouble :) I'm doing a game map editor so I am capturing all file I/O and redirecting it to store everything in a single file. The way you have path mostly in one .cs right now, if continued, will be ten times better than GitSharp. Thank you!

jfb said...

Also, how are you dealing with http://developer.db4o.com/Forums/tabid/98/aft/357/Default.aspx ?

Thanks

James

jfb said...

Here's a fix:

if (!res.AsyncWaitHandle.WaitOne (timeout > 0 ? timeout : Timeout.Infinite, true))

replacing

if (!res.AsyncWaitHandle.WaitOne (timeout, true))

in Extensions.cs. This fixes git:// failing with the default (unspecified) timeout.

It seems that *with* a timeout, SSH connections leave a thread running in a wait state, at least with the Fetch command I tested.

Lluis said...

@jfb: thanks for the fix, it's now in github. About the Sharpen code, it is all written from scratch and licensed as MIT. If you want to propose more changes, the best way is to clone the ngit repository in github and do pull requests. Thanks!