Monthly Archives: June 2007

Re: Don’t try this at home

As I mentioned recently, I had trouble with CDT for Windows with performance on my slower machine at home. Well, I’ve been able to trace it to the Harmony VM. Not only that, during testing, I was able to hang the VM when I went to do a CDT index rebuild. So for CDT for Windows 0.9, I’ve dropped the Harmony VM for now. The promise is there, but it’s clearly still a pre-release and they have some work to do to get it fast enough and to fix some other bugs that we raised against them as well.

So after switching to the Sun JRE, things are much better, even on my Core Duo laptop. And I was able to run it successfully on my 666MHz beast of a machine, albeit, it was still pretty slow, but no worse than everything else that runs there…

Introducing – "CDT for Windows"

As I’ve mentioned a few times on this blog and elsewhere, we often get requests for a simpler way to get new users up and running on their Windows machines. This is especially true for students and hobbyist who have a need to learn and play with C++ and don’t want to dive into Linux. As well, we get numerous bug reports from people who are trying to go through all the manual steps to set this all up and miss something.

Well, I am pleased to announce “CDT for Windows”. This is a distribution of the Eclipse Platform and the CDT with the MinGW GNU tool chain for Windows and a Java Runtime wrapped up in a simple Windows installer. With a few clicks you’ll be up and running debugging a C++ application.

The distribution is on SourceForge at http://cdt-windows.sourceforge.net/. It’s currently at version 0.9.1 and contains the Europa Platform and CDT 4. My objective is to get it at a 1.0 level by the end of September with CDT 4.0.1. I have also included the SDL portable multi-media library. My hope is to get the wxWidgets portable application library into it as well.

As with the CDT, I really want to build a community around “CDT for Windows”, mainly since I’m doing most of the work in my spare time (I’ve got a lot of work to do at QNX and the CDT too, you know 🙂 and need to share the work, which really hasn’t been too much so far. I’ve only done light testing so feel free to give it a spin, raise any issues you may find in the SourceForge tracker.

I think it looks pretty slick for something that only took me a couple of days to put together. I hope you find it useful as well.

CDT 4, Now Available!

It’s been quite a year for the CDT. The growth of our community has been very rewarding. And CDT 4 is the fruit of our efforts. There are lots of new features and I counted 1152 bugs marked fixed. We have a What’s New section in our user docs that doubles as our traditional Eclipse New & Noteworthy. You can see it on the Eclipse help machine.

Download instructions are available on our web site as well. There are lots of other ways to get it too. The Eclipse download site has the Eclipse Platform and the CDT in a single zip file. If you are on Linux, I’m sure you’ll see it soon in your favorite distribution. If you are on Windows and want to build Windows application with GNU tools, check my other announcement ;).

Don’t try this at home

I am putting the finishing touches on “CDT for Windows”, my self-contained CDT with MinGW distribution that I’ll be officially announcing later this week. To test it out, I thought I’d try it on my machines at home. One of them is this pretty old Pentium III 667 MHz (or as I like to call it in a throw back to my Iron Maiden days, 666 MHz, the processor of the beast, and I apologize if you’ve never heard of Iron Maiden 🙂 with 256 MB of RAM but with a healthy 80GB hard drive. We’ve been having real performance problems running anything other than a web browser on it, and even then, I can’t do things like TSN’s flash video clips. But, heck, I thought I’d give it a try anyway.

Whatever the minimal required machine for Eclipse is, this ain’t it. What a mess. It took forever to do anything. And all I did was create our Hello World project, build, and fire up the debugger. Now, I am using Apache’s Harmony JRE which is still pre-release, so maybe that’s not helping. But it’s amazing to see what memory starvation does on big software.

One thing I’ve heard off and on over the years is people complaining about the sluggishess of Eclipse and the CDT, especially on older Unix machines that had multiple users sharing resources. Now I see why. Luckily, developers using Eclipse usually have plenty of horse power and memory anyway and this isn’t as big an issue as it was a few years ago. But we still need to watch out. Even running the CDT on my 512MB, 1.8 GHz AMD Linux test machine at work has issues.

CDT 4, and so it begins

For all intents and purposes, barring emergency rebuilds, CDT 4.0 is done. If there are moments in my career that I hold as highlights, this is one of them. I am proud of the work that the committers and contributors have done and of the great feedback and bug reports from our community. Funny enough, I ended up having the final severe bug to solve, and it was a toughie, but I was glad to work my butt off and to put up my share to match all the great work everyone has done this year.

I have a webinar scheduled for July 12 and I invite you all to attend. I’ll be walking through all of the features of the CDT with focus on what’s new. I hope you’ll be as impressed with the new CDT as I am.

But for now, I need some sleep. It won’t be too long before I need to get back to work bringing CDT 4 to my QNX customers and start planning for next year and CDT Ganymede (the next Jupiter moon after Europa you know, at least that’s what they tell me…).

Fun with ANTLR v3

A colleague of mine puts it best, “I have a habit of chasing shiny objects”. I’m interested in so many things in this industry that I love to tinker with that I never actually get to finish any of them, other than a C++ parser I once dreamed of :).

The latest shiny object is the new ANTLR version 3, a fancy new parser generator. I’ve followed ANTLR with interest for a few years and we almost used it for CDT’s parser back in the 1.x days. But at the time we felt that hand coding was the only way we could successfully deal with C++ ambiguous statements (e.g. x * y, is that expression, or a declaration of a pointer to x).

The new version comes with an LL(*) parsing mechanism that uses infinite lookahead (the *) to decide what rules a sentence matches. If you aren’t familiar with what all that means, Terence Parr, the author of ANTLR, has written a great book that explains it all for you. The book itself is interesting since right now there is very little documentation other than the book. But it’s only $24 dollars for the “non-dead tree” PDF version and is an interesting way to help fund the work.

But, this is essentially how Johnny C and I wrote the CDT parsers. We start at the high level concepts and break them down into finer grained detail until you get to the individual characters, creating an Abstract Syntax Tree (AST) along the way. You can base interpretation choices on where you are in the analysis and by looking ahead into the source stream as much as you need. It’s a very powerful technique.

With ANTLR, however, you can specify the grammar at a higher level and have it generate a lot of boilerplate code for you. It may be the one parser generator tool that can convince me that it’s better than hand coding. But to figure this out, I decided to try building a parser. Mike and the CDT guys at IBM are already working on a C parser using LPG, an LR parser generator (LR is bottom up, which, while faster, doesn’t allow you to easily use context information when resolving rules, I prefer LL) and extending it for UPC, Unified Parallel C. And since I need to resolve some extensibility issues for GNU versus MSVC, and having a lot of experience in the past, I decided to try a C++ parser.

Now if ANTLR can handle that, I’m sold. It’ll be an interesting journey and should allow me to try out some ideas on improving how we did some of the things in CDT’s parser. Also this is just a prototype and I don’t really plan on replacing CDT’s parser with it. But it will help me learn ANTLR more and help me help others in the Eclipse community who want to use it. And who knows, another shiny object may fly by and take me on a totally different tangent anyway…

More word on massive multicore

I was just reading this article on Intel’s work on trying to bring massive multi-core processors to market. The had showed a demo back in February that I blogged about back then. However that chip didn’t really do anything other than run 80 threads at a time. It didn’t have I/O or memory from what I could tell.

So now the press is trying to figure out how Intel will productize that. I think the article asks some pretty irrelevant questions, like “Will they be x86 cores? Will they run today’s applications?” And the response is that they are working on it to make it easier for programmers to deal with.

But I think that’s a mistake. Why would you run your favorite e-mail program on an 80 core machine? Eclipse, now that I can see, especially if you’ve ever debugged a run-time workbench and saw 30+ worker threads going at it. But really all we’re doing there is taking the same old paradigm and stretching it beyond it’s limits. The hardest bugs I’ve had to solve were when multiple threads were contending for the same resource and someone forgot to synchronize them. Happens time and time again.

No, 80 cores are great. Mix that with 128 stream processors like those on the latest GPUs, and there’s some massive horsepower that we can take advantage of. You can hide all the complexity behind good compilers and run-times, but I don’t think you’ll get all the benefits of the hardware. I continue to believe that we need some new programming paradigm where we stop thinking of programs as a sequence of operations and more like a network of data flows. Turning to objects was the first step, but I think there’s further we can go.

A new way to get your CDT

Not only am I the CDT project lead, I’m also CDT’s release engineer. That means I get to write all the cool ant scripts and stuff that pumps out the CDT builds on a semi-regular basis. Actually, I’m more the release engineer team lead and cron is my underling that does almost all the work. Actually cron does such a good job I hardly ever have to do any release engineering at all unless I have to change something.

Well, the past couple of weeks I’ve had to change some things. The first one is to meet Europa’s guideline of having our jars digitally signed for security. That way, when you download the CDT, you know you’re getting the officially blessed version from Eclipse. I think the risks are pretty low, but it is possible for someone to make a set of zips that look like the CDT but aren’t and do something nasty things. The infrastructure created by the Eclipse team makes it easy to do the official signing so I figured why not.

The other issue we’ve been running into lately is the sheer size of the CDT builds. They passed the 400 MB mark with all 9 platforms and the two main features, runtime and sdk, that we build. With only 10 GB (now 15 GB) of disk quota at eclipse.org, we kept bumping into the limit and constantly needed to clean up builds. When you take a close look, the 9 platforms are almost identical except for the small shared libraries that we have in the cdt.core fragments. The update site, as a result, is much smaller, around 40 MB, since you don’t get that duplication.

So while I was wearing my release engineer hat anyway, I figured I’d fix all this. The recommended way to get the CDT is using our update site. We will have the runtime feature, the main one that you download, on the Europa update site. That feature along all of the others will also be on the CDT Europa update site. All this will be explained on the CDT’s website when we release. The jar files will also be compressed using pack200 to make downloads a lot faster and save bandwidth on the Eclipse servers and mirrors.

The other big change is that we will no longer be releasing feature and platform specific zip and tar.gz files. Instead, we will have a single zip file called the CDT Master that is an Archived Update Site. So you can install normally via our update site on Eclipse.org, or you can download this zip and install it, still using the update manager, but from the zip file instead. We are currently doing our nightly builds this way, and it works well.

This will hopefully make the CDT easier to get up and running while saving a lot of space and bandwidth for Eclipse. It’s good all around, but it is a change and, as always, I’d like to hear your feedback on it.

Happy 5th Anniversary CDT!

From: “John Duimovich”
Newsgroups: eclipse.tools.cdt
Subject: [ANN] – CDT project changes
Date: Fri, 7 Jun 2002 17:20:34 -0400

The Eclipse Tools PMC is pleased to announce some exciting developments to the CDT project hosted on eclipse.org. First, QNX will be contributing some C/C++ core technology from their recently announced product to eclipse.org. We believe that this will enhance the value of the technology hosted on eclipse.org and bring some industrial strength technology into the CDT project.

And so it began. Five years ago yesterday, the new CDT project was born. The QNX team had just gone through six months or so of compressed development in a remote location affectionately called the “Toolshed” (we’re known as the Tools team). They had basically gone from nothing other than the source to Eclipse and JDT to come up with a fully functional C/C++ IDE. But QNX is an RTOS and run-time company. Tools enable that business but take a huge investment to do really well, and it was becoming clear that a lot of good can come if we would share the work and benefits of that work with others in the open. A handful of other vendors liked what QNX had done and worked together on starting this new direction for the CDT.

A meeting was held at QNX headquarters on July 17’th of that year in what turned out to be the first ever CDT Summit. In attendance were people from QNX, Rational, Red Hat, and MontaVista. From day one, we enjoyed the spirit of “co-opetition”, co-operating competators, that still lives strong today. Here’s a fuzzy picture of Sebastien Marineau from QNX on the right, the first CDT project lead, who really got this whole thing off the ground. He’s sitting with John Prokopenko, a senior architect/manager from Rational and a mentor of mine.

I was at Rational at the time and this was my first exposure to the new CDT. And as we know now, it became a career changing moment for me. Who knew then that I’d spend the next five years living out a dream of mine, to make a software development tool that would help thousands of software developers write better code faster. I wouldn’t have passed that up for anything.

There are many people to thank for getting us here, including Sebastien and the team here at QNX for having the vision to share their great work with the world. To my former co-workers at Rational/IBM (who coincidentally I had lunch with today) for helping get this crazy DOM thing in place. To the gang at Intel (especially Leo!), TimeSys, Chris at TI now at IBM, and everyone else who formed that initial spurt of a community outside the original two, to all the product vendors and users and contributors that now form what I estimate to be our 400,000 member CDT community.

Thank you all! And happy 5th anniversary!

CDT 4, What a difference a community makes

If you’ve ever been to a CDT meeting, chances are you would have heard me state one of my mantras, “Don’t take a .0 version of an open source project and put it in a product”. Certainly, in the past with the CDT this has been very true. And it certainly hasn’t been the fault of the committers and contributors to the CDT who work on it with a passion and really want it to be the best C/C++ IDE it can be. A lot of it had to do with the lack of test pressure we’d get on these releases. And a lot had to do with the lack of feedback from the community until after the release. A lot of it had to do with a young community that was just starting to grow.

Over the last year, this has all changed. With the great bug reports we’ve been getting from the community and the volume of patches we’ve been getting from contributors, and the great work of our army of committers, I am happy to say that CDT 4.0.0, is the first .0 release of the CDT that I feel really good about. I’m sure everyone will be pleased with it from our open source community to the vendors that adopt it.

In the spirit of openness, I have to admit I have been very dissatisfied by our previous .0 releases. And I think I need to share what happened so that we can learn from it. Here are the “lowlights” of previous CDT .0 releases (BTW, the CDT 1.x.0 series actually went pretty well, so we’ll start with 2):

  • CDT 2.0.0 – first introduction of our new parser-based indexer and search. This was a dramatic change to the way we did indexing and the start of our performance woes. In fact I rewrote the CDT scanner (tokenizer) in 2.0.1 since it was brutally slow. We also introduced a new build model that wasn’t really ready for prime time and really needed to be exercised with more tool chains.
  • CDT 2.1.0 – This release was to meet one of the contributing vendor’s product needs and was done while the other vendors were focused on the next release. It was a bit difficult to watch the extent of work they were doing mainly on their own. This really showed the need to standardize our releases (thanks Callisto, Europa, etc!) and how the model we had been following can’t scale as the number of contributing, and consuming, vendors increase.
  • CDT 3.0.0 – This was the big DOM release. Again, massive changes to the indexing and search framework done by one vendor. Even worse, that vendor later had to withdraw its resources on that massive bulk of code due to business reasons. This really showed us the need to have a diverse set of committers working on each component. Contributors will come and go, and you need to build a robust community to survive it.
  • CDT 3.1.0 – Yet another new indexing framework. With the contributors to the original index gone, I took it upon myself to finally throw out our failing assumptions, 100% correctness as a requirement being the big one, and focus on a new strategy to address performance. At the end of the day, this new strategy is the right one and correctness was still pretty good. The problem was that I did it myself and bit off more than I can chew. It was an absolute necessity to get it done, though, but I did do a lot more work for 3.1.1 to make it product quality.

It’s been a while ride over the last 5 years on the CDT (BTW, it was 5 years yesterday, I need to blog about that too!). I think we’ve learn a lot from it, probably enough to write a book about it. Being in the open, I hope others can learn from it too. At the end of it all, I think the most important aspect of any software platform project, commercial or open source, is that of community. And the most important job of a project is to build that community. And that there is probably nothing harder to do than build a community. As our industry goes through this shift, we really need more examples of what to do and what not to do. Then, hopefully, we won’t have to always go through pain like this.