Round Table on Code Control SoftwareEd Popkov, Ann Harrison, Pavel Cisar, Dave Schnepper, Reed Mideke, David Warnock and others
Ann Harrison: One alternative for code control is to continue to use Marion. It's a home brew code management system based on InterBase. It uses blob filters to remove system specific stuff on input and replace it with the appropriate system specify stuff on output. Add a web interface and away we go.
Ed Popkov:That'd be a nice option, indeed. As I understood from your brief description, Marion is based on an approach that's slightly(?) different from the one recommended by GNU. Code transformation at check-out stage is a great feature.
[A web interface] is where our problems may begin. Technically it's just the matter of time. PERL can handle such things beautifully. But this will require Marion specs to become available. I don't know how acceptable is that for Inprise/NewCo.
Ann Harrison: One can only ask. They'd decided to go with a commercial code control system to eliminate the work of maintaining Marion and so they didn't have to explain again why they were using a home brew. I'd expect they'd throw that in the open source pot.
Dave Schnepper: Marion is a sensible, short-term solution. If Marion were marketed as a product 10 years ago, it would have been very successful, and state-of-the-art. The art has advanced over the last 10 years, but marion has not.
(Let's make sure marion is also open-sourced! Someone will get motovated to add proper merge facilities to it!)
TCS & TAN - the InterBase test-control-system and test-analysis-system are also extremely good tools. Also likely not state of the art anymore - but could have stood alone as products a few years ago.
Ed Popkov: Did Marion require alot of maintaining?
Ann Harrison: No, but anything greater than -10 was too much of a burden. My opinion.
Dave Schnepper: It tended to be a tradition that about once a year, some programmer about to go on a week's vacation, would throw some new features into it. Other than that, it needed backups like any other database.
We were trying to convert to ClearCase - as it had better features for merging, branching, and isolating projects from each other. At the time I left they still hadn't managed a conversion (I guess about a man-year of work went into trying to convert over). It seemed to work fine for source storage, but the build facilities weren't up to what was needed. (InterBase has some loops in it's dependency graph - gpre depends on include files generated while the engine is being built, which requires gpre to build the engine (etc)).
Ann Harrison: And if that man-year had gone into Marion instead?
Helen Borrie: Ann, do you think Marion is a contender against the options on the tables Ed made?
Ann Harrison: Yes, I do.
It is available on every platform where InterBase runs. A central database can serve all platforms. That helps keep the code from diverging. It was designed to handle the file types used in InterBase. It keeps logs and history and, since the database is InterBase, any type of report could be generated.
It currently has a CLI interface which should not be hard to hide behind a browser-based client.
As Dave Schnepper said, it doesn't do merging. When I was last working with it, there was some controversy about how merging should work and how exception conditions should be handled.
That's now crystal clear. One would need to alter to logic to allow two people to check out the same module - that's one check in an obvious place or perhaps an index that shouldn't be unique. You'd have to write a stored procedure that called a UDF to handle the actual merge. Again, not tough.
The code is quite straight-forward compared to the engine, but is a good exposure to the coding style used throughout the product set. The database access is through GDML - different but not difficult. Obviously, it's a flexible tool that we can adapt as we see fit.
Finally, nothing improves a cook like eating his own food. We would be making a clear statement of our belief in InterBase by using it ourselves.
Pavel Cisar: Merging is crucial feature for Internet-wide development. Exclusive check-outs doesn't work (at all) for such kind of development. Side note: CVS looks as the best way to go for us. It's not perfect, but there is more pro's than con's about it. CVS can't hurt, but other VC systems can (IMHO).
Ed Popkov: If it came to votes, I'll definitely give my voice to CVS. Myself, I find even RCS quite enough. However, there are people who don't know about these systems, also there are people aware of another packages. The idea is to work out a list of criteria and evaluate every major code control system with the help of this list. We can't just say `It's good, period'. First we have to realize what params are essential, then prove that the system of our choice is good. And let the best win :)
David Warnock: Sorry I don't agree, if we want Open Source Interbase to suceed then we have to make sure that you only need the standard open source tools to work with it. This applies to version control, compilers - the lot. So for version control we should just go with CVS.
Fabricio Araujo: After some thinking I have to agree with you... When IB source becomes available, it's very worth to attract OSS from other projects. So, if this is a need (and I guess it are) CVS is good for start. If later we discover its a s*, maybe we look for another or simply review these messages about this type of stuff for voting. Maybe a trial test ( a simple OSS project for community test CVS in the real life) are a good idea.... It's only a idea, however...
Ed Popkov: Nope. It's open and it's free. There's no such thing as Open Source standard. I can agree with compilers, but choice of version control system is totally upon the development group. This is an internal matter. People are getting the stable sources with releases, but as long as this is a DBMS, I'd think twice before making snapshots available.
David Warnock: What is? If you mean Interbase then
Of course there is [an Open source standard], it is just not official, go and take a poll of open source projects and check on the version control used. 99% will be cvs, all the open source hosting projects eg at VA Linux use CVS, every big project uses it (Apache, Linux [currently, will switch to bitkeeper], I believe the BSD projects do, Postgresql etc etc).
Remember that in the Open Source world Interbase is the newbie, newbies cannot decide a different standard in a volunteer community - nobody will join them.
Probably the key reasons why CVS is in this position are
But if we do not get read-only access to the development and stable source tree's then I am off - it is not then open source.
In fact compilers are less of an issue, as long as patches to support other compilers are accepted it is generally good to support more than one, there is no compiler that is available and best on all platforms (eg several projects do not use GCC on all platforms). My original statement on compilers is just that a free compiler must be supported. ie we must not have to buy Visual C++ to compile Interbase. Visual C++ can still be supported for those who want to use it.
Ed Popkov: I said it's free, meaning the open world, and then-open IB as a part of it. And as long as it's free, no one should be forced to take something for granted, or as an axiom.
If there are members of this community not familiar with CVS, or in doubts about its advantages, we (I said, my vote goes to CVS) have to convince them and provide everyone with analysis and resources.
The question is not to close the newly open source tree, but to explain why it's going to be available with CVS, and not, say, Uniteam. Myself I have a hands-on experience with RCS and StarTeam. I've heared and seen that CVS really works for big projects, and works good. As it's based on RCS, I don't expect any problems switching to it.
Michal Bukovjan: Well, of course, the system has to be good, but it has to be free in the first place, at least free clients. You cannot and should not expect anyone who wants to contribute his labor and resources for free to purchase a specific commercial VCS solution.
Chad Hower: Dave Warnock said
I further disagree, respectfully.
IB currently compiles in MSVC++. Thats not open source. What happens if we decide to write utils for IB and we do them in Delphi or CB? Oops, we cant. Dave won't let us, Delphi and CB arent open source. Windows? Guess well have to drop support for that too.
CVS will likely be our end choice. But if we blindly choose it wo consdering alternatives, then we are very closed minded. What if a free (but not open source) VCS was available? Lets say for instance we decided StarTeam was the perfect VCS, and Starteam liked our cause and donated free copies for our effort. You would turn it down because its not open source?
Jacco: I have heard that the newest "hype" in version control is the possibility of parallel developement, which means that to developers can "check-out" unit simultaniously and merging of the source can be done automatically under certain conditions (if for example the developers touched only different procedures/functions).
I don't know many of the VCS systems that we are investigating but parallel developement might be a feature to include for an open source project.
Chad Hower: Oh man no... This works only at a few places like Borland where you have huge units and cant segregate them. Paralell dev IMO for most places just complicates things quite a bit. You can change things way to easily without even having to think about it.
David Warnock: For an open source project there is currently only one standard that is used on 99% of all projects and that is CVS.
We should definately not use a commercial product for version control on an open source project. It just locks many developers out.
Chad Hower: Are my messages just not reaching you? By your definition, we should just let IB die, becuase 60% of the business world uses Oracle. Forget even LOOKING at alternatives, you've already made up our minds for all of us.
What if [a commercial product] were available free for this project? VCSs are usually licensed controlled by the server.
David Warnock: Anyway the anology is not really correct. A better one would be that if I want to attract Oracle developers to Interbase I have to make it easier for them, so I make sure that if I have an application using Oracle it is really easy to plug in Interbase and see the advantages.
In the same way if there are excellent open source programmers out there coding on various dbms (there are more than 10 in various stages) [or other projects, or new to open source but good C programmers] and I want to attract them to come and develop interbase I have to make it easy for them. As a vcs is probably the only tool the project really forces onto the developers I think it would take a brave newbie project to select a different tool to all the other projects.
::What if it were available free for this project? VCSs are usually licensed
Unless the company is going to contribute real money to proper interbase development (in return for the free advertising) I would be very unhappy, even the I would not be happy as I would worry about their influence.
I would anyway be interested to know if any commercial tools meet them (my expectation is that there are distintly different requirements and priorities for vcs tools for open source and commercial projects, therefore a commercial tool will be focused on the requirements of commerical software and is less likely to suit the needs of open source projects).
On a political note, does the feeling of this group carry any weight in the selection of VCS anyway?
On a technical note I have used both PCVS and Source Integrity on OS/2 and Windows (16 and 32 bit) prior to using CVS. My feeling is that Source Integrity has a fantastic feature of the sandbox, to my mind this is critical to large and distributed projects. Essentially PCVS lost track of the files once they were on your hard disk. CVS also has this feature, only unlike the GUI of Source Integrity I have never yet got it confused.
CVS works least well with binary files such as Delphi forms. These files really need locking when people are editing them and CVS discourages locking (you can set it up for locking but it's not the natural way). However, in distributed projects where people are not immediately accessible if they have locked a file you need access to I think locking is generally a bad thing (and therefore I personally think tools like Delphi are not very well suited to large distributed development).
Bitkeeper will have better support for Lines of Development (like sophisticated branches) and also has consistancy checks to protect against corruption. It also has a very neat feature of allowing stages. eg we could run a site copy of IB, we could then modify that as a team and only when all our team had completed the feature commit the whole back to the main project. That is very powerful and flexible and is hard work with CVS (and pretty impossible with Source Integrity).
My requirements for a VCS would be
I would personally turn it [a free, but not open source VCS] down yes. I believe the image of the open source Interbase amoungst the open source community from where we need (IMHO) to recruit coders (and to a much lesser extent users) would be harmed by such a decision.
I am sorry you think this is dictating (I am not sure how I could dictate from a position of zero power anyway) but to my mind the critical fact that needs to be taken into account is that we are planning an open source project not just a database server project. I would definately consider all VCS tools for a closed source project, but to my mind that is not really appropriate in this instance.
In other areas I am much more flexible and willing to consider different tools such as ANT for building instead of make. But a vcs is different, it is the only tool a coder has no choice about. If you say IB will compile with MSVC++ then I can check it out and make it compile with XYZ++ (and you might accept patches to make that the standard). I do not have the same freedom when it comes to the VCS, if I want the source I cannot choose to get it with SourceIntegrity if it is kept in PCVS.
Robert Love: What if InterBase provides the version control and we connect to their version control server? We should have plans just in case, they don't do this, but it has been one of my thoughts.
::IB only compiles in MSVC++ on Windows as MSVC++ is only available for windows.
I'll make a slight correction to that. IB ships (currently) compiled & linked under MSVC. It compiles fine under Borland C. (And there's likely a gnu for windows? I've never checked -- but IB compiles & ships fine under gnu on other platforms.
::On a political note, does the feeling of this group carry any weight in
If we come up with a list of requirements as you and several others have, and then evaluate the available tools for those requirements, I think it would go a long way towards impacting the decision that NewCo makes as to the VCS tool that they will use. After all, the volunteers that will be working on the OS IB will have to use this VCS, so their opinion will matter.
Mike Eden: On and off, I have used CVS quite a bit over the years. Of all the copy-model CM tools I have seen it probably works the best. It has reasonable support for branching and merging, labeling, and (with the help of install targets in makefiles) release management. Its not ClearCase - but the price is right :)
The thing that drives me nuts with CVS is the import utility. The utility to import existing code to create the initial source repository can drive you to drink. I mention this because it is not clear to me how the IB community source control efforts will interact with whatever Newco does. If they use something different, than someone may be constantly having to merge Newco's changes into the Community's repository - and that could be painful. A tool will probably need to be developed to automate this.
Vince Duggan: Let's assume for the moment, that NewCo does it right, and there is no reason to assume they won't.
They will therefore have a system such as CVS, and everybody in the community who is involved will have access to this. Therefore there is no reason for the community to duplicate the work.
What we should be doing with this discussion, is putting pressure on NewCo to choose the system we select as the most appropriate. We should not be assuming that NewCo will do their own (incorrect) thing, and the community will be doing our own (correct) thing.
If NewCo does it wrong, however....
Reed Mideke: IMHO, this is exactly the right approach to take. I firmly believe that the people who go on to NewCO will do their best to make it work for the community. After all, that's what their success or failure will be based on.
Obviously, there will be plenty of problems, but given flexabilty and hard work on both sides it should work out just fine.
Just to throw more oil on the fire, I'll remind you that there is NO one source control system for the linux kernel. Some people keep their trees in source control, but development is done by sending patches to the mailing list. It works pretty darn well.
Not to say that IB should follow the same model, but even open source projects that do have a centrol source control have some limit to who is given write access to the source control system. Typically, a person who makes a fix but does not have a (for example) CVS login sends a patch to the mailing list or someone who does have access. This ensures that changes to the main tree are done someone who has demonstrated ability to work with the code. If someone sends enough good patches, then they are given a direct login. Remember, even if a patch doesn't make it into the main tree immediately, that doesn't prevent anyone who finds it useful from using it.
CVS seems to work quite well for a lot of projects. I would stay away from strongly platform centric systems (MS source safe...) or pricey commercial ones (ClearCase), since both will make it harder for people to get involved.
In internet development, diff and patch are your friends ;-) With these, it doesn't matter too much if different people use different source control systems for their own work.
Jason Wharton: NOTE: When I say CVS I mean CVS and comparable solutions...
Touching on this some gave me an idea about some of the other issues being discussed. Would it be possible for newCo to use three basic levels here?
My gut feel here is that Marion is well suited for a tight-knit engineering group to work with a large base of code. It probably has a lot of built-in mechanisms that cater to the development of InterBase. What it apparently doesn't do well is interface to the masses. Since it doesn't allow anything more than exclusive locks on files and because it won't merge them automatically.
One thing that I totally agree on is that the tools on the client side that are going to be open sourced should be directly administered in the CVS system.
Sorry if it seems I'm speaking without much knowledge of both CVS and Marion because frankly I don't! ;-)
Dave Schnepper: This was done once before -- the first LIBS ports were done via a team working inside of CVS with a link to marion. A nightly script performed a sync between the cvs and marion trees.
The only thing Marion really has that is customized to InterBase is some automatic "porting" -- things that could easily be done with sed or perl scripts.
For me - the key thing is the interface between the open community and Internal source tree. Someone mentioned that email diffs are used a lot in the Linux community, and I agree with that. Changes need a lot of review before they are merged into a master base -- so email diffs are the way to do that. NewCo can use whatever makes sense internally. There should also be a master "public tree" for community submissions.
Key issue also: Coding style. The InterBase coding style is fairly rigid, and different from many other code bases I've worked on. I am a strong believer in consistency, however. Whenever there's a chunk of code that is styled differently, I wonder about whether the author really knew what they were doing. (This is simple things, like indentation, spacing, and line breaks).
Reed Mideke: As the person who was responsible for source control and builds at IB until very recently, I'll try to shed some light on the situation with source control at IB.
A number of people, including Ann, have suggested that either everybody or at least NewCO could continue using marion. The problem with this is that even before all this happened marion had long been in disrepair, and we had reached it's limits in a number of ways. We were in the process of switching to Rationals ClearCase. When I left, the source was stored in marion and automatically mirrored into ClearCase.
ClearCase is an excellent system in many ways, but suffers from:
Marion is a decent system, however, it was not maintained for a very long time, and lacks some essential features. It stores diffs, but does not allow you to reconstruct previous versions without MANUALLY applying the diffs in order. Diffs for different changes are created using different diff programs (how's that for a tongue twister ?), and a misconfigured developer workstation might cause no diffs at all to be stored for a change.
Marion also does not support ANY form of branching or parallel development, so branches and code freezes are done by making a backup of the GDB. This was a serious drawback before open source, and would be much worse if there are all kinds of patch versions and even more platforms.
The existing system at IB worked as follows:
At that point, a copy is made (we'll call it v6.0), and any further changes required for the release are made to both v6.0 marion and dev marion. Additionally, any changes for 6.x or 7.x are made to dev only.
After v6.0 is released, the v6.0 marion could be used as the basis for a bug fix release, and any 6.0 ports to secondary platforms. If 6.0 linux was started, it would begin as a copy of v6.0. When finished, the linux specific changes would be merged back into dev, so that the next linux port would be easy. As you can see, this becomes increasingly complex as there are more releases on more platforms.
Marion also does not support a hierarchy (directory structure) that is more than two levels deep.
None of marions problems are insurmountable, and in hindsight, it might have been easier to make marion do what we wanted rather than switch to ClearCase (which has a tremendous overhead for configuration). Marion could be made into a good source control system, but it is a LONG way from being comparable with things like ClearCase and Perforce.
My feeling is that a solution which mirrors (both ways) between marion and CVS (or any two source control systems) is not a very good idea.
If a suitable CVS configuration for IB can be created, then everyone (including the NewCO development staff) should use it. Maintaining two source controls systems is always more work that maintaining one.
If marion is to be used at all, it needs someone dedicated to maintaining it, and making the changes required for it to work in the new situation.
IB development is not particularly dependent on marion. I had complete builds working from clearcase, nightly, in parallel to the official builds. The makefiles for the clearcase version clean up a lot of cruft from the old system. Since they are 99.9% GNU make syntax, they would be a good starting point for open source IB.
The one really neat thing about marion is being able to make SQL or QLI queries on the source and history directly. ;-) The ClearCase query language sucks compared to a real RDBMS.
Ann's comments about 'eating our own dogfood' by using a IB based source control system are valid. However, this is dependant on someone actively maintaining the system.
Doug Chamberlin: I'm greatly perplexed by the thought of someone re-inventing the wheel for anything other than pure entertainment. Given Reed's list of shortcomings of Marion there is only one plausible reason I have heard to continue using it - that it uses Interbase as a storage system. Even that reason is not a very good one, IMHO.
Shouldn't we focus our attention and energy on the main task at hand, that of ensuring that Interbase 6.0 is released promptly and maintained adequately?
Dave Schnepper: I think there was an overfocus on marion's shortcomings.
Marion's best feature (2nd best being using IB) is the change history it keeps. There has been innumerable times I used that as part of a bug fixing strategy.
Many change control systems have links to bug tracking systems -- marion's strategy was bit different, and it made it easy to do these kind of history searches.
As for the main task at hand -- that kinda depends on the where the bottleneck is for getting 6.0 out the door. If it's going to be mainly NewCo folk pushing it out, then continuing to use marion may be the best (short-term) strategy. Changing tools is always a short-term disruption. (Though, of course, changing to a better tool could shorten cycles -- "Sharpen the saw") .
Reed Mideke: I agree [there was an overfocus on marion's shortcomings]. It was abandoned not because of a fundamental flaw, but because no-one maintained it. So the descision to abandon it was trying to solve a managment problem with software. Almost never works.
[What you say about bug-fixing strategy] is true. But other real source control systems have this ability too. ;-)
I agree [changing to a better tool could shorten cycles].. The plan (at IB) was to keep using marion through 6.0 As you mentioned , outside contributions can still be integrated easily if they are created with diff and patch.
Dalton Calford: How much work do you think there would be to update Marion?
Since you have intimate knowledge of Marion and what is lacking in it, would you be willing to list the development projects needed to make it more suitable? (or even become a team leader for such a project if the source for Marion also becomes available)
I would be willing to commit time and money towards such a project, if it looks feasible and the community shows an interest in using it.
Reed Mideke: The main things I see that marion needs are:
I kind of agree with the comment from Doug Chamberlin about re-inventing the wheel, but on the other hand, open source can scale in strange ways. If there are people interested in the source control system who would not contribute much to the product, than having them work on the the SCS is not really a loss. It's a gain if they make a system better than the one that would be used otherwise.
I'm not voting for or against using marion at this point. Just trying to add some data.
Doug Chamberlin: I see the main task at hand as more long term. I'm thinking about what the investment over the next few years should be for the long term health of the product, now that it is open source. That is, how should NewCo and the volunteer community invest their efforts over the next year or two?
Focussing on the development and maintenance of the core Interbase product seems wise. Focusing on the development and maintenance of marion, to add say branching support, seems foolish if there are perfectly capable alternatives available.
Short term decisions about how to get 6.0 out the door are a different matter altogether. Expediency would be more valuable here, especially if NewCo will be doing 99% of the work.
Randal Carpenter: Seems smart to me. No learning curve for developers at NewCo cause they use it already...we can learn it quick, and the developers all know ib so should be able to fix marion pretty efficiently and quickly and make it widely available for ib developers who may or may not be able to afford to put down $$$ on a code manager.
David Warnock: Reed, thanks for the insights into practice at Interbase. So far as I can see the three most likely options at present are
It seems to me that the key advantage of marion is the ability to seach the changelogs in sql. Would it be possible to move completely to cvs except add some server scripts fired by the commits which commit to marion. Then use marion just for the read only searching that david suggests.
I mean much less than the mirroring concept. Nobody would commit or checkout in marion, all access to the real code would be via cvs. But marion would continue to provide the search capabilities that it seems cvs lacks.
Reed Mideke: I do have to qualify my statment above. I was specifically refering to automatic mirroring.
If NewCO uses marion, and some outside developers keep their source in CVS (or clearcase or, god forbid, MS source safe), this is not a big deal, as long as the medium of interchange is patches that are applied with human supervision.
Dave Schnepper also mentioned CVS used during the first LIBS project. I started at IB after that was over, so I can't say how well it worked, but my expirience is that a one way mirror (so for example all checkins are made into marion, and nightly updates are made to another system) is trivial, but a two way mirror is asking for trouble.
I don't know how CVS works with respect to integrating it with external tools, but this sould like it could be an excellent open source project using IB.
As Dave Schnepper and Doug Chamberlin have pointed out, this is probably not too big of a concern for the short term (IE 6.0). Since 6.0 was already in feature freeze before this started, I think the biggest contributions from outside will be in tracking down bugs. But, since theres not much the rest of us can do until NewCO actually appears, it makes sense to be as prepared as we can for the the future.