Thanks for the post and the blog. I read it, and agree that it has some great ideas.

To me, the crucial question is whether someone has actually found the F# project limitations (no sub-folders, files must be kept in compilation order) a hindrance in their work. If so, how much of a hindrance - is it a show stopper, or a major factor, or just an annoyance.

Personally, so far I've only found it an annoyance. I can't say that it's worse than that.

Thanks. And, thanks for the trick to create folders in the project :)

By on 2/17/2010 3:59 PM ()

...
Thanks. And, thanks for the trick to create folders in the project :)

Be careful - adding new fs files using add file dialog (right click on the folder itself) can break the project file. If it happens you will not be able to open the project in VS. To whip it back into shape you will have to edit the project file manually and group all <compile> elements from the same folder together
<input id="gwProxy" type="hidden"><!--Session data--><input onclick="jsCall();" id="jsProxy" type="hidden">

By on 2/22/2010 1:06 PM ()

...
Be careful - adding new fs files [to the sub-folder]...
<input id="gwProxy" type="hidden"><!--Session data--><input onclick="jsCall();" id="jsProxy" type="hidden">

Just to add some detail to my post, I haven't yet experienced much pain from not being able to fs files into a sub-folder, but I would very much like to put data and resources into a sub-folder (to keep the top level uncluttered). I expect that I'll be using your tip for that.

Thanks again for kicking off this discussion - it's been very productive.

By on 2/23/2010 3:09 PM ()

1 year, about 100 *.fs files solution. Files kept in compilation order is good thing. You have basic declarations first in project file tree and logic closer to the end. As for folders it would be nice to have them, but hard to imagine how to implement them and keep compilation order clearly visible. Maybe something like sub projects will be nice. Compile all sub-projects first and project code later.

By on 2/18/2010 1:05 AM ()

It is a little annoying -- it's one thing that the order is required (greater minds than mine have stated this is necessary for type inference), but that if it <i>is <i>required (e.g., it can't work any other way), then why not have VS manage the sort order for me (perhaps w/a command) by doing a stable sort based on the dependency-tree? And/or have a concrete (compiler ordered) view and a friendly (user ordered) view of the files. (It still seems like a rather baroque thing for a programmer to think about these days.) Is it possible to have more than one compilable (but ambigous) ordering of all files, such that machine ordering that can be kept from the user is not possible? (This is related to the need for explict "and" for mutual reference as well and the inability to have these span compilation units, when other languages seems to be able to figure this out.) What is it about F# (or the language family) that makes something like a multi-file/pass analysis not an option to remove these restrictions? If there's a canonical source I'm missing for this, please let me know and I'll RTFM.

By on 2/18/2010 8:09 PM ()

It's not a bug, it's a feature.

It is possible to have more that one ordering of files that compile and have different meanings, so 'in theory' it's not possible to have a tool figure out dependencies for you. (I would bet that 'in practice' a heuristic tool could typically get it right most of the time; I haven't tried to carefully reason through it.)

The diagnostics from type inference (as well as the performance of the compiler when doing type inference) suffer greatly where there are large recursive scopes. The cases where people are forced to ask for help when they can't figure out why type inference is computing some type... these cases all involve "let rec...and", mutually-recursive classes ("type...and"), or members (which are implicitly let-rec). See for example

The more mutually recursive things you have, the more 'entangled' the inferred types of everything depend on everything else, and then when a type error is introduced, it is very hard to provide useful diagnostics that point to the correct location in the chain, or give an error message that suggests the cascade of inferences that lead the the failure.

A linear order of program fragments simplifies thing greatly; when a prefix of a program typechecks, you can typically localize a type error to the suffix that introduced it.

Even with all considerations of type inference aside, I still like the ordering feature. It forces you to manage dependencies. Nearly every big project suffers from dependency-management issues, and F# is a great tool for aiding dependency management. Let me find an internal email where I was writing about this... ah here we go:

It might be just choice of terminology – perhaps you’d prefer “design layering” to “architectural layering”. Regardless of the nomenclature, F# provides a couple of mechanisms for within-assembly encapsulation and dependency-management not found in e.g. C#. First, F# signature files effectively allow you to create a “friendship zone” within a source code file. Entities can be public within that one file, but then hidden by the signature file so that downstream files can see only those entities “published” by the signature file. Second, the file ordering ensures that entities in File4.fs cannot refer to entities in File5.fs, which enforces a layering of components within the project.

In theory, one can use assemblies for both of the benefits mentioned above – I think that many small assemblies could be a great way to deal with architectural layering and encapsulation in .Net. In my own experience, though, “assembly boundaries” are typically chosen as a result of other criteria (which include performance and deployment considerations), and assemblies always end up being “too big” to effectively encapsulate individual components or layer an architecture…

…I cannot resist telling a little “war story” here… Back when I worked on WCF, our architects envisioned a two-layer system. The bottom layer was just about bindings and channels and messages – raw components for building channel stacks and sending and receiving messages. The top layer was itself divided into two sub-layers: at the very top there was a programming model for authoring CLR interfaces marked up with attributes that described service/operation/message/data contracts, and below that (but above the bottom layer) was a ‘description’ layer which was intended to allow alternative programming models to targets the same description OM (and build a corresponding runtime). Though we did finally ship a product that realized this architecture, I recall constantly having to fight to preserve it as the layers all bled together. In particular I recall working on a DCR in a Beta milestone to remove some ‘top layer types’ that had gotten into the ‘bottom layer API’. You might wonder how that came to be in the first place, but when you have 100 people from multiple product units who all have a stake in (the gigantic) System.ServiceModel.dll, it is easy for individual design decisions to lose track of the overall architecture.

The point is, complexity is our constant enemy, and so we should welcome tools that can aid us in combating it. I can’t help but think that if WCF had been written in F#, then the binding/channels/messages stuff would have been the first third of the files in the project, and the description/runtime stuff would have been the next third, and the programming model would have been the last third, and thanks to the “F# file ordering”, it would have been very hard to introduce the aforementioned design defect (the one that required a late DCR to remedy). In C# (which is what System.ServiceModel.dll was actually authored in) all the types were visible to one another and so there were no language-enforced barriers to prevent API changes that ‘broke the layering’.

The anecdote about WCF is not an example that exists in isolation. The .Net Framework 2.0 has this problem in spades; System.dll, System.Conifguration.dll, and System.Xml.dll are all hopelessly entangled with one another. This manifests in a variety of ugly ways. For example, I found a simple repro in the VS debugger that effectively crashes the debuggee when hitting a breakpoint while trying to loads symbols, caused by the circular dependencies among these assemblies. Another story: a friend of mine was a developer on the inital versions of Silverlight and was tasked with trying to trim down these three assemblies, and the first arduous task was trying to untangle the circular dependencies. "Mututal recursion for free" is very convenient on a small scale, but it will destroy you on a large scale.

VS2008 shipped a week later than planned, because VS2008 had a dependency on SQL server, and SQL server had a dependency on VS, and whoops! in the end they couldn't produce a full product version where everything had the same build number, and had to scramble to make it work.

This isn't a message confined to F# or to .Net. It's been a long time since I read Large Scale C++ Software Design, but I recall "dependency management" as being a central theme of that book as well. Poor dependency management not only makes code hard to read and refactor, it also tends to make your build system slower, make integrations harder...

I'm evangelizing an unpopular position here, but my experience is that everything in the world is better when you're forced to consider and manage "dependency order among software components" at every level of the system. The specific UI/tooling for F# may not yet be ideal, but I think the <i>principle <i>is right. This is a burden you want. It <i>is <i>more work. "Unit testing" is also more work, but we've gotten to the point where the consensus is that work is "worth it" in that it saves you time in the long run. I feel the same way about 'ordering'. There are dependencies among the classes and methods in your system. You ignore those dependencies at your own peril. A system that forces you to consider this dependency graph (roughly, the topological sort of components) is likely to steer you into developing software with cleaner architectures, better system layering, and fewer needless dependencies.

By on 2/18/2010 10:30 PM ()

Great food for thought, thanks...

One hears much about the pure language aspects of funcational programming and F# these days, but not much along these lines - perhaps this can be a start of a set of articles.

If the explict layering within a project is useful, then some sort of formalism sounds appropriate - "the first third of the files", "the next third" and "the next third" doesn't quite seem to fit the bill of "management" - perhaps folder support or actual "barriers" or "zones" are the way to go.

So I'll need to try to re-adjust my thinking of sig files (haven't bothered with them at all) and file ordering as anacronisms, and think of them as tools...

By on 2/18/2010 11:17 PM ()

Yeah, I'd be interested in reading or taking part in write a series of articles on using F# for "programming in the large". Most articles available at the moment seem to focus on programming in the small.

By on 2/19/2010 2:05 AM ()

I think there is a misunderstanding on what the main gripe with the F# project system is. Nobody is really questioning why F# needs to have compile order set manually. It has been explained a thousand times. As a matter of fact, this is not an issue when working with F#.

The issue is that the solution explorer is used for this feature, instead of a better and more appropriate place. In my opinion, project structure and compilation order are entirely different concepts. When the solution explorer is "hacked" to be a compilation order setter instead, one loses quite a lot of features that are expected in an IDE. Visual Studio is called an IDE, because it has things like a built in convenient project structure manager. With F#, that feature is lost. I mean, come on guys, do you seriously expect the F# IDE to be competitive like it is today? One has to have certain masochistic tendencies to enjoy rummaging through 200 files in a single folder. I don't buy the argument that you can still have directory structures on the disk. Again, I don't want to be switching between VS and Explorer, I expect an IDE to provide functionality for project structure management.

Even an extremely simple compile order management view that Mike has shown is miles better than the solution F# provides by default, and I can't believe that in this day and age of really advanced IDEs, we're expected to get by with something that plunges us back to programming with a fancy notepad and explorer.

It is not a question of why implement a better solution, this really boils down to why not? One does not need a super sophisticated solution explorer that can handle everything from compile ordering to predicting the future. One just needs a separate bit of GUI for the compile order, and the solution explorer can continue to work as it did before.

By on 2/19/2010 4:37 AM ()

(oh no, I just spend a bit of time typing up a long reply and then lost it, darn! here's the summary...)

There are still plenty of people asking why ordering matters in the language.

Yes, the tooling could be improved; Mike has demonstrated e.g. how to tease apart the 'ordering' from the 'explorer' that are currently confounded in Solution Explorer, for example; there are probably a lot of options to explore in this space.

The F# IDE today is great compared to tooling experiences of many functional languages. The F# tooling is still lagging behind C# and VB. But those languages have an 8+ year "head start" implementing industrial-strength tooling, we can't expect F# to catch up fully in its first release. One expects things to continue to improve over time.

'Why not implement feature X?' The answer is always the same, see e.g.

By on 2/19/2010 8:17 AM ()

I understand the huge costs involved, so I get the idea. Although it's a little bit apples-and-oranges (or at least McIntosh and Granny Smith) to me as (a) top-level code in C# is clearly a bad idea :) and (b) we're talking about "only" some GUI glue here (in the case of ordering) in the VS extension space, not something quite as subtle as language changes - though I suppose when you start talking about crossing team boundaries inside of MS and have to deal with pixels that users see, there's a whole new level of pain to be had. And it's true that ordering ain't all that bad in the scheme of things -- I'd rather see clean water in Haiti or the debugging experience improved first (faiure to debug into local fn's sometimes, adding inspection during pipeline |> evaluation, etc.)

As others have stated, it's only because F# is so cool that we expect everything else to be as nice! - maybe you shouldn't have done such a good job on the first go. (Of course, if we only had an alternate lighter-weight lambda syntax, and... :))

I hope JetBrains has some F# lovers there so we get a good F# version of ReSharper with F# re-factorings, etc. BTW, just to end with another "why can't we have feature X" - are we going to get "rename" in the final bits? - or are there imperial complications there as well?

By on 2/19/2010 9:13 AM ()

BTW, just to end with another "why can't we have feature X" - are we going to get "rename" in the final bits? - or are there imperial complications there as well?

:) Just to set expectations, if you don't see the feature in the RC, then it will not be in the RTM release. Features for VS2010 are all done; we are just doing a few high-priority bug fixes at this point.

By on 2/19/2010 11:01 AM ()

BTW, just to end with another "why can't we have feature X" - are we going to get "rename" in the final bits? - or are there imperial complications there as well?

:) Just to set expectations, if you don't see the feature in the RC, then it will not be in the RTM release. Features for VS2010 are all done; we are just doing a few high-priority bug fixes at this point.

Brian,
Do you have a sense for whether VS2010's new extensibility model makes it easy enough to add these features that they could be created as community projects? Or is not enough of the compiler's internal data available for this to be practical?

By on 2/19/2010 11:32 AM ()

Brian,
Do you have a sense for whether VS2010's new extensibility model makes it easy enough to add these features that they could be created as community projects? Or is not enough of the compiler's internal data available for this to be practical?

(Compiler internal data structures are I think irrelevant, the key bits are the VS project system internal data structures.)

I don't have a crisp sense. In the original post on the thread, Michael has shown what can be done with a 'flavoring' of the project system via public extensibility APIs, but it still potentially runs up against some more fundamental limitations. (Alternatively, you can always create your own project system, but that is a <i>lot <i>more work, and also requires knowing about internal APIs to talk to the F# language service (and thus is not supported).) (I don't think there's anything specific to VS2010 here; my guess is that the same general capabilities and limitations happens with VS2008, at least as far as the solution explorer/ordering/etc issues we're talking about.)

By on 2/19/2010 11:50 AM ()

<quote><quote> Brian, Do you have a sense for whether VS2010's new extensibility model makes it easy enough to add these features that they could be created as community projects? Or is not enough of the compiler's internal data available for this to be practical? </quote> (Compiler internal data structures are I think irrelevant, the key bits are the VS project system internal data structures.) I don't have a crisp sense. In the original post on the thread, Michael has shown what can be done with a 'flavoring' of the project system via public extensibility APIs, but it still potentially runs up against some more fundamental limitations. (Alternatively, you can always create your own project system, but that is a <i>lot <i>more work, and also requires knowing about internal APIs to talk to the F# language service (and thus is not supported).) (I don't think there's anything specific to VS2010 here; my guess is that the same general capabilities and limitations happens with VS2008, at least as far as the solution explorer/ordering/etc issues we're talking about.)</quote> Sorry, in addition to the project system and file reordering (which isn't a big deal to me), I also intended to refer to anotheraccount's "rename" suggestion (and potentially other features which the C#/VB IDEs expose but F# doesnt') when asking about extending VS2010.

By on 2/19/2010 12:00 PM ()

Sorry, in addition to the project system and file reordering (which isn't a big deal to me), I also intended to refer to anotheraccount's "rename" suggestion (and potentially other features which the C#/VB IDEs expose but F# doesnt') when asking about extending VS2010.

Ah, I see. I don't think that C#/VB expose any terrific APIs for this, and I <i>think <i>most 3rd-party components that do cool refactorings have e.g. their own source code parsers they've built from scratch. So once again this doesn't rely on knowing compiler internals, rather just leveraging the VS extensibility that allows you to hook up new menu items and UI and whatnot. I don't have deep expertise here to know all the details.

By on 2/19/2010 12:11 PM ()

<quote><quote> Sorry, in addition to the project system and file reordering (which isn't a big deal to me), I also intended to refer to anotheraccount's "rename" suggestion (and potentially other features which the C#/VB IDEs exposes but F# doesnt') when asking about extending VS2010. </quote> Ah, I see. I don't think that C#/VB expose any terrific APIs for this, and I <i>think <i>most 3rd-party components that do cool refactorings have e.g. their own source code parsers they've built from scratch....</quote> I am facing the same problem - I have to parse the F# source code design time to extract the bistro controllers and their dependencies. I do not need the full blown parser - but I should be able to extract some information even if the source is incomplete or has syntax errors. Rather than re inventing the wheel I would love to use the built in parsers. Curious enough MS CodeDom API even has a facility to provide access to the parsers - the ICodePaser interface. Unfortunately F# CodeDom compiler object does not implement this interface. Right now I am looking to use CodePlex Irony project to build the grammar full enough for my needs. Did I mention that I hate re-inventing the wheel? With C# I was able to build LALR (sub)grammar good enough to extract what I need. I am not sure how difficult will it be for F#. As to the new Visual Studio extensibility model - I used it to build a django editor and it is leaps and bounds better than the old one. Unfortunately as far as I know at this time it only covers the text editor. All the rest, including the project system stays pretty much where it was - COM based with a thin layer and incomplete of managed objects.

By on 2/22/2010 6:11 AM ()

MS seems to want to make it possible, but not <i>too easy<i> to create extensions that depend on leveraging languages becasue they don't provide access to the parsers. This is directly related to a project I'm doing - I have the choice of taking an old C#2.x grammar I wrote in Prolog and using it as a meta-grammar to generate fsyacc code and then adding 3.x features (ick), start from scratch w/another parser (arg), or pay \$10K for someone else's hand-coded parser that's not even rule-based (ack). And I <i>really <i>don't want to deal with F#'s light syntax. So please MS, bring out your parsers!

By on 2/19/2010 12:42 PM ()

Could top level quotations help here?

1
2
3
4
5
6

[<ReflectedDefinition>]
let main () =
// do stuff...
()

let mainExpr = <@ main @>
By on 2/22/2010 6:41 AM ()

Could top level quotations help here?

Correct me if I am wrong but to get anything from quotations the entire project has to be compilable. If this assumption is correct than quotations are less than useful design time where you have to deal with incomplete code.

BTW runtime quotations is the means we currently use to access this information.
<input id="gwProxy" type="hidden"><!--Session data--><input onclick="jsCall();" id="jsProxy" type="hidden"><input id="gwProxy" type="hidden"><!--Session data--><input onclick="jsCall();" id="jsProxy" type="hidden"><input id="gwProxy" type="hidden"><!--Session data--><input onclick="jsCall();" id="jsProxy" type="hidden">

By on 2/22/2010 8:03 AM ()