QCon SF 2009: Don Box & Amanda Laucher, Codename "M": Language, Data, and Modeling, Oh My!

These are my unedited notes from Don Box & Amanda Laucher's talk about Codename "M": Language, Data, and Modeling, Oh My!

• [Don has promised not to tweet. That's a good start.]
• Interesting similarities btw/ M and MPS
• M is a language for data - one of the most interesting places of data is a sequence of Unicode code points
• Great support for text processing perceived as critical
• Example: extract some data from text - tweets
• Intellipad - default environment for writing grammar files, this is where tool support shows up first, then in Visual Studio

• M code for a language definition (a function from text to something else, declaratively specified):

  module QCon
  {
  language Simple
  {
  syntax Main = empty;
  }
  }
  

• Main is the rule that is the entry point

• Open file in three-pane (or rather, four-pane) mode
• Show source file, grammar, show output - empty file yields Main []
• Amanda makes a great assistant ;-)
• Non-empty file produces errors
• Change "empty" to "any" - file with a single character works, more than that produces errors; change to any+, validates again

• Simple language for interpreting tweets:

  module QCon
  {
  language Simple
  {
  syntax Main = Tweet;
  syntax Tweet = Content*;
  syntax Content
  = RawText
  | Handle
  | HashTag;
  token RawText = (any - ("#"|"@"))+
  token Handle = "@" Name;
  token HashTag = "#" Name;
  token Name = (any - ("@|"#"|" ""))+;
  }
  }
  

• regular expressions at the token layer, context-free grammar at the syntax level

• Amanda: "Only crap languages make you define something before you have to use it"
• Discussion betwen Josh, Amanda, Don about whether or not the grammar is correct
• Good point about interactive grammar development using the three-pane editor
• Pattern names used to display data in a structured way
• Adding @Classification["Keyword"] syntax-colors the source
• M is structurally typed
• M consists of lists and records
• Generating the right-hand side:
• Intellipad crashes! [Boom!] :-)
• The spec for M is licensed under the Microsoft OSP (which makes people as happy as they can be working with Microsoft)
• Javascript-based implementation of parts of the language; subset of the three-pane mode
• Toolchain is written in C#, parser written in M, more and more compilers written in M

• Intellipad crashes! Again! [Boom!] :-)

  module QCon
  {
  language Simple
  {
  syntax Main = v:Tweet => v;
  syntax Tweet = v:Content* => v;
  syntax Content
  = v:RawText => v
  | v:Handle => v
  | v:HashTag => v;
  token RawText = (any - ("#"|"@"))+
  token Handle = "@" v:Name => v;
  token HashTag = "#" v:Name => v;
  token Name = (any - ("@|"#"|" ""))+;
  }
  }
  

• bubbles up the actual content, result is just a list of the strings extracted

  module QCon
  {
  language Simple
  {
  syntax Main = v:Tweet => v;
  syntax Tweet = v:Content* => v;
  syntax Content
  = v:RawText => v
  | v:Handle => v
  | v:HashTag => v;
  token RawText = v:(any - ("#"|"@"))+ => { Kind => "RawText", Text => v }
  token Handle = "@" v:Name => { Kind => "Handle", Name => v };
  token HashTag = "#" v:Name => { Kind => "HashTag", Topic => v };
  token Name = (any - ("@|"#"|" ""))+;
  }
  }
  

• Question from Josh: Isn't this mixing lexing and production rules? Don: Goal is to have no difference, but it can be pulled out

• Next: Consuming stuff
• Using TDD to Don's tweets
• VS project includes language grammar defined earlier
• Amanda: Can one debug a grammar? Don: Answered later
• M runtime can be hosted inside a C# program
• Recently stopped internally to use .NET 3.5/VS 2008, now exclusively on .NET 4 and VS 2010
• Language defintion is included in the program binary

• Showing off some

  var language = Language.load(/* MImage */ typeof(Program).Assembly, "QCon", "Simple"); // yields runtime that can parse a program
  dynamic result = language.ParseString(input);
  bool hasHashTag = false;
  foreach (var content in result)
  {
  hasHashTag = content.Kind == "HashTag";
  if (hasHashTag)
  break;
  }
  AssertEqual(true, hashHashTag);
  

• Demo is actually working

• Change AssertEqual to Assert.IsTrue to get around exception thrown
• Question from Ian Robinson: Q. Can LINQ be used to walk the result? A. Not yet, as dyamic and LINQ don't mix yet
• Not going to write any more C# types, has written all that are in him ;-)

• "This will fail!"

  var language = Language.load(/* MImage */ typeof(Program).Assembly, "QCon", "Simple"); // yields runtime that can parse a program
  dynamic result = language.ParseString(input);
  var query = (from content in ((IEnumerable)result).OfType<dynamic>()
  where content.Kind == "HashTag"
  select content).Any();
  Assert.IsTrue(query);
  

• It failed indeed.

• "This talk is not about integration of two features I don't work on"
• M is optionally typed, structurally typed
• optional typing: syntax Bob = any* : Integer32 => 42;
• only partially plumbed in the current version

• Example for structural typing (not really working yet):

  module Ola
  {
  type HashTagRec =
  {
  Kind : Text where value == "HashTag";
  Topic : Text;
  }
  }
  

• Rudimentary grammar debugging: set breakpoints in input source text; 4th pane shows up, shows matching stack

• Syntactical and semantical editing support
• Semantics relies on hooks that are not there yet
• M is built using an M Grammar
• Language completion for M is built using C#
• Ambiguators for GLR need to be written in C#
• One of the metrics used to evaluate the language: XML. There is a grammar for XML, briefly demoed.
• MPS guys have a different religion – Microsoft believes people have a text editor they love
• <?magnum PI?> :-)
• Comment from Martin Fowler: Main difference to classic tools such as ANTLR is the dynamic - no code generation needed
• Comment from Ola Bini: Didn't see the type-checking and debugging before – now he's impressed