Back to Questions
4.3k
views2
comments
Sequence Comprehensions, XmlSchemaInference and Xml file comparison
f#functional

Hi,

In learning F# and .NET I decided to replace an ancient little tool I use to compare two xml files within certain tolerances (e.g. all pairs of xs:decimal values within a tolerance of 10e-4 pass - otherwise the comparison fails.) Rather than use the xsd.exe utility I chose to use the schema inference and class generation capabilities of .NET so that any pair of xml files could rapidly be imported and compared without requiring the .NET SDK to be installed. First I infer a schema, create a csharp source file and compile to an assembly with these five functions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

 ///Infers a Xml Schema Definition file (xsd) from an instance xml file
let inferXsdFromXml xmlpath =
    use reader      = XmlReader.Create(xmlpath: string)
    let schemaset   = (new XmlSchemaInference()).InferSchema(reader)
    let arr  = [|  |] : XmlSchema array
    schemaset.Schemas().CopyTo(arr, 0)
    let xsdpath = Path.ChangeExtension(xmlpath, "xsd")
    use xr = new XmlTextWriter(xsdpath, Encoding.Default)
    xr.Formatting <- Xml.Formatting.Indented
    arr.[0].Write(xr)
    xsdpath

///Create a C# class from the given xsd file
let createClassFromXsd xsdpath targetnamespace =
    use fs  = new FileStream(xsdpath, FileMode.Open )
    let xsd = XmlSchema.Read( fs, null)
    let set = new XmlSchemaSet() in set.Add(xsd) |> ignore ; set.Compile()
    let schemas = new XmlSchemas()
    schemas.Add(xsd) |> ignore
    let importer = new XmlSchemaImporter( schemas )
    let ns  = new CodeNamespace( targetnamespace )
    let exp = new XmlCodeExporter( ns )
    for elm in xsd.Elements.Values do
        let map = importer.ImportTypeMapping( (elm :?> XmlSchemaElement).QualifiedName )
        exp.ExportTypeMapping( map )
    let pro = new Microsoft.CSharp.CSharpCodeProvider()
    let csf = Path.ChangeExtension(xsdpath, "cs")
    use sw = new StreamWriter(csf)
    pro.GenerateCodeFromNamespace(ns, sw, new CodeGeneratorOptions())
    csf

///Pass the filepath of a C# source file to compile to an assembly
let compileCSharpFile filepath referenceasms =
  let pro = new CSharpCodeProvider()
  let dll = Path.ChangeExtension(filepath, "dll")
  let opt = new CompilerParameters(GenerateExecutable = false, OutputAssembly = dll)
  opt.ReferencedAssemblies.AddRange referenceasms
  pro.CompileAssemblyFromFile( opt, [|filepath|] )

///Pass the filepath of a C# source file created using xsd.exe to compile to an assembly
let compileCSharpXsdFile filepath =
  compileCSharpFile filepath  [|"System.dll"; "System.Data.dll"; "System.Xml.dll"|]

///Create an assembly from the given xml file
let createAsmFromXml xmlpath targetnamespace =
  let xsdpath = inferXsdFromXml xmlpath
  let cspath = createClassFromXsd xsdpath targetnamespace
  compileCSharpXsdFile cspath

Now an assembly can be created on the fly from an xml file and loaded so:

1
2

createAsmFromXml "C:/KalmanPredictions.xml" "Kalman"
#r "C:/KalmanPredictions.dll"

The XmlSchemaInference has limitations and so therefore not all possible xml files will work, but any xml file that is handled by xsd.exe will be correctly handled here. We can now load and save the xml files into an object of the class Kalman.Predictions using these two functions:

1
2
3
4
5
6
7
8
9
10
11
12
13

///Takes the path of an xml file and load it into the generated class.
let xmlLoad<'ty> file =
  if File.Exists(file) = false then failwithf "Error: %s does not exist." file
  let xs = new XmlSerializer(typeof<'ty>)
  let xtr = new XmlTextReader(file)
  xs.Deserialize(xtr) :?> 'ty
  
///Takes an obj of class 'ty and saves it to the given file path.
let xmlSave (ob : 'ty) file =
  let xs = new XmlSerializer(type 'ty)
  use xtr = new XmlTextWriter( (file : string), null, Formatting = Formatting.Indented ) in
    xs.Serialize(xtr, ob)    
  if File.Exists(file) = false then failwithf "Error: file %s could not be created." file

To compare two instance xml documents within a tolerance I first created a sequence comprehension that recursively yields the public properties of the KalmanPredictions object. By yielding a triple (o, prop, index) containing the object, the property and the array index we can later get and set its value. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

 /// This function enumerates the leaf properties of a composite object 
let propertySeq x = 
  let rec propSeq o =
    seq { let props = (box o).GetType().GetProperties()
          for prop in props do
            let v = prop.GetValue(o, null)
            when not (v = null)
            match v with
            | :? Double  | :? Decimal  | :? XmlText
            | :? String -> yield (o, prop, null)
            | :? Array as arr   ->  let ind = ref 0
                                    for elm in arr do
                                        match elm with
                                        | :? Double | :? Decimal | :? String | :? XmlText
                                            ->  yield (o, prop, [|box !ind|])
                                                do ind := !ind + 1
                                        | _ ->  yield! propSeq elm
            |  _                ->  yield! propSeq v  }
  propSeq (box x)

The replaceIfPass overwrites the result object's value if it's within tolerance of the benchmark object, thereby making the benchmark and result node identical. Only result nodes that fall outside the tolerance will not be overwritten.

1
2
3
4
5
6
7
8
9
10
11
12

///Checks whether the inputs are numeric, then runs the comparison function
///If the comparison passes, the benchmark value is copied to the result file.
let replaceIfPass compare bench result = 
    let (o1, p1, i1) , (o2, p2, i2) = bench, result
    let v1, v2  = getPropVal bench , getPropVal result 
    match (v1, v2) with
    | :? Double, :? Double
    | :? Decimal, :? Decimal  -> let  d1, d2 = (v1 :?> IConvertible), (v2 :?> IConvertible)
                                 let  pass, diff = compare (d1, d2)
                                 if   pass then p2.SetValue(o2, box v1, i1) ; true 
                                 else false
    | _                       -> v1 = v2

propertySeq and replaceIfPass functions only cater for the types I needed, but both could be easily extended. We can now read our two xml files, create two seqs then zip them together in the compareSeqs function so: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

 let benchmark = xmlLoad<Kalman.Predictions>  benchfile
let result    = xmlLoad<Kalman.Predictions>  resultfile
                          
///Sequence of the leaf objects (including the property information and array index) in the benchmark and result composite objects
let benchSeq, resSeq =  propertySeq benchmark, propertySeq result

///Zips the two sequences into a sequence of pairs and then iterates over the pairs running the compare function
let compareSeqs seq1 seq2 compare =
    for (a,b) in Seq.combine seq1 seq2 do
        if not (compare a b) then Console.WriteLine("Diff: Bench {0}; Result {1}", getPropVal a, getPropVal b)

///Here we run the compareSeqs function using the tolerance passed to the diffAgainst function
///Running this function changes the 'result' object in memory.
compareSeqs benchSeq resSeq (replaceIfPass (diffAgainst 0.0001)))

let resultfile2 = Path.ChangeExtension(resultfile, "tol.xml")
xmlSave<Kalman.Predictions> result resultfile2

This allows us now to write the two files back out and then pass the resulting files to XmlDiffPatch to visually compare the xml nodes that failed the comparison. It would be interesting to use the FSharp.CodeProvider and perhaps create a few custom TypeMappings for example to replace nulls with 'a options or other functional idioms. We could also extend this sample to create TypedDataSet and Soap bindings on the fly.

Although this is a toy example it taught me some powerful paradigms of F# and .NET and can be run interactively to boot. Btw the original tool was over 3500 lines of C++, less capable, untyped and rather ugly. Any comments/corrections on the techniques used here would be appreciated. Please let me know if anyone would like a complete code example.

Regards,

Daniel

I've added a code snippet and helper functions required to use this below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

 
#light

open System
open System.CodeDom
open System.CodeDom.Compiler
open System.IO
open System.Text
open System.Xml
open System.Xml.Schema
open System.Xml.Serialization
open System.Globalization
open System.Reflection
open Microsoft.CSharp

 ///Required for toDouble function
let nfi = new NumberFormatInfo()
///Allows any object with the interface IConvertible to be converted to System.Double
let toDouble dbl = (dbl :> IConvertible).ToDouble(nfi)

///Gets the value of any .NET property given the object the property is defined on,
///the reflected property's definition and the index of the require element (if the property is an array.)
let getPropVal (o, p, i) = (p :PropertyInfo).GetValue(o, i)

///Returns a tuple of a boolean indicating whether the two inputs are within tolerance
///and the absolute difference between them.
let diffAgainst tol (a,b)  = 
    let d1, d2 = toDouble a, toDouble b
    let diff = Math.Abs(d1 - d2) in (diff <= tol, diff)

///Returns a boolean indicating whether the two inputs are within the given tolerance
let toleranceCheck tol (a,b) =
    let res = diffAgainst tol (a,b) in match res with passfail, diff -> passfail

<HINTS id=hah_hints></HINTS>

.

Hi Daniel,

This all looks great. Don't have any comment on the coding style but was wondering if you could dynmically load the dll once created. I believe CodeDom has some features to this. This would mean you would need to create overloads (or more easly different versions of the xmlLoad and xmlSave) that receive a type object rather than a type parameter. This might be easier for folks that want to use it from a program rather than from F# interactive.

Also this looks like it might make a nice base for a Sourceforge/CodePlex project. Have you considered registering it on one of these sites.

Cheers,
Rob

By on 12/7/2007 8:54 AM ()

Hi Robert,

I had similar thoughts concerning the dynamic loading of the dll. Loading an assembly is simple. As you suggest an amendment to xmlLoad is required:

1
2
3
4
5
6
7
8
9

let xmlLoadDyn (ty :Type) file =
  if File.Exists(file) = false then failwithf "Error: %s does not exist." file
  let xs = new XmlSerializer(ty)
  let xtr = new XmlTextReader(file)
  xs.Deserialize(xtr)

let asm = Assembly.LoadFrom(@"KalmanPredictions.dll")
let kal = asm.CreateInstance("Kalman.Predictions")
let kalobj = xmlLoadDyn <| kal.GetType() <| @"kalpred1.xml"

This works to load the assembly and correctly deserializes the file, but it does not load the Assembly into the FSharp namespace (my terminology is no doubt wrong here.) I'm not sure if it is possible to cast kalobj into a object of type Kalman.Preditions.

I was actually wondering whether dynamic loading could be supported in the VsAddin (as opposed to VsFsi), so not only #r STRING_LITERAL, but #r LET_BOUND_VALUE_OF_PATH_TO_ASSEMBLY might be possible .Using #r STRING_LITERAL gives you immediate Intellisense and type inference, and supporting this in a programmatic way would be awesome.

The behaviour I'm interested in supporting is similar the the R environment, where I can point a reader to a csv file. It reads the file then allows immediate programmatic access to an object representation of it, with statistics, querying and visualization. This is no doubt done with dynamic typing in R, but perhaps using the dynamic compilation I showed above (or something far smarter as is Don's way) F# could do this with static typing.

Regards,

Daniel

By on 12/7/2007 10:48 AM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper