System.Xml.XmlWriter Memory Footprint

toortoor Member Posts: 52
edited 2014-10-13 in NAV Three Tier
Hi there,

for writing large xml files I want to use System.Xml.XmlWriter.
So I wrote a small test codeunit to check if it works well from within NAV.

The resulting xml file looks like expected, but the NST grabs some hundred MBs auf memory when running the code and it doesn't release the memory after completing the code (or closing the client).

When running the "same" code in C# the program grabs less than 5 MB of memory.

What the hell is NAV doing here?
Any ideas on how to avoid the huge memory consumption?
I've seen the same behavior when using XmlReader a while ago.

I tested with NAV 2013 R2 CU11.

The NAV code:
xmlWriterSettings := xmlWriterSettings.XmlWriterSettings();
xmlWriterSettings.Indent := TRUE;

xmlWriter := xmlWriter.Create('c:\temp\nav.xml', xmlWriterSettings);
xmlWriter.WriteStartElement('Items', 'http://mynamspace');

FOR i := 1 TO 200000 DO BEGIN

  xmlWriter.WriteStartElement('InventoryRecord');

  xmlWriter.WriteElementString('ItemNo', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo2', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo3', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo4', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo5', 'ITM000001');
  xmlWriter.WriteElementString('ItemNo6', 'ITM000001');

  xmlWriter.WriteEndElement();

END;

xmlWriter.WriteEndElement();
xmlWriter.Flush();

xmlWriter.Close();

The C# code:
using System;
using System.Xml;

namespace ConsoleApplication4
{
    class Program
    {
        static void Main(string[] args)
        {
            XmlWriterSettings settings = new XmlWriterSettings();
            settings.Indent = true;

            using (XmlWriter writer = XmlWriter.Create(@"c:\temp\inventory.xml", settings))
            {
                writer.WriteStartElement("Items", "http://mynamspace");
                for (int i = 0; i < 200000; i++)
                {
                    writer.WriteStartElement("InventoryRecord");
                    writer.WriteElementString("ItemNo", "ITM000001");
                    writer.WriteElementString("ItemNo2", "ITM000001");
                    writer.WriteElementString("ItemNo3", "ITM000001");
                    writer.WriteElementString("ItemNo4", "ITM000001");
                    writer.WriteElementString("ItemNo5", "ITM000001");
                    writer.WriteElementString("ItemNo6", "ITM000001");
                    writer.WriteEndElement();
                }
                writer.WriteEndElement();
                writer.Flush();
            }

            Console.WriteLine("done");
            Console.ReadLine();

        }
    }
}


thanks & best regards
Tobias

Comments

  • CorinaSCorinaS Member Posts: 3
    Hi Tobias,

    First of all, thank you for posting your feedback!
    We are sorry to hear you ran into this issue and feel this way.

    I understand that you have also logged a case with us in this regards, and my colleague is currently working on it.
    His feedback to me was that he was able to reproduce and is in the process of creating a CR with the DEV team to check if this is a problem.

    We will contact you back via our support case as soon as we have updates, but please do let me know if there is anything more I can assist with at this point.

    Thanks!
    Corina Soare

    Support Engineer
    Microsoft Dynamics NAV
    Follow EMEA Microsoft Dynamics NAV Support on Twitter @MSDynNAVSupport!
  • VjekoVjeko Member Posts: 55
    I don't have time to verify my claims here, but the problem is that your C# and C/AL code are not exactly equivalent. In C#, you put your XmlWriter into a using block, and the using block is automatically calling the Dispose method of the IDisposable interface. In your C/AL code, you never did that. That's why you end up having a lot of unmanaged resources (in other words - a memory leak) even after your session closes.

    Furthermore, the XmlWriter class allocates the unmanaged resources, making it mandatory to clean them up, using the Dispose method.

    A quick fix could be to make the XmlWriter a local variable in a function, and do all the calls locally. If Microsoft documentation is still valid, they will call the Dispose method for you if a variable is an IDisposable. This was implicit in .NET Interop in NAV 2009 R2, when all local DotNet variables were used in a using block in C#. In NAV 2013 it's a bit different, so maybe this call to Dispose happens somewhere else, explicitly, I can't check that.

    Another quick fix would be to call XmlWriter.Dispose immediately after you call XmlWriter.Close - that will clean up the resources. However, calling Dispose directly can have unexpected side effects, because NAV might be calling it for you if it is a local variable - check the first solution, then the second one.

    I hope this helps.
    (Co-)author of "Implementing Microsoft Dynamics NAV 2009"
    http://vjeko.com/
  • toortoor Member Posts: 52
    What I've got from MS:
    Reasons lay in situation that c\al calls in the loop .net method.
    [...]
    so when we have big loops, we have situation when memory is not released inside loop and increase with every step
    [...]
    Developer proposed workaround to put method WriteElementString to separated local procedure, then “out of scope” comes exactly when NAV goes out of procedure.

    With this information I've created the two functions below.
    Memory consumption is now pretty low, CPU consumption is higher (because the XmlWriter will be created and disposed very often, I think), but that's not a problem in my case.

    In the first function I initialize a global stream variable and write the "xml-frame" with a global XmlWriter.
    Inside the loop I call the second function.
    The second function creates a local XmlWriter an writes some records to the global stream.
    When the second function finishes, the "local resources" will be released.

    It's weird, to write to the same stream from multiple XmlWriters, and you can end up with invalid xml pretty fast, but as long as it works I'm fine with this.
    WriteXml()
    Ns := 'http://mynamespace';
    Stream := IoFile.Create('c:\temp\test.xml');
    
    XmlWriterSettings := XmlWriterSettings.XmlWriterSettings();
    XmlWriterSettings.CloseOutput := TRUE;
    XmlWriterSettings.OmitXmlDeclaration := FALSE;
    XmlWriterSettings.Encoding := Enc.UTF8;
    
    XmlWriter := XmlWriter.Create(Stream, XmlWriterSettings);
    XmlWriter.WriteStartElement('InventoryRecords', Ns);
    XmlWriter.Flush();
    
    XmlWriterSettings.CloseOutput := FALSE;
    XmlWriterSettings.OmitXmlDeclaration := TRUE;
    XmlWriterSettings.ConformanceLevel := ConformanceLevel.Fragment;
    
    FOR i := 1 TO 5000 DO BEGIN
      WriteRecords(i=1, 500);
    END;
    
    XmlWriter.WriteEndElement();
    XmlWriter.Flush();
    XmlWriter.Close();
    
    Stream.Close();
    Stream.Dispose();
    
    WriteRecords(useGlobalWriter : Boolean;numOfRecsToWrite : Integer)
    IF NOT useGlobalWriter THEN BEGIN
      xmlWriterLoc := xmlWriterLoc.Create(Stream, XmlWriterSettings);
    END ELSE BEGIN
      xmlWriterLoc := XmlWriter;
    END;
    
    FOR i := 1 TO numOfRecsToWrite DO BEGIN
    
      xmlWriterLoc.WriteStartElement('InventoryRecord');
    
      xmlWriterLoc.WriteElementString('ItemNo', 'ITM000001');
      xmlWriterLoc.WriteElementString('ItemNo2', 'ITM000001');
      xmlWriterLoc.WriteElementString('ItemNo3', 'ITM000001');
      xmlWriterLoc.WriteElementString('ItemNo4', 'ITM000001');
      xmlWriterLoc.WriteElementString('ItemNo5', 'ITM000001');
      xmlWriterLoc.WriteElementString('ItemNo6', 'ITM000001');
    
      xmlWriterLoc.WriteEndElement();
    
    END;
    
    xmlWriterLoc.Flush();
    
    IF NOT useGlobalWriter THEN BEGIN
      xmlWriterLoc.Close();
      CLEAR(xmlWriterLoc);
    END;
    
  • mdPartnerNLmdPartnerNL Member Posts: 802
    so you keep on using 1 global var?
  • toortoor Member Posts: 52
    I don't understand you question, sorry.

    I'm using one global stream variable, one global XmlWriter and a lot of local XmlWriter.
    All the XmlWriter write to the global stream.

    Because the local XmlWriter goes out of scope everytime the function ends, there happen some kind of memory cleanup.
  • mdPartnerNLmdPartnerNL Member Posts: 802
    Ok, now I understand, thx.
Sign In or Register to comment.