This project has moved. For the latest updates, please go here.

Out of Memory Exception

Nov 6, 2013 at 10:36 PM
Edited Nov 6, 2013 at 10:39 PM
I am testing a large (29MB) file. The parser is EdiFabric.Definitions.X12_004010_834.M_834.

Environment: Win 7 Pro SP1, 8GB RAM

Inner Exception:
Exception of type 'System.OutOfMemoryException' was thrown.

Stacktrace:

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at System.Xml.XmlTextWriter.WriteStartElement(String prefix, String localName, String ns)
at System.Xml.Linq.ElementWriter.WriteStartElement(XElement e)
at System.Xml.Linq.ElementWriter.WriteElement(XElement e)
at System.Xml.Linq.XElement.WriteTo(XmlWriter writer)
at System.Xml.Linq.XElement.System.Xml.Serialization.IXmlSerializable.WriteXml(XmlWriter writer)
at System.Xml.Serialization.XmlSerializationWriter.WriteSerializable(IXmlSerializable serializable, String name, String ns, Boolean isNullable, Boolean wrapped)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterINTERCHANGE.Write5_GROUP(String n, String ns, GROUP o, Boolean isNullable, Boolean needType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterINTERCHANGE.Write7_INTERCHANGE(String n, String ns, INTERCHANGE o, Boolean isNullable, Boolean needType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationWriterINTERCHANGE.Write8_INTERCHANGE(Object o)

Test Code:
        string path = @"C:\LargeFile.txt";
        Stream s = File.OpenRead(path);
        var interchange = ediParser.ToInterchange<INTERCHANGE>(s, null);
        var xml = ediParser.Serialize(interchange);
Nov 6, 2013 at 11:52 PM
Greg,

have you had any luck converting smaller files? any Idea at what point it breaks?

Are you using 64 or 32 bit and what are you compiling for? (if AnyCpu try 64Bit)

Is it possible to put the file somewhere public (ie dropbox)

Regards,
Mike
Nov 7, 2013 at 5:20 PM
Small files work fine. Sorry, I can't send the files - they contain confidential info. Compiling as x64 runs without exceptions. Here is where it breaks on an x86 compile:

The class is EdiFabric.Framework.Helpers.EdiHelper:
    public static XElement Serialize<T>(T instance)
    {
        var serializer = new XmlSerializer(typeof(T), typeof(T).Namespace);
        using (var ms = new MemoryStream())
        {
            serializer.Serialize(ms, instance); //<-- the exception is thrown here
            ms.Position = 0;
            return XElement.Load(ms);
        }
    }
Coordinator
Nov 8, 2013 at 7:47 AM
Hi Greg,

I believe this is due to the large EDI file you are trying to parse. Never seen edi file that large, or anything > 20kb. When edi file is converted to xml, the size may grow x 10 times, so bear in mind that .NET would try to load at least ~290 mb in memory. If there are cross references inside your object, .NET may duplicate those as well, which will lead to additional increase.

You can change the serialize method to not use memory stream, but use streaming instead with xmltextwriter for example or try any other serialization that is better suited to support large files.

Regards,
Kamen