Rank: Advanced Member Groups: Member
Joined: 7/14/2014 Posts: 40
|
I'm working on running a large scale test of EO.Pdf to evaluate its potential use in our systems. This test is primarily to get a sense of performance, backlogs, and load demand that the component would be subjected to as part of its normal daily usage in our systems. Secondary purpose of the test is assessing how many documents fail to convert (and why).
The test application is running 4 "worker threads" that are draining an internal queue and calling EO.Pdf for conversion and one thread that is responsible for populating the queue based on the input file contents (which specifies the files and time when it should be submitted). It is running on 4 CPU Windows 2008R2 x64 VM w/16GB of memory using .NET 4.5.1 and EO.Pdf version 5.0.85.2 (EO.Total v109) - with no license key (we're still doing trial). Based on my other thread (http://www.essentialobjects.com/forum/postst8426_EOPDF-Out-of-Memory-and-Zombie-Rundll32exe.aspx) I'm running each call to EO.Pdf in a separate child AppDomain which is setup before the call and destroyed after the call (I actually meant to use one of the more refined approaches but left the wrong approach enabled in the test application when I ran it).
My first full test set is 4768 HTML documents - this represents a heavier than average volume day's worth of documents. What I experienced during the first run of the test was that after successfully converting 1924 HTML documents (about 2 hours and 13 minutes into the test), it suddenly started failing to convert filings and failed on every single test document after that after 2-3 secs into each conversion.
000157093113000020|2014-07-15T19:42:03.3705|0|0.00:00:00.0000|0|0.00:00:02.1143|0.00:00:02.1143|False|Thread 4: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Value cannot be null. Parameter name: hdc ---> System.ArgumentNullException: Value cannot be null. Parameter name: hdc at System.Drawing.Graphics.FromHdc(IntPtr hdc) at EO.Pdf.Internal.d2.a(BinaryReader A_0, Size A_1, IntPtr A_2, Int32 A_3) at EO.Pdf.Internal.d2..ctor(BinaryReader A_0, Size A_1, Int32 A_2) at EO.Pdf.Internal.hb..ctor(BinaryReader A_0) at EO.Pdf.Internal.be.a(Byte[] A_0, Single A_1, Single A_2, Single A_3, Single& A_4, Double A_5, Double A_6) at EO.Pdf.Internal.lu.a(String A_0, Single& A_1) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44 000110465913063818|2014-07-15T19:42:02.3705|0|0.00:00:00.0000|0|0.00:00:03.1143|0.00:00:03.1143|False|Thread 5: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Arithmetic operation resulted in an overflow. ---> System.OverflowException: Arithmetic operation resulted in an overflow. at EO.Pdf.Internal.en.a(IntPtr A_0) at EO.Pdf.Internal.en.a(j4 A_0) at EO.Pdf.Internal.en.b(j4 A_0) at EO.Pdf.Drawing.PdfFont..ctor(j4 A_0, Single A_1) at EO.Pdf.Internal.fo.b(kb A_0) at EO.Pdf.Internal.fo.a(j8[] A_0) at EO.Pdf.Internal.fo..ctor(lu A_0, Boolean A_1, Int32 A_2, kv A_3, Int32 A_4, Single A_5, Rectangle A_6, PdfPage A_7, d2 A_8, Int32 A_9, Graphics A_10, Dictionary`2 A_11) at EO.Pdf.Internal.lu.a(PdfPage A_0, Boolean A_1, Int32 A_2, Rectangle A_3, kv A_4, hb A_5, Int32& A_6, Graphics A_7) at EO.Pdf.Internal.lu.a(Int32 A_0, HtmlToPdfResult A_1, Int32& A_2) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44 000095015613000066|2014-07-15T19:42:02.3705|0|0.00:00:00.0000|0|0.00:00:03.1993|0.00:00:03.1993|False|Thread 6: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Value cannot be null. Parameter name: hdc ---> System.ArgumentNullException: Value cannot be null. Parameter name: hdc at System.Drawing.Graphics.FromHdc(IntPtr hdc) at EO.Pdf.Internal.d2.a(BinaryReader A_0, Size A_1, IntPtr A_2, Int32 A_3) at EO.Pdf.Internal.d2..ctor(BinaryReader A_0, Size A_1, Int32 A_2) at EO.Pdf.Internal.hb..ctor(BinaryReader A_0) at EO.Pdf.Internal.be.a(Byte[] A_0, Single A_1, Single A_2, Single A_3, Single& A_4, Double A_5, Double A_6) at EO.Pdf.Internal.lu.a(String A_0, Single& A_1) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44 000153261813000006|2014-07-15T19:42:03.3705|0|0.00:00:00.0000|0|0.00:00:02.2237|0.00:00:02.2237|False|Thread 3: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Value cannot be null. Parameter name: hdc ---> System.ArgumentNullException: Value cannot be null. Parameter name: hdc at System.Drawing.Graphics.FromHdc(IntPtr hdc) at EO.Pdf.Internal.d2.a(BinaryReader A_0, Size A_1, IntPtr A_2, Int32 A_3) at EO.Pdf.Internal.d2..ctor(BinaryReader A_0, Size A_1, Int32 A_2) at EO.Pdf.Internal.hb..ctor(BinaryReader A_0) at EO.Pdf.Internal.be.a(Byte[] A_0, Single A_1, Single A_2, Single A_3, Single& A_4, Double A_5, Double A_6) at EO.Pdf.Internal.lu.a(String A_0, Single& A_1) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44
I thought maybe I was pushing the application too hard and was not giving .NET (or Windows OS) enough time to do GC and other housekeeping (due to seeing "hdc" in the exception). I had compressed out all idle gaps in the simulation to expedite the run - if all workers are idle then the queuing thread immediately jumps to the next time slot of files and starts injecting them into the queue. Being 2+ hours into the test I thought this unlikely (as I would have expected it to die much sooner if I was really preventing GC) however I modified the test application to allow idle gaps of 10 seconds or less between batches to remain and then re-ran the the test. (10 second is a long time on a modern multi-CPU server...)
The second run of the test (same document set) starting failing again after 1924 HTML documents (about 4 hours and 8 minutes into the test - compared to the first run you can see how much time those idle gaps really added up). The first document to report failure is different but it's at the same point/files in the test as the last run.
000110465913063818|2014-07-16T04:50:47.7424|0|0.00:00:00.0000|0|0.00:00:02.8819|0.00:00:02.8819|False|Thread 5: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Arithmetic operation resulted in an overflow. ---> System.OverflowException: Arithmetic operation resulted in an overflow. at EO.Pdf.Internal.en.a(IntPtr A_0) at EO.Pdf.Internal.en.a(j4 A_0) at EO.Pdf.Internal.en.b(j4 A_0) at EO.Pdf.Drawing.PdfFont..ctor(j4 A_0, Single A_1) at EO.Pdf.Internal.fo.b(kb A_0) at EO.Pdf.Internal.fo.a(j8[] A_0) at EO.Pdf.Internal.fo..ctor(lu A_0, Boolean A_1, Int32 A_2, kv A_3, Int32 A_4, Single A_5, Rectangle A_6, PdfPage A_7, d2 A_8, Int32 A_9, Graphics A_10, Dictionary`2 A_11) at EO.Pdf.Internal.lu.a(PdfPage A_0, Boolean A_1, Int32 A_2, Rectangle A_3, kv A_4, hb A_5, Int32& A_6, Graphics A_7) at EO.Pdf.Internal.lu.a(Int32 A_0, HtmlToPdfResult A_1, Int32& A_2) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44 000095015613000066|2014-07-16T04:50:47.7424|0|0.00:00:00.0000|0|0.00:00:02.9727|0.00:00:02.9727|False|Thread 3: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Value cannot be null. Parameter name: hdc ---> System.ArgumentNullException: Value cannot be null. Parameter name: hdc at System.Drawing.Graphics.FromHdc(IntPtr hdc) at EO.Pdf.Internal.d2.a(BinaryReader A_0, Size A_1, IntPtr A_2, Int32 A_3) at EO.Pdf.Internal.d2..ctor(BinaryReader A_0, Size A_1, Int32 A_2) at EO.Pdf.Internal.hb..ctor(BinaryReader A_0) at EO.Pdf.Internal.be.a(Byte[] A_0, Single A_1, Single A_2, Single A_3, Single& A_4, Double A_5, Double A_6) at EO.Pdf.Internal.lu.a(String A_0, Single& A_1) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44 000157093113000020|2014-07-16T04:50:48.7424|0|0.00:00:00.0000|0|0.00:00:02.0830|0.00:00:02.0830|False|Thread 4: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Value cannot be null. Parameter name: hdc ---> System.ArgumentNullException: Value cannot be null. Parameter name: hdc at System.Drawing.Graphics.FromHdc(IntPtr hdc) at EO.Pdf.Internal.d2.a(BinaryReader A_0, Size A_1, IntPtr A_2, Int32 A_3) at EO.Pdf.Internal.d2..ctor(BinaryReader A_0, Size A_1, Int32 A_2) at EO.Pdf.Internal.hb..ctor(BinaryReader A_0) at EO.Pdf.Internal.be.a(Byte[] A_0, Single A_1, Single A_2, Single A_3, Single& A_4, Double A_5, Double A_6) at EO.Pdf.Internal.lu.a(String A_0, Single& A_1) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44 000153261813000006|2014-07-16T04:50:48.7424|0|0.00:00:00.0000|0|0.00:00:02.2442|0.00:00:02.2442|False|Thread 6: Exception caught: EO.Pdf.HtmlToPdfException: Convertion failed. Value cannot be null. Parameter name: hdc ---> System.ArgumentNullException: Value cannot be null. Parameter name: hdc at System.Drawing.Graphics.FromHdc(IntPtr hdc) at EO.Pdf.Internal.d2.a(BinaryReader A_0, Size A_1, IntPtr A_2, Int32 A_3) at EO.Pdf.Internal.d2..ctor(BinaryReader A_0, Size A_1, Int32 A_2) at EO.Pdf.Internal.hb..ctor(BinaryReader A_0) at EO.Pdf.Internal.be.a(Byte[] A_0, Single A_1, Single A_2, Single A_3, Single& A_4, Double A_5, Double A_6) at EO.Pdf.Internal.lu.a(String A_0, Single& A_1) at EO.Pdf.Internal.lu.d() --- End of inner exception stack trace --- at EO.Pdf.HtmlToPdfException.b(Exception A_0) at EO.Pdf.Internal.lu.d() at EO.Pdf.HtmlToPdfSession.RenderAsPDF(PdfDocument doc) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, PdfDocument doc, HtmlToPdfOptions options) at EO.Pdf.HtmlToPdf.ConvertUrl(String url, String pdfFileName, HtmlToPdfOptions options) at FeedReplay.Worker.PdfConverter.Process(Request req) in c:\Projects\Source\ThrowAways\FeedReplay\Worker.cs:line 44
I pulled 136 entries out of the test's input file that led up to these files, included them, and a few files after to create a mini-test to see if I could figure out if there was something in particular about these files that would cause a crash. The test however ran to completion w/out any conversion failures. The files of this mini-test are submitted in the same order and spacing as in the full test. I reran the test and noted at one point that the memory of the test application itself (not the Rundll32.exe child processes) shot up pretty high (from 350MB to about 2.3GB) for about 2 minutes during the middle of the test and then returned to normal ranges (back to 350MB) well before the test was over. Since the test application is a 64-bit application, the server has 16GB of memory, and this is essentially the only thing running on the server while I'm concerned/puzzled by the spike I don't see it as being a smoking gun for the sudden conversion failure situation that I see during the full tests. I ran the mini-test again and captured a number of performance counters using perfmon (which I can provide). None of the runs failed to convert any of the filings however.
|