Different default stack size across platforms

[markusmobius]

I am using HtmlAgilityPack within a heavily threaded application to process html files. Occassionally there are badly misformed HTML files with many nested child objects which cause a stack overflow error.

HtmlAgilityPack has an option to limit child nodes:

                            //get the title
                            try
                            {
                                var htmlDoc = new HtmlDocument();
                                htmlDoc.OptionMaxNestedChildNodes = 200; //this prevents stack overflow
                                htmlDoc.LoadHtml(page.html);
                                var t = htmlDoc.DocumentNode.Descendants("title").FirstOrDefault();
                                if (t != null)
                                {
                                    title = System.Net.WebUtility.HtmlDecode(t.InnerText);
                                    //get rid of tabs, linebreaks
                                    title = title.Replace('\r', ' ').Replace('\t', ' ').Trim().Replace("\n", "###");
                                }
                            }
                            catch (Exception e)
                            {
                                //ignore title
                            }

[This code is running in one of 16 threads. The entire program is running on a 16-core Ubuntu or Azure VM.]

This fixed the error on Ubuntu. However, on Windows I keep getting stack overflow error. Are there differences in stack management between Ubuntu and Windows that could cause this?

Update: I fixed it by manually increasing maxStackSize in the Thread constructor to 16MB. My understanding is that it's 1MB for 32bit and 4MB for 64 bit processes. It it possible that the (a) the stack is used differently on Linux and Window netcore runtime or (b) the default stack size values are different on Linux and Windows?

[Dotnet-GitSync-Bot]

I couldn't add an area label to this Issue.

Checkout this page to find out which area owner to ping, or please add exactly one area label to help train me in the future.

[scalablecory]

You will probably want to file a bug with the Html Agility Pack project to see if a recursionless algorithm is possible, but we should be able to answer the default stack size question.

CC @jkotas

[janvorli]

The default stack size for a thread is platform specific and it also differs between primary thread and the secondary threads. The primary thread size limit is set when an application by the OS / OS specific runtime libraries. E.g. on Linux, it can be set by "ulimit -s" command, on many Linux distros it is 8MB by default. We can set the default stack size for the secondary threads though and we do that. If a stack size was specified in the Thread constructor, we use that value (after applying minimum stack limit imposed by the OS / OS specific runtime). If it was not specified, we use a default value that was set by the COMPlus_DefaultStackSize env variable in case it was set (please note the env variable is set to a hexadecimal value without any prefix / suffix, so e.g. 10000 means 1MB). If it was not set, we use whatever default the OS / OS specific runtime sets. There is also DEFAULT_STACK_SIZE host configuration property that can be used by custom hosts.
The only special case is Linux distros based on MUSL library which sets the default secondary stack size to a ridiculously low value of 80kB, so we override it to 1.5MB instead). The default value on glibc based linux distros is based on the primary thread stack size limit. On OSX, the default secondary stack size is 512KB. On Windows, we set the default stack size using a compiler option when building the host executables to 1.5MB.