Sep 102014
 

Issue

Recently a colleague asked this question. They had a customer who was experiencing a heap corruption so as expected we enabled PageHeap but there was a catch. The application had to run for a long time (around 30 days) in order to reproduce the crash and we had no idea what’s causing the crash.

How do we enable PageHeap?

We can enable standard PageHeap using following command run from an admin command prompt: gflags /p /enable ImageFileName
To enable full PageHeap use the following: gflags /p /enable ImageFileName /full 

(MSDN) Use care in interpreting the Enable page heap check box for an image file in the GFlags dialog box. It indicates that page heap verification is enabled for an image file, but it does not indicate whether it is full or standard page heap verification. If the check results from selecting the check box, then full page heap verification is enabled for the image file. However, if the check results from use of the command-line interface, then the check can represent the enabling of either full or standard page heap verification for the image file.

Why application hung?

So customer enabled PageHeap and went home. Came back next day to see that the application has stopped responding and is hung. The application hung apparently after enabling PageHeap and as we know of PageHeap: every allocation is paged to the page file. So guess why would the hang take place? PageFile size!

Resolution

The customer had set PageFile to its default size which apparently was not enough in this case. We suggested to increase the PageFile size and the hang went away. This resolved the issue. Note that if you enable PageHeap and then go home no matter what’s the PageFile size eventually the result will be unpredictable as the PageFile size is finite. You might need to tweak your PageHeap settings and make it per module or non-full standard page heap.

Conclusion

Please note there are different variants of PageHeap. In this case we needed a full PageHeap so please note this will be pretty heavy on the PageFile.

Sep 032014
 

Issue

Recently had a customer who was facing this issue. His code looked as follows (assuming COM’s initialized) which apparently is trying to setup an ActiveX control…

HWND hWnd = CreateWindow(_T(ATLAXWIN_CLASS),
_T(“SysMon”),
WS_CHILD,
0, 10, 100, 200,
hwndParent,
NULL,
hInstance,
NULL);

As soon as above code executes the application crashes.

image

Why loading ActiveX failed?

Callstack for the crash looks as follows…

ConsoleApplication1.exe!ATL::CComPolyObject<ATL::CAxHostWindow>::CComPolyObject<ATL::CAxHostWindow>
ConsoleApplication1.exe!ATL::CComCreator<ATL::CComPolyObject<ATL::CAxHostWindow> >::CreateInstance
ConsoleApplication1.exe!AtlAxCreateControlLicEx
ConsoleApplication1.exe!AtlAxCreateControlLic
ConsoleApplication1.exe!ATL::AtlAxWindowProc
user32.dll!UserCallWinProcCheckWow
user32.dll!DispatchClientMessage
user32.dll!__fnINLPCREATESTRUCT
ntdll.dll!KiUserCallbackDispatch    Unknown
user32.dll!ZwUserCreateWindowEx    Unknown
user32.dll!VerNtUserCreateWindowEx
user32.dll!CreateWindowInternal
user32.dll!CreateWindowExW
ConsoleApplication1.exe!wmain
ConsoleApplication1.exe!__tmainCRTStartup
ConsoleApplication1.exe!wmainCRTStartup
kernel32.dll!BaseThreadInitThunk
ntdll.dll!RtlUserThreadStart

 

Autos debug window showed following…

image

Resolution

The highlighted elements gave ample hint. This is a console application with no ATL boilerplate code generated by project wizard for this project. Ideally ATL requires a global CComModule instance to initialize itself. This is how the CComModule constructor looks like…

CComModule()
{
// Should have only one instance of a class
// derived from CComModule in a project.
ATLASSERT(_pModule == NULL);
<span style="background-color: #ffff00;">_pModule = this;</span>
#if !defined(_ATL_NATIVE_INITIALIZATION)
#pragma warning(push)  // disable 4483
#pragma warning(disable:4483)
using namespace __identifier("<AtlImplementationDetails>");
#pragma warning(pop)  // disable 4483
ATLASSERT(ComModuleInitialized == false);
// If ComModuleHelper.Module == NULL it mean that DllMain has not been called, so we assume CComModule lives in
// an exe and not in a dll
if (ComModuleHelper.Module != NULL)
{
ATLASSERT(ComModuleHelper.Module == this);
_DllMain(ComModuleHelper.Instance, DLL_PROCESS_ATTACH, NULL, ComModuleHelper.ObjectMap, ComModuleHelper.LibraryId);
}
ComModuleInitialized = true;
#endif
}

This function further initializes _pModule instance to point to this global instance of ours eventually. So after we declare CComModule global instance this is how _pAtlModule (it points to our  global instance) will look after instantiation of our global CComModule…

image

Crash resolved.

Aug 282014
 
What’s DEVPATH

Being a .net developer you might have wondered if at all there is a way to tell .net assembly binding and loading mechanism (fusion) to look for your assemblies in your shared assembly folder as well. This will be really helpful if your assembly is used with multiple applications so that if they fail to locate an assembly in the appbase or privatepath’s then they start looking for it in your ‘bin’ folder. This prevents you from frequently installing and uninstalling the assembly to GAC for testing purpose.

This is exactly what DEVPATH environment variable does.

How to set DEVPATH

Following example shows you how to setup just need to set assembly look up path to this variable. For e.g.

set DEVPATH=”c:\sharedassemblies\”

To enable DEVPATH look up you’ll also need to modify machine.config file, in my case its located here: C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config

<configuration>

<runtime>
<developmentMode developerInstallation=”true”/>
</runtime>

</configuration>

Verify DEVPATH is used

So next time when application runs and starts looking for your assembly it will for sure look in the above path. If fusion logger is enabled and if you open the log file for your assembly you should see something like this…

===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\myapp.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Found assembly in DEVOVERRIDE path C:\sharedassemblies\Test.DLL

Recently had a customer who asked this question and this looks like to have helped them so thought of sharing this out with you folks as well.

Watch outs
  1. Please be very aware that you’ve set this feature on. You could end up having erratic behavior if somehow DEVPATH has a path containing unsupported path chars like parentheses or if DEVPATH is set to empty.
  2. Please only use this feature in development environments as this feature by passes all normal assembly lookup features as fusion looks up DEVPATH first if enabled.
  3. Please note fusion will look up DEVPATH as if its an AppBase folder so be aware that it will look up subdirectories as well in this DEVPATH. Also all valid assembly name look ups will be done, for e.g. .dll, .exe etc.
Aug 272014
 
What is a Symbol?

When you compile your executable the compiler generates debugging symbol information for every file it compiles and then linker assembles all of these symbol information into one file called PDB file or the Program DataBase file. Every variable, function in your application code can be called as a symbol which implies there will be private and public symbols.

This generated .pdb file’s full local path is embedded into the executable file. This comes in handy while you are debugging this application on your development machine. The PDB file will be picked up by the debugger since full path to the PDB is embedded into the executable by the linker. This will help the debugger figure, line numbers, file names, callstacks, local and global vars.

Why do we need Symbols?

Symbol file or .pdb file contains information which are actually not needed when you run your application but these come in handy when debugging application for bugs. Without pdb files or symbol files figuring out bugs or exact callstacks will be a pain on Windows. If that’s the case you might ask then why is this information not embedded into the executable? The answer is symbols are not always needed hence they are dumped into a separate file called .pdb so that you can debug when needed and also you can choose who see’s what symbols in turn making it hard for people to reverse engineer your code.

What does a PDB file contain?

They can contain a variety of information. For e.g.

  • Source code information: Line numbers, file names
  • Variables: Global and Local variables mapped to their addresses
  • Function names mapped to their addresses
  • FPO information to get correct call stack.
  • etc

Windows Debugger installation contains utilities to check out a PDB file namely: symchk, agestore, symstore, pdbcopy etc.

Public and Private Symbols

When linker generates a PDB file it contain both private and public debugging symbol information. Of course you can configure what it generates in the linker property pages.

Private symbol data contains following (mostly)

  • Global and Local Variables.
  • Functions
  • All user defined types.
  • Line number and source file information.

Public symbol table contains following…

  • Functions (just the address)
  • Global variables that are visible across obj files.

As you might have inferred private symbol files will be bigger in size compared to public symbols files. Also since private symbol file contains public symbols information as well, we can generate a separate public symbol file from this private symbols file. We use a tool called pdbcopy.exe for this purpose, comes with the windows debugger installation.

Symbol Path

So how do we tell the debugger where to look for symbols. One of my favorites is to use the environment variable_NT_SYMBOL_PATH. This variable provides us the flexibility to specify cache directories for downloaded symbols, we can even specify per symbol server cache directory.

Following value for _NT_SYMBOL_PATH downloads symbols from the server and puts into C:\Symbols folder.

cache*c:\Symbols;SRV*http://symbolserver;srv*http://anotherserver;srv*http://onemoreserver

Following value for _NT_SYMBOL_PATH downloads symbols from the server and puts into C:\Symbols folder and downloads symbols from http://anotherserver to c:\anotherserver_cache_folder.

cache*c:\Symbols;SRV*http://symbolserver;srv*c:\anotherserver_cache_folder*http://anotherserver

Windows debugger provides commands to controls symbol path, .sympath, .symfix. I use .symfix to quickly setup a default symbol path and symbols will be downloaded to a sym folder under the debugger folder. While .sympath is a cool command. If you would quickly add a symbol path to the debugger, just do the following…

.sympath+ C:\AnotherSymbolFolder
.reload

Controlling Symbol Loading in Windows Debugger

The debugger provides a command called .symopt. If we run the command without any arguments its shows our current symbol loading settings, for e.g.

Output from .symopt

So we see in this case we’ve configured to load line number information, and since we haven’t said SYMOPT_PUBLICS_ONLY, then private symbols are loaded. SYMOPT_AUTO_PUBLICS tells debugger to look for public symbols only as a last resort.

More information on symbols loading options can be found here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff558827(v=vs.85).aspx

Along with this to see a list of modules for which symbol loading failed use command ‘lme’. To get a verbose output of the symbol loading process in the debugger use “!sym noisy” to turn it off use “!sym quiet”.

Conclusion

Always keep your symbols handy. Never know when you might need them.

Aug 052014
 

Do you run into following error when trying to run a VS2013 .net application on Windows XP?

Error: Not a valid Win32 application!

This happens because the application is targeting .net framework 4.5 which is not supported on Windows XP. Target a lower framework to get your application working on XP.

Aug 052014
 

Had a customer who a opened a ticket for help on this error. The project built fine on Windows 8 but after upgrading to Windows 8.1 he started seeing above error on compilation using a 64 bit toolset.

After much troubleshooting we found the issue to be caused by a pre-release version of WDK installed on his machine. RCToolPath pointed to an older version of WDK8.1.

Customer uninstalled pre-release of WDK (to be precise: Windows Driver Kit for Windows 8.1 Preview) and installed WDK RTM. This resolved the tracker error in this case.

Blogging on this just in case you’re pulling your hair out trying to figure.

Aug 052014
 

The C Runtime is being refactored…

Note CRT is being refactored. The implementation is split into different modules. Read more here: http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.aspx 

Relevant excerpt from above blog post:

In order to unify these different CRTs, we have split the CRT into three pieces:

  1. VCRuntime (vcruntime140.dll): This DLL contains all of the runtime functionality required for things like process startup and exception handling, and functionality that is coupled to the compiler for one reason or another. We mayneed to make breaking changes to this library in the future.

  2. AppCRT (appcrt140.dll): This DLL contains all of the functionality that is usable on all platforms. This includes the heap, the math library, the stdio and locale libraries, most of the string manipulation functions, the time library, and a handful of other functions. We will maintain backwards compatibility for this part of the CRT.

  3. DesktopCRT (desktopcrt140.dll): This DLL contains all of the functionality that is usable only by desktop apps. Notably, this includes the functions for working with multibyte strings, the exec and spawn process management functions, and the direct-to-console I/O functions. We will maintain backwards compatibility for this part of the CRT.

Aug 052014
 

What are DateTime Custom Formatters?

A date and time format string defines the text representation of a DateTime or DateTimeOffset value that results from a formatting operation . It can also define the representation of a date and time value that is required in a parsing operation in order to successfully convert the string to a date and time. A custom format string consists of one or more custom date and time format specifiers. Any string that is not a standard date and time format string is interpreted as a custom date and time format string.

Custom date and time format strings can be used with both DateTime and DateTimeOffset values.

Issue with DateTime Custom Formatters?

Recently a customer opened a ticket thinking that there is an issue with the DateTime Custom Formatters. They had following piece of code.

namespace DateTimeDefect
{
 class Test
 {
   static void Main(string[] args)
   {
     DateTime dt = DateTime.Now;

     Console.WriteLine(dt.ToString("f"));
     Console.WriteLine(dt.ToString("ff"));
     Console.WriteLine(dt.ToString("fff"));

     Console.ReadLine();
   }
 }
}

Except for dt.ToString(“f”) the other two ToString calls were returning correct results. Here’s the output…

Friday, May 30, 2014 12:53 AM (Here’s the issue, customer expects a numeric value as shown below)
53
534

Here’s what MSDN says about custom formatter ‘f’, ‘ff’, ‘fff’.

“f”: The tenths of a second in a date and time value. More information: The “f” Custom Format Specifier.

Sample:
6/15/2009 13:45:30.617 -> 6
6/15/2009 13:45:30.050 -> 0

“ff”: The hundredths of a second in a date and time value. More information: The “ff” Custom Format Specifier.

Sample:
6/15/2009 13:45:30.617 -> 61
6/15/2009 13:45:30.005 -> 00

“fff”: The milliseconds in a date and time value. More information: The “fff” Custom Format Specifier.

Sample:
6/15/2009 13:45:30.617 -> 617
6/15/2009 13:45:30.0005 –> 000

 So what’s going wrong here with DateTime Custom Formatters

The issue here is that dt.ToString(“f”) is not recognized as a custom format specifier. Please read below documentation from MSDN…

A custom date and time format string consists of two or more characters. Date and time formatting methods interpret any single-character string as a standard date and time format string. If they do not recognize the character as a valid format specifier, they throw a FormatException. For example, a format string that consists only of the specifier “h” is interpreted as a standard date and time format string. However, in this particular case, an exception is thrown because there is no “h” standard date and time format specifier.

To use any of the custom date and time format specifiers as the only specifier in a format string (that is, to use the “d”, “f”, “F”, “g”, “h”, “H”, “K”, “m”, “M”, “s”, “t”, “y”, “z”, “:”, or “/” custom format specifier by itself), include a space before or after the specifier, or include a percent (“%”) format specifier before the single custom date and time specifier.

For example, “%h” is interpreted as a custom date and time format string that displays the hour represented by the current date and time value. You can also use the ” h” or “h ” format string, although this includes a space in the result string along with the hour. The following example illustrates these three format strings.

Resolution

So in this case the date time formatting method is interpreting this single character string as a standard date and time format string. To work around this, following are the options that you have…

1. Use dt.ToString(“%f”)
2. Use dt.ToString(“ f”);// add a space before ‘f’
3. Use dt.ToString(“f ”);// add a space after ‘f’

So if we change above code to…

static void Main(string[] args)
{
  DateTime dt = DateTime.Now;
  Console.WriteLine(dt.ToString("%f"));
  Console.WriteLine(dt.ToString("ff"));
  Console.WriteLine(dt.ToString("fff"));
  Console.ReadLine();
}

Now program output looks as follows…

8
87
873

Based on last value output you can verify the first two. Issue resolved.

Feb 272014
 

What does Side by Side Mean?

Side by Side implementation allows binaries co-exist side by side even with identical names. Well internally the binaries are placed into different folders based on type, name, version, processorArchitecture and publickeytoken. All these elements make up a unique folder/file name. For developers all they need is to embed a manifest into their application which I guess most of you would know. Mostly side by side dependencies are specified via #pragma comment statements or Visual Studio creates a manifest file which is then embedded into the binary using mt.exe. Visual Studio creates a manifest file in the intermediate output folder which follows the following naming convention.

AppOutputFileName.exe.embed.manifest

How is a manifest embedded into an application?

If you build your application then you should see following line in the build output window…

1>Copyright (C) Microsoft Corporation. All rights reserved.
1>Linking...
1>Embedding manifest...

This file will be embedded into the binary as a resource of type RT_MANIFEST which is just an XML file. The OS application loader will pick up this file from the application’s resource section and will figure out application dependencies from the manifest entries.

Viewing Manifest file embedded into an executable file

Viewing Manifest file embedded into an executable file

What does Side by Side Solve?

The intention was to solve dll hell but this itself went on to become a bigger hell making bloggers like me to blog on this issue. Side by Side errors are hard to figure out hence there is a dedicated tool to help figure the errors out. Side by Side concept is cool but got screwed by the numerous ifs and buts that got into this technology.

How does a Side by Side error look like?

Side by side errors are troublesome to troubleshoot. You run an MFC/CRT application on customer machine and you run into error dialogs similar to the one shown below…

image

Don’t get overawed by the error. Its quite easy to troubleshoot, hmm well.

How to troubleshoot Side by Side errors using sxstrace?

As the error message suggests let use sxstrace.exe. The usage of sxstrace is pretty easy to understand…

C:\windows\system32>sxstrace
WinSxs Tracing Utility.
Usage: SxsTrace [Options]
Options:
   Trace -logfile:FileName [-nostop]
       Enabling tracing for sxs.
       Tracing log is saved to FileName.
       If -nostop is specified, will not prompt to stop tracing.
   Parse -logfile:FileName -outfile:ParsedFile  [-filter:AppName]
       Translate the raw trace file into a human readable format and save the re
sult to ParsedFile.
       Use -filter option to filter the output.
   Stoptrace
       Stop the trace if it is not stopped before.
Example:  SxsTrace Trace -logfile:SxsTrace.etl
          SxsTrace Parse -logfile:SxsTrace.etl -outfile:SxsTrace.txt
Collecting sxstrace logs

The command usage message shows us two sample commands and that’s exactly what we’re going to try. Please make sure you’re running an elevated command prompt…

Run the following command…

C:\>SxsTrace Trace -logfile:SxsTrace.etl
Tracing started. Trace will be saved to file SxsTrace.etl.
Press Enter to stop tracing...

So now you’re in tracing mode. Go ahead and run your application which threw the side by side error. Press enter on the command prompt window once you’re done repro’ing the error, this will stop the side by side tracing that’s going on. Once you press enter the ETL trace file will be dumped into the current folder. The dumped trace file is not in human readable format…

Binary output from SxsTrace tool

Binary output from SxsTrace tool

Parsing sxstrace logs

To make it readable, we’ll need to parse this file using sxstrace tool. Run following command to do that…

C:\>SxsTrace Parse -logfile:SxsTrace.etl -outfile:SxsTrace.txt
Parsing log file SxsTrace.etl...
Parsing finished! Output saved to file SxsTrace.txt.

So now we have a text file as output. Lets open the file and find out what went wrong… Contents are as follows…

Parsed output from sxstrace

Parsed output from sxstrace

I’ve annotated the above screenshot for your convenience.

Sample location of a side by side assembly

So basically side by side works based on version of a dll. All side by side binaries go in the winsxs folder located in C:\Windows. For e.g. on my machine msvcr90d.dll is located in the following folder…

Viewing a side by side assembly

Viewing a side by side assembly

If you noticed, the folder name is made up of version number as well. So dll’s belonging to different versions are put in unique folders hence they exists “side by side” hence the name “side by side”.

So the above error means the application couldn’t find msvcr90d.dll in the above location. The way I would solve this is to create a setup project in VC9 and install the merge modules onto the target machine. Please note the dll’s are debug binaries else you could have just installed the redist’s.

Hope this will help you.