The Realm of the Verbal Processor

Out of Memory Error – Boot.ini 3GB Switch

Recently while working at a client I was installing the BITS 2.5 update for Server 2003 (as part of the prereqs for ConfigMgr), but when the server rebooted at the end of the update it essentially died. No more Remote Desktop connection…the server wasn’t even pingable. I didn’t have access to the client’s datacenter, so I was pretty much hamstrung until they could get me access to the console to look at the log files. Once I finally got to the server (booted into Safe Mode), the log files were chugging full of red. Some of the log entries were:

The Application log had the following (key messages in red):

Event ID:    4613
Description:
The COM+ Event System detected an unexpected error from a Win32 API call at line 917 of d:\nt\com\complus\src\events\tier2\notify.cpp.  A call to PostThreadMessage failed with error code : "The operation completed successfully. " 

Event ID:    4613
Description:
The COM+ Event System detected an unexpected error from a Win32 API call at line 558 of d:\nt\com\complus\src\events\tier2\notify.cpp.  A call to MsgWaitForMultipleObjects failed with error code 8: "Not enough storage is available to process this command. " 

Event ID:    8193
Computer:    SERVERNAME
Description:
Volume Shadow Copy Service error: Unexpected error calling routine IEventSystem::Store.  hr = 0x80040206.

Event ID:    1508
Description:
Windows was unable to load the registry. This is often caused by insufficient memory or insufficient security rights.

DETAIL – The system has attempted to load or restore a file into the registry, but the specified file is not in a registry file format.  for C:\Documents and Settings\accountname\ntuser.dat

Event ID:    1502
Description:
Windows cannot load the locally stored profile. Possible causes of this error include insufficient security rights or a corrupt local profile. If this problem persists, contact your network administrator.

DETAIL – The system has attempted to load or restore a file into the registry, but the specified file is not in a registry file format.

In the System Log was the following string of events:

Event Source:    NtServicePack
Event ID:    4377
Description:
Windows Server 2003 Hotfix KB923845 was installed.

Event ID:    1074
Description:
The process svchost.exe has initiated the restart of computer SERVERNAME on behalf of user NT AUTHORITY\SYSTEM for the following reason: No title for this reason could be found
Reason Code: 0x80070020
Shutdown Type: restart
Event ID:    3113
Description:
Initialization failed because the requested service redirector could not be started.

Event ID:    2508
Description:
The server service was unable to load the server driver.

Event Type:    Information
Event Source:    Application Popup
Event ID:    26
Description:
Application popup: wscript.exe – Application Error : The application failed to initialize properly (0xc0000142).

Event Type:    Information
Event Source:    Application Popup
Event ID:    26
Description:
Application popup:  : \SystemRoot\system32\DRIVERS\srv.sys failed to load

Event Type:    Error
Event Source:    Application Popup
Event ID:    333
Description:
An I/O operation initiated by the Registry failed unrecoverably. The Registry could not read in, or write out, or flush, one of the files that contain the system’s image of the Registry.

The key messages above are the ones related to insufficient memory or “not enough storage”. Now this server has 12GB of RAM. That should not be a problem. Then I remembered an email from the client’s server team where they told me that they had set the /3GB switch in the boot.ini file. I did a little digging on that switch and came across articles here, here, here, here here and here.

Essentially it boils down to some common misunderstanding about what the 3GB switch does. The client has a requirement to use Server 2003 32-bit for this installation. Normally, 32-bit Windows operating systems have a 4GB RAM limit. Often people mistakenly believe that the /3GB switch expands the use of RAM above the 4GB limit. In actuality, the /PAE switch is used for this. Utilizing that switch Server 2003 Enterprise SP2 will then support 32GB of RAM. So…what does the /3GB switch do?

Paraphrasing one of the articles linked above, 32-bit Windows OSes implement a virtual memory space that is limited to 4GB. Normally this is split into two sections…2GB for Kernel and 2GB for User-mode processes. The /3GB switch makes 3GB available for user-mode processes, but leaves only 1GB available for Kernel.

This appears to be exactly what happened. Because the 3GB switch was set and only 1GB was available for kernel operations, the system essentially ran out of memory for completing the installation of BITS. Once I removed the 3GB switch (and left the PAE switch intact) and rebooted the system, it came back up.

December 2, 2009 - Posted by Jarvis | ConfigMgr