This week I’ve been beating on a problem with an OSD Task Sequence. It is a TS for deploying the Vista image that I build with my Build and Capture TS. What I was noticing was that in my Deploy TS, it was simply taking entirely too long to download the WIM file. Now granted it is a 3.5gig file, but it was taking approximately 15 minutes to download. With the gigabit connection, it should have been about a minute and a half. WHY???
I enabled command support on my boot image, ran the TS, and pulled up an F8 CMD prompt to take a look at the smsts.log. It has some errors where the HTTP transport was failing with a 405 code. After failing three times, it then switched to SMB and proceeded to download…very slowly. Here is what I saw in the log:
I have been more than a bit annoyed with the MP on my SCCM server. I have SCCM SP1 running on Server 2008. It is also using a remote SQL server that is running on Server 2003 x64 edition. In looking at the component status, it shows up as a fat red X. Looking further in, the problem is on the SMS_MP_CONTROL_MANAGER component. Every 60 minutes I have a message that comes up in that component stating:
MP Control Manager detected MP is not responding to HTTP requests. The http error is 12029.
Here is part two of my interview with Wally Mead. This section is a bit shorter than the first one…in the flow of the questions this just seemed like the best place to break it up. I plan to post the third and final part of the interview next Tuesday, so be sure to check back for the final ten minutes.
My plan is to do some more of these personal type interviews, so I posed a question yesterday…who would you like to see interviewed? Joey started off a fantastic list of people. (He also emailed me offline saying that he would be impressed if I got his whole list…heck…I’ll be impressed if I get half of his list! Although I do already have one of those lined up!) I’d love to hear from you who you would like to know more about. So…go to the post and leave a comment voting for who you would like. If the person is already mentioned, feel free to throw in another vote for them.
But…without further rambling by me…heeeeeeeerree’sss Wally!
Back in October 2007 when I was doing my original Pilot deployment of SCCM (RTM), I posted on the TechNet forums asking questions about whether putting the site database on a SQL cluster that was running on the Itanium (IA64) architecture was a supported installation scenario. Stan White replied back that both clustered as well as Itanium were supported. After installing, everything was working fine…no problems worth mentioning.
Fast forward seven months. I’m in the midst of my SCCM rebuild. I’m doing so on Server 2008 and SCCM SP1. Everything appeared to be working except that the SMS_SITE_SQL_BACKUP component refused to install on the SQL server. In the sitecomp.log file was the following message (only three lines out of a couple of hundred log entries):
Based solely on the number of hits that the Wally Mead interview has gotten, I would say that the SMS/SCCM community has a fair amount of interest in getting to know more about some of the people that we see on the forums or presenting at conferences. So…who would you like to see interviewed?
Leave a comment below to let me know who you’d like to hear from. If someone has already mentioned someone that you are interested in, leave a comment anyway…let it be like adding a vote. Don’t know if I will be able to make an interview happen with those that you are interested in hearing from, but I’ll do my best. Just give me ideas of who you are interested in getting to know.
Oh yeah, I will be working on cleaning up the second part of the Wally interview and hopefully posting it later this week. Check back to hear it…or subscribe to my RSS feed to be automatically alerted.
Over the last several months I’ve developed a relationship with Wally Mead. Wally is a Senior Program Manager at Microsoft where he works with the System Center Configuration Manager product team. He handled a bug report that I submitted about SCCM (here and here…bug was fixed in SP1) and we connected offline afterwards. He had seen in the bug submission that I work for Campus Crusade for Christ, and he has had interaction with CCC in the past.
This week Wally has been in Orlando at Tech-Ed, so I invited him to come out and let me take him on a tour of CCC’s headquarters. After the tour we sat down and Wally allowed me to interview him. Below is the first portion of the interview. I will post parts 2 and 3 sometime next week.
I’d like to also offer a public thank you to Wally for taking the time to sit down with me. It was a very enjoyable time, and if Chris ends up coming to UCF, I look forward to seeing you more often!
Another thing…who else would you like to see interviewed? Let me know on that page and I’ll see what I can do.
[Update 6-22-08: It was brought to my attention in the comments below that if you don't have the Adobe Flash Plugin on your computer, you will not see the audio player that is just below the "Enjoy!" line. You can get it here if you don't already have it.]
As part of my reinstall of my SCCM server, I have been beating on a problem for way longer than I should have…and finally got it fixed. It is related to SCCM not being able to sync with WSUS. Background: we are a small environment by SMS/SCCM standards, so we are able to work quite well with all components on the same server. The only thing that is external to “the” SCCM box is that we have SQL off-box.
When I set up WSUS/SCCM as the documentation recommended, I got some crazy errors. I have posted the relevant sections below. The key portion of the wsyncmgr.log was:
Sync failed: WSUS server not configured. Source: CWSyncMgr::DoSync
Having dealt with a similar issue before, I was pretty sure that the errors has something to do with the way that SCCM connects as Local System and the internet settings associated with the System account. In particular I had dealt with a proxy issue before that was causing similar errors. In doing my research on the problem I came across a couple of posts that proved helpful. In this one, a Microsoft guy posted that two ways to fix this were to either run IE as Local System or to use the ProxyCfg.exe tool (Server 2003). I’m on Server 2008, so I found another post that showed how to do the proxycfg stuff in Server 2008 using netsh. After doing the second option (netsh), everything looked good as far as the proxy was concerned…it wasn’t using a proxy. That wasn’t the problem.
So…what about starting IE as Local System and looking at those settings? I’ve posted before about how to run an app as Local System. The Vista option in that post works on Server 2008.
Once I did that, I went into IE settings and drilled down through Tools | Internet Options | Connections | LAN Settings. The checkbox for “Automatically detect settings” was checked. I unchecked that box. Just for good measure I rebooted. That was probably not necessary, but I wanted to see the process start fresh after a reboot…easier to track in the log files.
Once that took effect and I triggered a synchronization, SCCM was able to sync with WSUS. Matter of fact…it is still synchronizing as I finish typing up this post. All because of a stupid little checkbox.
>>>>>>>>> Relevant Log Files <<<<<<<<<
In the WCM.log file…something is definitely not right:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host~~ at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)~~ at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~ — End of inner exception stack trace —~~ at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~ at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~ at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)~~ — End of inner exception stack trace —~~ at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)~~ at Microsoft.UpdateServices.Administration.AdminProxy.GetUpdateServer(String serverName, Boolean useSecureConnection, Int32 portNumber)~~ at Microsoft.SystemsManagementServer.WSUS.WSUSServer.ConnectToWSUSServer(String ServerName, Boolean UseSSL, Int32 PortNumber)
Remote configuration failed on WSUS Server.
In the WSUSctrl.log were these entries which seemed to indicate that SCCM was configured correctly to talk to WSUS.
Found WSUS Admin dll of assembly version Microsoft.UpdateServices.Administration, Version… Found WSUS Admin dll of assembly version Microsoft.UpdateServices.Administration, Version… Found WSUS Admin dll of assembly version Microsoft.UpdateServices.Administration, Version… The installed WSUS build has the valid and supported WSUS Administration DLL assembly version … Successfully connected to local WSUS server Local WSUS Server Proxy settings are correctly configured as Proxy Name and Proxy Port 80 Successfully connected to local WSUS server There are no unhealthy WSUS Server components on WSUS Server SCCMserver Successfully checked database connection on WSUS server SCCMserver
Then in the wsyncmgr.log there is this failure when SCCM tries to run a synchronization:
Performing sync on retry schedule STATMSG: ID=6701 SEV=I LEV=M SOURCE=”SMS Server” COMP=”SMS_WSUS_SYNC_MANAGER” SYS=SCCMserver SITE=SITEcode …. Sync failed: WSUS server not configured. Source: CWSyncMgr::DoSync STATMSG: ID=6703 SEV=E LEV=M SOURCE=”SMS Server” COMP=”SMS_WSUS_SYNC_MANAGER” SYS=SCCMserver SITE=SITEcode PID=1212 TID=2992 GMTDATE=Tue Jun 03 17:52:14.874 2008 ISTR0=”CWSyncMgr::DoSync” ISTR1=”WSUS server not configured” … Sync failed. Will retry in 60 minutes Sync time: 0d00h05m00s Waiting 60 minutes for requests…
On Friday I was troubleshooting an issue on our SCCM server related to updating a package on a Distribution Point. It kept failing with errors like the following:
2302 “SMS Distribution Manager failed to process package “”Random App”" (package ID = ABC00003).” Possible cause: … 2342 “SMS Distribution Manager is starting to distribute package “”Random App”" to distribution point “”[""Display=\\SCCMServer\""]MSWNET:[""SMS_SITE=ABC""]\\SCCMServer\”".” 2300 “SMS Distribution Manager is beginning to process package “”Random App”" (package ID = ABC00003).” 30125 “User “”domain\user”" added new distribution points to a package named “”Random”" (ABC00003).”
After digging around for a while I came to discover that the IIS services had failed to start. When I tried to start the IIS service, I got the lovely message that applicationhost.config is not well-formed XML. Now…that file on my server is over 1000 lines long. I don’t know if it is well formed or not. So…I hit Google and came across a page talking about an issue where that file will get corrupted if you are using either OneCare or Forefront Client Security as your anti-virus software. I’m using FCS on my server.
The workaround listed on that page is:
Create the following key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MpFilter\Parameters” .
Add a DWORD value “ScanOnCleanup” and set it to 0.
Restart OneCare/Forefront service.
I did the above workaround, but what about the corrupted applicationhost.config? I took a gamble…perhaps the file is corrupted, but the text is intact. I simply opened the applicationhost.config file in EditPlus, saved it as a new file, and copied the new file to the proper location.
IIS Started. SCCM distributed the package. I went home happy.
As part of my installation of SCCM on Server 2008, I have needed to install WSUS on Server 2008 as well. Being the good little engineer that I am, I looked at the WSUS Release Notes to ensure that I knew about any issues with installing WSUS on Server 2008. When I did so, I discovered that there is a typo in the Release Notes. At the bottom of the Release Notes is a section labeled “WSUS 3.0 on Windows Server 2008″ that lists two issues…only one of which applied to my scenario.
Issue 1: The IIS 7.0 configuration file must be updated before running WSUS 3.0
Before running WSUS 3.0 on Windows Server 2008, the IIS configuration file must be updated. You will need to take the following steps:
1. Open the IIS configuration file: %WINDIR%\system32\inetsrv\applicationhost.config
2. In the <System.webServer><modules> tag, remove <add name=”CustomErrorMode”> if it exists.
3. In the <System.webServer><modules> tag, add <remove name=”CustomErrorMode”>.
So, I went into the applicationhost.config file and made that modification. The “add name” tag wasn’t there, so all I had to do was add the “remove name” tag. After adding it exactly as the release notes listed, I went into my WSUS setup. When it got to the section where it would ask about whether to use the Default IIS website or a custom website, I got an error message instead. It stated:
“The wizard could not detect any existing Web sites. To try again, verify that IIS is running, use IIS Manager to resolve the problem and then click Next.”
When I opened up IIS Manager I got this error message:
There was an error when trying to connect. Do you want to retype your credentials and try again?
When I looked at line 844 (which was easy to find in EditPlus), that was the “</modules>” line from above. That line is fine. However (knowing that sometimes the error is earlier in the file than where it shows up as an error to the application)…line 843 does have an error in it…an error straight out of the release notes. The “remove name” tag is missing a forward slash. It should be:
<remove name=”CustomErrorMode”/>
After making that change and starting IIS, WSUS was much happier when it installed.
Sometime around 8:30-8:45PM Eastern Time, the SP1 update for SCCM went live on the Microsoft Download Center. Here is the direct link to the download. (See Update #2 below) As I type this I am 49% done with the download. You know that that means? I get to install it tomorrow morning!
[Update: 9:10PM...my download finished!]
[Update #2: Note that the download link above is to the Upgrade version. You need to have a fully functioning SCCM RTM site to use it. There is also a slipstreamed eval version. Had I thought for half a second last night I would have known that they wouldn't put a fully licensed (with embedded product key) download on the public site. And since I didn't download the correct version last night, I'm stuck downloading in the middle of the day along with everyone else...I need the slipstreamed version to install on Server 2008. Last night it downloaded in 20 minutes or so. I've been downloading for an hour so far...with an estimated two hours more to go.]
Just got alerted that SP1 for SCCM has been signed off on and should hit the download center soon!
[Note: this post was much shorter than planned. I was sitting in the exam room at my wife's OB when I got the email alert. I was blogging it from my phone when the doctor walked in. Had to finish up quickly!]
I learned this a while back, but never blogged it. I was building a Build and Capture Task Sequence in SCCM. At the end when it started the capture, I wanted it to upload the WIM to my DSL which happens to be on a Distributed File System share. It failed. I can’t remember the error that it gave, but eventually I determined that it simply couldn’t copy to the DFS share. I pointed it to a regular share (that went to the same folder as the DFS share), and it worked fine.
I use the DFS share as the source to all of my programs without any problems. But for the Build and Capture, I have to reference a plain server share instead.
At the risk of sounding a bit schizophrenic, I’ve changed my mind again…I’m going to reinstall SCCM on Server 2008. Several reasons behind the change of plans. This will also serve as a bit of a response to Brian Mason’s comment on my previous SCCM post.
First, I had forgotten how much tweaking I had to do in the OS on Server 2003 to get certain things to work. In particular, there were several things I had to tweak to get patching working. WSUS and SCCM weren’t playing well together. After installing Server 2003 on Friday and following my notes for what fixed the issue last time, and checking several things over the weekend…patching still isn’t working. Feeling the need to get patching working was my main reason for wanting to not wait on SP1. But then I thought some more and realized reasons 2 and 3 below. (And with the issues I was experiencing, I’m not convinced that I could get it completely working before SP1 hits…and I really don’t want to go through this process twice.)
Second, I don’t have that many users on SCCM right now (approximately 100). This need for a rebuild happened before I moved my entire environment to it, so it makes it possible to put a pause on SCCM until SP1 is released this month.
Third, a combination of not that many new patches in May and not having that many machines on SCCM. I can set a few GPO settings to point those machines to Microsoft Update for this month’s patches…then get them back on SCCM after the reinstall.
Fourth, I’m hoping that SP1 will be released sooner (this week) rather than later (end of the month…next week). We shall see.
So, tomorrow I go back in and drive our VM guy crazy again. I need those RDM disks added to a Server 2008 VM instead of the Server 2003 VM that they are currently on…yes…the same Server 2008 machine that he just moved them off of. Sorry Cesar!
It took a really long time to get the VM set up correctly for my reinstall of the SCCM server. Finally got it around 2pm yesterday and proceeded to install Server 2008. After starting that process and being really impressed with my first look at Server 2008, I started looking into the process for installing SCCM. Then I came across several posts about SCCM not being supported on Server 2008 til SP1. The firmest one I found was here. In it Wally Mead states (emphasis added), “SP1 is the only release that does support Windows Server 2008 for any site system role other than the System Health Validator. So yes, you do have to wait, there is no workaround.” That would be a pretty firm statement. And with some of the issues I was running into…it wasn’t worth the chaos…especially not when I really need to get this box back up and running ASAP.
Now the bummer is that SP1 will be released by the end of the month…heck, it might be released next week. But I can’t wait that long to get this machine running again. I will probably be doing another migration to Server 2008 within a month of the release of SP1, but I just can’t do it now.
What a difference a couple of days makes. Last week my plan was to migrate the other 90% of our headquarters off of SMS 2003 and over to SCCM. Before doing that I wanted to fix a disk issue. Basically when the guy who set up the VM for my SCCM server did so, he set it up wrong…he set it up to use VHDs instead of Raw Device Mapping for the drives. Basically I got slower disks…and SMS/SCCM really needs fast disks. Not a problem in my initial setup…only had 10% of our environment on it. But before migrating the rest, I wanted fast disk.
So…our virtual server guy started the process of moving the four drives from VMDK to RDM. He was running into a lot of problems and finally (as I understand it) figured out that he can’t do what he was trying to do. In the process though, the C: drive (OS installation drive) got corrupted. Because of misunderstandings on both sides, I didn’t have a backup of the OS drive or the VM file. I do have a successful SCCM/SQL backup though.
Time to test that disaster recovery plan!
I’m also taking this opportunity (I’ll call it that) to do another switch. I had originally installed on Server 2003 x64. My new install will be Server 2008 32bit. I’m waiting on the VM guy to finish the OS install so that I can do the other pieces. The nice thing is that I did keep very good notes about all of the configuration that I had done on the original server, so I should be able to rebuild it fairly quickly.
There are reasons for the double switch (2003/2008 and 32/64). The 32/64 issue is related to my friend Tim’s post. The 2008 switch is to be ready to utilize OSD Multicast when SCCM R2 ships.
I’ve been troubleshooting an issue with Computer Associations in SCCM. The issue that I have seen multiple times is that the state capture appears to run successfully…there is a USMT file of the size that I expect in the State Migration Point folder. However when I run the state restore, it fails very early in the process. As it is requesting the state store, it fails out with a message that states:
Task Sequence: Restore User State has failed with the error code (0×00004005). Please contact your system administrator or helpdesk operator.
Well…I am my system administrator, and this error is bothering me!
Once I dig into the smsts.log file, I see these errors:
SMP request to “SCCM.fqdn” failed with error: E_SMPERROR_RESTORING_CLIENTID_NOT_FOUND (207)
Request to SMP ‘SCCM.fqdn’ failed with error (Code 0×80004005). Trying next SMP.
I’m taking a wild guess here…and I could be wrong…I’ll be posting on Technet and updating this post when I find out more. My guess is that this is tied to when I created the computer association linking the old computer with the new one. This is a side-by-side migration…replacing a user’s old computer with a new one. I ran the capture TS on her old computer. The errors above are on her new one. I am wondering if it matters when the computer association linking the old to the new is created. I forgot to create that association until after the capture TS had already run on her old computer. That capture TS created an “in-place” association for her old computer. After the capture was complete, I created a side-by-sideassociation. Both of these are tied to the same state location.
I may be completely wrong here (certainly wouldn’t be the first time), but is there anything that the State Capture TS puts into what it uploads that restricts what it will restore to? I.E. does the side-by-side association need to exist before running the Capture TS for it to be able to restore to the new computer?
Another possibility that will be determined when I get the smsts.log off of the old computer (laptop…user has it off site today) is if it fits into the discussion of this thread…where the status message is not recieved by the site server that the capture process is complete…so the state is not recorded as completed.
[Update: I got the log file from the user this morning. It does fit that thread above. I'm posting to TechNet to see if I can get an answer as to how to automatically resolve this issue. I've seen it happen way too frequently.]
The Bible Study at MMS was really good. We spent four mornings going through the book of Philemon. We met in Rod and Megan Trent’s suite at the Venetian. It was a really good time to start the day off with a song or two, a bit of prayer, and some time in the Bible before heading into the crazy MMS schedule. Over the course of the four days we had about fifteen different people in the study. There was a core group that was there every morning. Some people flew out early or were unable to come on certain mornings. It was really neat to be able to connect with people on a personal/spiritual level in the midst of a massive IT conference in the midst of “Sin City”. The picture below was taken Thursday morning. As a group we agreed that this was the “First Annual MMS Bible Study”. Very cool.
It was truly amazing the number of people who came up to me last week as a result of the video at MMS. All week I had people recognizing me from the video. Some just stopped me to say they liked the video. Quite a few wanted to pick my brain about what I had done with OSD in SCCM. Honestly…it was a lot of fun to be able to meet so many people and to hopefully help out a few.
One of the funniest comments came from Brian Mason. I had met him last year at MMS when I tracked him down to ask for input on deploying patches with ITMU in SMS 2003. Every time he saw me in the hallways after the video he called me a “rock star”. He had me laughing. I really enjoyed a couple of the conversations I had with Brian between/during sessions…the one that we had while skipping the last keynote was exceptionally helpful. Thanks Brian!
I also had a series of funny run-ins on the way home from MMS. As I was in the Southwest Airlines check-in line in Las Vegas, the guy in line behind me was flying to Nashville. He recognized me from the video and made a comment about it. Later, after landing in Orlando, the guy sitting directly in front of me on the plane also mentioned seeing the video. Then…while standing in the baggage claim area waiting on my luggage to come through, I had another guy from the conference come up to me saying something about the video. It was really wild.
To everyone who came up to me throughout the week…thanks for the encouragement. I hope I helped a couple of you. Keep checking back to this site. I will continue to post about the SCCM things that I am working through. Hopefully it will help some of you.
It was cool seeing the video on the screen at the State of the Nation session at MMS. Here are a couple of pics I took during the session. That is Bill Anderson from Microsoft on the stage. I like the title of the slide he used to intro the video…”Kindergarten MCSE”. The second shot shows a portion of the video where Marybeth is clicking to deploy a computer using SCCM’s OSD.
This is the video that I posted about yesterday that was shown during the “State of the Nation” session at MMS this morning. It features me talking about how we are using SCCM OSD, and concludes with my five year old daughter using SCCM to deploy seven computers. Enjoy!
Over the last several months I’ve developed a relationship with Wally Mead. Wally is a Senior Program Manager at Microsoft where he works with the System Center Configuration Manager product team. He handled a bug report that I submitted about SCCM (
