Interesting Memory Leak in Configuration Manager Agent (ccmexec.exe)

Last week I encountered an interesting issue with the System Center Configuration Manager 2012 R2 client. The issue is seen as a memory leak in the ccmexec.exe process, driving up the memory consumption in excess of 10GB of memory.

We saw the issue only on Windows 2008 R2 machines where an specific update (KB2775511 – Enterprise Update – See https://support.microsoft.com/da-dk/kb/2775511)

When examining the CCM\Logs nothing was shown but examining the process using Process Explorer and Process Monitor showed a large number of threads being started and ending. These threads have a stack trace looking like

ntoskrnl.exe!memset+0x61a

ntoskrnl.exe!KeWaitForMultipleObjects+0xd52

ntoskrnl.exe!KeWaitForMutexObject+0x19f

ntoskrnl.exe!PoStartNextPowerIrp+0xba4

ntoskrnl.exe!PoStartNextPowerIrp+0x1821

ntoskrnl.exe!KeWaitForMultipleObjects+0xf5d

ntoskrnl.exe!KeRemoveQueueEx+0x323

ntoskrnl.exe!ExQueryAttributeInformation+0x1803

ntoskrnl.exe!KeDetachProcess+0x4d6

ntoskrnl.exe!KeSynchronizeExecution+0x3a23

ntdll.dll!ZwWaitForWorkViaWorkerFactory+0xa

ntdll.dll!RtlValidateHeap+0x39b

kernel32.dll!BaseThreadInitThunk+0xd

ntdll.dll!RtlUserThreadStart+0x21

I have been working with Microsoft Support (I still think of them as PSS) on the issue and they have confirmed the problem and tracked it to CBS (Component Base Servicing). For an introduction to CBS see http://blogs.technet.com/b/askperf/archive/2008/04/23/understanding-component-based-servicing.aspx

According to Microsoft the issue can happen on a number of updates but I have only seen the issue when deploying kb2775511. And that update should not, in my opinion, be installed using Software Updates since a number of updates must be installed afterwards to fix problems introduced with the update.

Posted in Configuration Manager | Tagged , , | Leave a comment

Fix for IE 11 hangs on various operations

I have been troubleshooting a number of issues with IE11 after applying the patches for August 2015. The problem manifests itself as application hangs in IE11 but examining the dumps reveals that it is actually Forefront Endpoint Protection (FEP), System Center Endpoint Protection or Microsoft Security Essentials causing the hang.

The problem was introduced in MS15-084 and can be resolved by applying https://support.microsoft.com/en-us/kb/3092627

Posted in Configuration Manager, FEP | Tagged , , , , | Leave a comment

BitDefender AVC and Configuration Manager 2012 R2

I have recently worked on a issue where the customer was experiencing extremely bad performance on a Configuration Manager 2012 R2 site. The backlog for processing status messages was over two hours.

After going over the eventlog, disk performance and other obvious areas the attention was swiched to drivers running in ring 0.

The customer is using (and quite happy with) BitDefender GravityZone – see http://www.bitdefender.com/business/security.html

One of the options that can be enabled in BitDefender is AVC (Active Virus Control). Was we found was that even if the administrator has configured AV exclusions for various processes and files AVC will still examine those.

We disabled AVC on the Configuration Manager servers and found that after a few moments performance started to improve. It still took a few days (!!) before all backlogs were cleared.

The important lesson is that you should consider your AV solutions and optimize it for Configuration Manager.

Posted in Configuration Manager | Tagged , , | 1 Comment

Follow-up on problem 0x8007000E

As I wrote about on March 19 (https://larsnorman.wordpress.com/2015/03/19/windows-update-agent-returning-0x8007000e/) I have been having issues with incorrrect scans for security updates which can be traced down to an out-of-memory condition in the Windows Update Agent.

A large number of people have been in touch telling me about how they have the samme issue and things they have tried. And at the same time I have been busy trying to find a workable solution.

To see how many machines are impacted at a given time (and to find machines that can examined in more detail) you can use the following query

SELECT
  rs.Name0 as ‘Name’, SUM(ucs.status) as ‘Num updates’
FROM
  v_R_System rs
left join
  v_UpdateComplianceStatus UCS
    on ucs.ResourceID = rs.ResourceID
left join 
  BGB_ResStatus BGB
    on rs.ResourceID = BGB.ResourceID
where
  rs.Client0 = 1
group by
  rs.Name0
having COUNT(UCS.Status) = 0
order by rs.Name0

(the join with BGB is in so I can filter on machines that are currently online)

I have now seen massive improvement following two steps:

1. Decline updates marked as superseded in the WSUS database. See the process described here: http://www.tecknowledgebase.com/43/how-to-identify-and-decline-superseded-updates-in-wsus/ (thanks Mark!). Please note that running the WSUS Cleanup Wizard is not enough.

2. Change the WUAUServ service to run in its own memoryspace by running sc config wuauserv type= own After configuring the service you should restart the service. Note the discussion on failed scans below.

A few notes of interest. Declining the updates in WSUS must be performed at all your WSUS servers, including WSUS on CAS, secondary sites and any WSUS servers for redundancy or locatated on DMZs to provide access for IBCM machines. I ran the WSUS Cleanup Wizard post decline. Also while you are at it you could consider cleaning out updates for any OSes you do not longer support.

A noticed a sharp increase in load on WSUS servers post cleanup. The size of the daily IIS log file for WSUS was larger.

Also the WUAUServ (Windows Update Agent) service would still consume 1.1 – 1.3 GB of RAM on the 32bit clients so in reality the problem is only postponed until Microsoft releases a real update (which I was told by a number of contacts is forthcoming).

Approx. 5% of the machines I tested on would fail the scan with the same error code.

In testing I noticed that a number of machines would report back a failed scan when I configured the Windows Update Agent service and restarted it. So I have decided to only configure the service and wait for a restart. If you are not concerned about the scan statistics you should restart the service right away.

Posted in Configuration Manager | Tagged , , | Leave a comment

A fantastic document on Software Update Management

For the last few weeks I have spend quite some time troubleshooting problems with Software Update Management on Configuration Manager 2012R2. As part of my research I found a very nice document by Vinay Pamnani (from Microsoft).

Vinay does a great job of showing all the steps going on in log files and provides quite a few good tips for troubleshooting SUM and SCCM in general.

Highly recommended.

See the document here: http://www.microsoft.com/en-us/download/details.aspx?id=44578

Posted in Configuration Manager | Tagged , , , , | Leave a comment

Interesting WMI issue on DPs

I recently worked on an issue with a few remote DPs where content couldn’t be added to the DP. What we saw was a package being copied in what looked like a succesfull manner followed by

CSendFileAction::SendFiles failed; 0x80010108 SMS_PACKAGE_TRANSFER_MANAGER 20/03/2015 08:23:34 4940 (0x134C)

CSendFileAction::SendFiles failed; 0x80010108 SMS_PACKAGE_TRANSFER_MANAGER 20/03/2015 08:23:34 4940 (0x134C)

Sending failed. Failure count = 69, Restart time = 20/03/2015 09:23:34 Romance Standard Time SMS_PACKAGE_TRANSFER_MANAGER 20/03/2015 08:23:34 4940 (0x134C)

So what is going on here? The error code (0x8010108) is RPC_E_DISCONNECTED

To figure out the actual Distribution Point either examining earlier log entries from the same thread or take the failure count and use Distribution Point Job Queue Manager to find the DP based on the number of retried.

In this case the problem was found on the DP where WMI was unresponsive. A restart didn’t fix the issue. We had to resort to a winmgmt.exe /resetrepository.

Posted in Uncategorized | Leave a comment

Windows Update Agent returning 0x8007000E

I have lately been working on an issue where Configuration Manager 2012R2 is reporting a large number of machines with a status of unknown for various updates. I have narrowed down the issue to an out-of-memory condition in Windows Update agent (as seen in WindowsUpdate.log). Here you will see something like

ISusInternal::GetUpdateMetadata2 failed, hr=8007000E

The same error can be found in various other Configuration Manager 2012 logs.

I have an open case with Microsoft PSS on this issue and is hoping for a fix soon. The problem is only seen on x86 based systems (mine are all Windows 7, but I assume the problem could hit Windows 8/8.1 as well). Note that the issue is all about the size of the catalog – not the number of updates per deployment or number of deployments.

A few things that has helped a bit:

1. Applying the latest version of the Windows Update Agent (from https://support.microsoft.com/en-us/kb/949104). The new agent initially looked like it fixed the problem but now we are experiencing the problem again.

2. Running the WUAUServ service in it’s own svchost.exe process

net stop wuauserv

sc config wuauserv type= own

net start wuauser

But the real solution is to contact Microsoft and make sure they are aware of this issue

Posted in Configuration Manager | Tagged , , | 6 Comments