Speed decreases gradually when deploying

Comments

14 comments

  • Adam Ruth

    It sounds like a problem with the bandwidth throttling.  Does the problem happen when it is set to 100% in the preferences?  This will disable the throttling completely.

    0
    Comment actions Permalink
  • Jprater

    Also, when it does this, I am unable to abort the deployment. I cannot end the PDQDeployPro.exe task or stop the service either. I usually have to restart the computer.

    0
    Comment actions Permalink
  • Jprater

    I changed the throttling to 100%. Still does it. It starts out at 10 MB/sec then just slows down to 30 KB/s. The percentage complete also stops. Usually stuck around 30% or so. It's almost like the entire deployment stalls.

    0
    Comment actions Permalink
  • Adam Ruth

    With throttling disabled PDQ Deploy Pro is just sending the file through the Windows API for file copy.  Do you have any anti-virus software that could be interfering with the file copy?  We've seen a number of problems with anti-virus preventing the file copy, but never quite like this, it's usually just an error message.

    0
    Comment actions Permalink
  • Jprater

    We are running Symantec EndPoint Protection. I have disabled the antivirus software and the problem persists. Here is a video of what it is doing:

    http://thoughtreactor.com/pdq_deploy_problem.avi

    0
    Comment actions Permalink
  • Jprater

    And I was wrong in my original statement--I am pushing our an .msp file, not an .msi.

    0
    Comment actions Permalink
  • Adam Ruth

    Something is causing the file copy to stop completely at around 28% (the bandwidth count is just a running average, so it would eventually drop to 0 KB/sec). 

    It could be either that all file access to the computer stops during the copy, or it's just the individual copy itself.  Knowing which would help to narrow down where the problem is.  Can you try running the deployment and then when it stops copying trying to copy the same file to the target computer using Windows Explorer?  You can open \\[target]\ADMIN$\PDQDeployPro\exec and copy the same MSP file (with a different name). If the copy succeeds through Windows Explorer then it's a problem local to a single copy process. 

    0
    Comment actions Permalink
  • Chris Donnell

    So, I started trying out PDQ Deploy yesterday and I seem to be having the same problem - although I have a wider sample of symptioms (I really want it to work, so in my 'spare time' I've been trying different things)...

    What I've noticed, is that the copy doesn't get slower - the copy stops altogether, the reported bandwidth just keeps going down because it's likely working as a simple average over the whole time and not reporting actual recent usage.  So if it went to 5 seconds before crapping out, by the time you've reached 10 seconds it'd going to drop to 1/2 the bandwidth, etc.

    I can sometimes get a deployment to work, but more often than not I can't.  And the process freezes up hard, I need to reboot in order to try again.  I seem to eventually (depending on how patient I am) deploy to a particular computer.

    I can copy files to the ADMIN share manually (although I don't see a PDQDeployPro directory, there is a different PDQDeploy directory though).

    I've even tried installing PDQDeploy on a second computer and it seems to work exactly the same (the computers here were generally installed from the same image, but there isn't anything strange on it...).

    Disabling Symantec Endpoint Protection (on the computer running PDQ Deploy) doesn't seem to make a difference.

    I haven't had a chance to do a test on a fresh install of Windows 7 - but there's definitely something up in that it's acting strange for me on two different computer and for Jprater as well.  Any ideas would be appreciated - as the tool would be totally awesome if it didn't take 2-3 reboots per successful deployment!

    0
    Comment actions Permalink
  • SelfMan

    For testing purposes, can you connect the two computers using a different switch? Have you tried to update the network adapter driver and have you tried to change the network adapter settings? (i.e. disable jumbo frame support, manipulate checksum calculation etc. - on the server AND on the client too)

    0
    Comment actions Permalink
  • Adam Ruth

    Chris,

    Thank you for posting your experience.  The reported bandwidth is using a simple (i.e. braindead) average, so you are right about it dropping to zero. 

    We've been working on narrowing down where the problem is, and we're pretty close, though we still haven't been able to duplicate it in our lab environment.  Can you provide a bit of information?

    Were you using the Pull File Copy option?

    How much data was being copied when you saw it fail?  How many files and how large were they?

    Do you notice it stops at the same % or does it vary?

    One thing we're going to do in the next update is to have a second file copy method that can be enabled with a registry setting.  This second method uses a different Windows API call than is currently used.  It's slower, but it may be more reliable.

    Thanks again for any information you can provide.

    0
    Comment actions Permalink
  • Adam Ruth

    The latest beta of PDQ Deploy is out now with the changes I described.  It has another method of copying which avoids the Windows copy APIs.  It's slower than the current method because it's two separate operations, one to read the file and one to write it, so about half as fast. But it should tell us if the problem is in the copy API, which is a pimary suspect due to the symptoms you describe. 

    In order to enable it you need to add a registry value to the console computer.

    [HKEY_LOCAL_MACHINE\SOFTWARE\Admin Arsenal\Runner Engine]
    "Secondary Copy"=dword:00000001

    I've attached a .reg file for your convienence.

    Also, in case we need more information, when using pull copy a log file will be created on the target. Hopefully it will tell us better where the copy is halting, whether it's between files or in the middle of one. It will be in the PDQDeployRunner directory on the target computer:

    %windir%\PDQDeployRunner\1\filecopy.log

    Thanks for any help you can provide in quashing this annoying bug.

    0
    Comment actions Permalink
  • Chris Donnell

    Adam,

    Sorry if I've been ignoring you - just been working on other things and haven't had a ton of time to mess with this.

    I did try downloading the beta and decided to try with no changes to what I was doing other than installing the beta.  It worked the same, hard crash of the PDQ Deploy service such that only a reboot would fix.

    Then I tried using the registry key and had no change - although I don't think it would have had a change, since I was using free mode, so pull wasn't an option.  (I had figured I'd try and evalutate the free functionality before starting the trial.)

    Since all this was focused on the pull mode, I figured I'd just bite the bullet and get the trial and that seems to work fine.  I also backed out the registtry entry and it still seems to work fine.  I guess the problem is all in the push mode available in the free mode.

    I was likely to purchase anyway (assuming I could get it functional), so this isn't a huge deal - although still no idea why the free mode isn't working and is crashing so hard.

    Thanks!

     

    0
    Comment actions Permalink
  • Adam Ruth

    Chris,

    Thank you for taking the time to try it out.  I forgot to mention that after making the registry change you need to restart the PDQDeploy service in order for the change take effect. My apologies.

    Also, can you look in your application event log?  If PDQ Deploy crashed then there may be an event in there from Windows with some details about the crash, mostly which module caused it. It's pretty rare for a .NET app to crash that hard, it usually means that it was in the middle of a native API call when it happened. 

    0
    Comment actions Permalink
  • Adam Ruth

    We've finally been able to duplicate this issue here in our lab, or at least duplicate the symptom.  In our case the cause was an out-of-date NIC driver.  We were able to consistently get the copy to grind to a halt when the "Large Send Offload (IPv6)" option was turned on in the driver advanced settings.  When disabling this setting or disabling IPv6 the copy would return to normal.  Updating the driver to the current version cleared it up completely.

    This may not be the same cause for you, but it's something to look into.

    0
    Comment actions Permalink

Please sign in to leave a comment.