I can’t say this feels right to me:
Notice the 603 terabyte value in the “I/O Other Bytes” column for csrss.exe. Oh yeah, and the box has only been up for 24 hours. The number seems to climb by 50 or 60 GB a second when I’m connected via RDC. If I logoff then back on, the count starts again.
I’ve tried Googling it, but found no solutions.
Anybody got any ideas? It doesn’t actually seem to be causing a problem, it’s just out of the ordinary.
The box is a 1.8GHz Core2 Duo running Windows Server 2003 R2 Standard x64 Edition.
Update – I (think) I found the solution!
Following some suggestions in this post’s comments and on Google that this might be a virus, Edward suggested I fire up a copy of SysInternals Process Explorer on the server. I’ve used the SysInternals tools in the past, but not being a server guy by trade meant that I’d forgotten about them.
After finding the process, I was quickly comforted that this wasn’t a virus. The signature was verified, it was executing from the correct path (system32) and it was running under NT AUTHORITY\SYSTEM where I expected it to.
If we recall the original symptoms, the count was reset if I logged off RDC and back on again. This didn’t really prove much other than the fact that the process restarted at login. Swapping to the “Performance Graph” tab is Process Explorer did however show some interesting results.
Whenever I minimized my RDC window, it stopped requesting data and the I/O bytes graph was stagnant. As soon as I brought the RDC window back up and started interacting with it, the values skyrocketed. My screenshot doesn’t show it, but that graph was peaking to 80GB!
The end result would seem to be: “that’s just how much stuff is going on to support RDP”.
To me that’s an incredible amount on information being shuffled around and process, but I guess this is why the actual data on the wire is so efficient. Unless anything else comes up, I’m happy to file this in the “amazing but normal” pile.
Tatham Oddie 
Here’s a link to a page that talks about this process, and how it *might* be a trojan. Might wanna check it out.
http://www.auditmypc.com/process/csrss.asp
@Chris – I did actually suspect this myself, however test would indicate that it is not. I think I have found the root problem. I will update the blog post with what it is, and how I worked it out, later today.
Im have some problem.. =((
Are You fix this problem ?!
@DAKnn – I’ve updated the post with the answer.
I have same problem, IO Byte Delta/s from CSRSS.exe is huge. However, when I run some program on top instead of Process Explorer, the IO Byte from CSRSS drop to zero but from explorer.exe raise to 8GB/s ??
It seems IO Byte other is due to Remote Desktop.
Finally, I remote my server using DRAC, all IO bytes gone (below 1MB/s)
We first saw this about two months ago, coincidental with troubleshooting an Oracle / application performance problem. RealVNC also caused some I/O spikes that we didn’t like to see. NetMeeting Remote Desktop Sharing does not, but it regularly has to be manually restarted at the console, so is less than ideal for our needs.
We had turned off Terminal Services, to prevent this from recurring. That did nothing for our performance problems, which seem to lie elsewhere. A sister organization is having this problem on another server that runs Citrix, but not Oracle. So I don’t think it’s an Oracle problem.
The first reference to this anywhere I can find is in in microsoft.public.windows.server.general on 31 Jul 2006. I bet this is widespread, but low-impact, only being found by admins looking at performance and finding this. In this case, “I/O Other” could refer to video I/O.
We are working with our well-known system vendor to find a patch or a fix. We had already tried http://support.microsoft.com/kb/934330 (which is for CPU, * not * I/O Other). No difference. Hoping for a hotfix from Microsoft RSN.
Running Windows XP x64 Edition and see something similar after installing Feburary Windows Updates. I am running nothing but the base system.
*
Outlook Profiler automatic profile creation
*
Just an update. If you will notice (using process explorer) under the Thread tab for the csrss.exe process that CSwitch Deltas occurs for winserv.dll which http://support.microsoft.com/kb/934330 mentions. Looks like this issue is not yet fixed…
Did anyone experience this when the rdp client was NOT maximized ? I can’t figure out why im still having this problem…
Thanks for the info… I have the same issue with csrss.exe getting HUGE “I/O other” throughput on an Oracle iAS windows server 2003 EE… After reading this blog, i feel more comfortable in suspecting the RDP sessions being the culprit.
Much appreciated!!!
The “File Control Byte/sec” counter of the System performance object displays values that are larger than expected on a computer that is running a 64-bit version of Windows Server 2003
http://support.microsoft.com/kb/935987
I have one production SBS system which has been exhibiting this same behavior since March. No resolution yet, but I’m working on it. We did disable RDP complete for around 4-6 weeks, but the intermittent terrible performance problems continued, so I don’t think our problem is caused by RDP. The consistent metric I’ve seen is that every time that we have these terribly slow episodes, the performance mmc snap-in shows the disk queue as being maxed out. Programs that are currently/already loaded into RAM are still pretty snappy, but anything that requires disk I/O is terribly slow. We have delays of 3-6 minutes just to log into the server, open new programs, and our database type programs are totally unusable during these times. My company is a Microsoft Partner, so we are working with them to try to get a resolution. If I find one, I’ll be happy to share. Hopefully it will help others here.
We have a similiar problem on our server running ACCPAC. although it is a high spec server, the high I/O being generated hampers the performance badly. Is there anyone that maybe has a resolution to this problem yet?
Having similar issues with SBS2003, is there any updates on this I/O issue yet?
Hi guys,
i’ve the same grave issue on my server.
Is thare anyone that know the fix of this problem?
same problem on one of our recently installed citrix server on W2K3 Server with SP2 š¦
there will hopefully be resolution for this problem soon.
http://support.microsoft.com/kb/935987
Having this problem too on Server 2003 (32 bit)
Not sure the provided hotfix works for me, it seems to be 64-bit specific. Guess I’ll have to wait, but at least it doesen’t do any harm (does not consume resources) and is just an annoyance.
Seeing a similar problem, Windows Server 2003 R2 Enterprise 32-bit. In this case it’s a subversion server (running VisualSVN). I can’t really say if this is the cause of our performance problems (with Subversion), but I am concerned. Has anyone seen this problem on another version of Windows, e.g. 2008 x64?
Migrate your apps to a non-windows operating system.. One that does not limit you to two logins and has an apparently crummy x terminal.
That’s not a solution, if you’d know anything about the corporate world you’d know that recompiling and porting software for alternative operating systems is a crapshoot that can cost tons of time and money.
Problem still exist as of date, I’m running a XP 2003 SP2 w/ all latest security updates, remote terminal cause high IO other, strange that it only causes this when there are something moving around on the screen, if you leave it completely still w/ nothing moving, not even your cursor, the IO drops to 1mb or less..
Good thing I don’t need to access it much after I get all the programs running, spent a good 10 hours trying to find a fix but no dice.
You have to click ‘verify’ to verify the signature (if the image file tab says ‘not verified’ š
I noticed this odd behavior, too, as I was tracking down an issue with a server. Sure enough, minimizing the RDP connection to the server caused the Other Bytes to go down to 0 until it was maximized again. Thanks for the webpage about this issue!
My hunch is that it has to do with the video display. Still, the GBs/sec seems too high even for that.
Any solutions for this in 2010 ?
My Windows 2003 terminal server (5 clients, but only 2-3 logged on simultaneously) is constantly trashing the disk and incredibly slow, even though only a few users are logged in via RDP and are working with programs that don’t cause a lot of I/O.
I have searched everywhere, and turned off all Indexing and all Antivirus Background Tasks that could be responsible, but csrss.exe is still constantly accessing the disk.
The only positive thing I can find about this desaster is that
a) both harddisks in my RAID1 setup are holding up fine
b) the RHEL 5 server sitting next to the Windows 2003 machine serves 10x more people and is still sooo much faster
Any solutions? I read that Microsoft will not release and further Service Packs for 2003 š¦
I’ve found the solution after thousands of fixes and updates!!
http://support.microsoft.com/kb/956438
This looks really promising. I’ve installed this Hotfix and at least so far I/O other counts seem fine and very low. I’ll report back tomorrow night (after a day of users working on the server) if the problems are truly solved…
Thank you rew for posting this!
Problem is definitely gone for me (2003R2 x64) after installing this Hotfix and this Hotfix alone. Thanks again – finally some performance on this Server.
It’s amazing, after applied kb956438, this problem is gone away! Thanks rew and any.
I also applied kb956438 and it fixed the problem. Thanks rew and any. Oh, 2003R2 32bit.
We did apply it over 40 servers and all ok !
Good job and thanks to all !
Wow! Thanks, this helped, kb956438 is a definite solution for this!! Thank you very much rew….
You’ve hit the ball out the park! Inredicble!
Just FYI:
worked for me too, THANKS š