DescriptionTim van Wegberg
2002-09-26 06:57:10 EDT
When a linux server runs for a prolonged period, it's memory usage seems to gradually increase, causing increased swapfile usage (especially during level switches):
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
24738 ut2003 19 0 120M 95M 3688 S 33.6 18.9 505:23 ucc-bin
24726 ut2003 15 0 209M 143M 5912 R 29.1 28.3 458:24 ucc-bin
24748 ut2003 17 0 104M 78M 3644 R 21.2 15.6 1171m ucc-bin
24715 ut2003 14 0 244M 142M 3584 S 11.4 28.1 546:56 ucc-bin
This is for four 14 player servers, normally it looks like this:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
15908 ut2003 20 0 70048 68M 5904 R 52.5 13.5 9:39 ucc-bin
15851 ut2003 17 0 73556 71M 5900 S 50.7 14.2 30:05 ucc-bin
15856 ut2003 16 0 74236 72M 6084 R 48.4 14.3 32:00 ucc-bin
This bug didn't show in 1077, but back then the servers didn't get as much cpu time because they were frequently rebooted to apply/test new settings. This particular box runs DM, TDM and bombing run servers, but the bug also shows on another (identical) box running CTF and various 2vs2 servers. Web admin is enabled on all servers.
No known fixes as of yet, the relaunch command does not release the memory, have not tried obj garbage yet.
More information like memory, cpu and bandwidth usage on a timeline might be available later.
Specs:
UT2003 Patch 1080b with updated Packages.md5 file.
Server: Supermicro 370DLR twin 1.0Ghz P3's, 512MB RAM
OS: Debian 3.0 Woody "Stable"
Kernel: 2.4.19 with the EVMS patches
Ok I've been running the server now for 3 days no crashes but it is leaking
memory below is the start and end of my log:
CTF-Citadel with 0 Players = 131260 KB 0.0% CPU
DM-Antalus with 0 Players = 151484 KB 0.0% CPU
..
CTF-Citadel with 0 Players = 186036 KB 0.0% CPU
DM-Antalus with 0 Players = 277988 KB 0.0% CPU
Created attachment 28[details]
Some stats
Here's some statistics showing the swapfile grow over time on two of our
servers. it doesnt seem to grow at regular intervals or with regular steps.
I've included CPU and bandwidth usage to give an impression of the busyness of
the server. Neither one seems to have any influence.
More info on this one:
DOM-Core with 0 Players = 57492 KB 0.0% CPU
DOM-Core with 0 Players = 61660 KB 0.0% CPU
Between these two snapshots ( 9 hours ) no players entered the server yet it
leaked
4.1MB.
While playing any 1 map the memory usage doesnt appear to bad:
DOM-Outrigger with 13 Players = 108400 KB 24.8% CPU
DOM-Outrigger with 13 Players = 108404 KB 32.9% CPU
DOM-Outrigger with 13 Players = 108404 KB 32.1% CPU
DOM-Outrigger with 14 Players = 108432 KB 36.8% CPU
DOM-Outrigger with 14 Players = 108432 KB 36.8% CPU
DOM-Outrigger with 15 Players = 108452 KB 37.5% CPU
DOM-Ruination with 15 Players = 107848 KB 24.4% CPU
DOM-Ruination with 14 Players = 107848 KB 31.8% CPU
DOM-Ruination with 14 Players = 107848 KB 31.7% CPU
DOM-Ruination with 14 Players = 107848 KB 33.3% CPU
DOM-Ruination with 15 Players = 107848 KB 35.8% CPU
DOM-Ruination with 15 Players = 107848 KB 35.4% CPU
DOM-Ruination with 16 Players = 107848 KB 41.0% CPU
DOM-Ruination with 16 Players = 107848 KB 41.7% CPU
DOM-Ruination with 18 Players = 107848 KB 43.5% CPU
DOM-Ruination with 17 Players = 107848 KB 45.3% CPU
DOM-Ruination with 20 Players = 107848 KB 43.2% CPU
DOM-Ruination with 16 Players = 107848 KB 46.8% CPU
DOM-Ruination with 17 Players = 107848 KB 45.8% CPU
DOM-Ruination with 20 Players = 107848 KB 45.8% CPU
DOM-Ruination with 15 Players = 107848 KB 45.6% CPU
DOM-Ruination with 16 Players = 107848 KB 38.9% CPU
Here even though players are joining an leaving the usuage accualy dropped.
The key point seems to be map changes:
DOM-Ruination with 16 Players = 107848 KB 38.9% CPU
DOM-ScorchedEarth with 14 Players = 117036 KB 19.4% CPU
DOM-ScorchedEarth with 9 Players = 117036 KB 17.3% CPU
DOM-SepukkuGorge with 9 Players = 104792 KB 16.4% CPU
DOM-SepukkuGorge with 12 Players = 104808 KB 27.7% CPU
DOM-Suntemple with 6 Players = 104800 KB 16.4% CPU
DOM-Suntemple with 12 Players = 104832 KB 24.3% CPU
DOM-Core with 6 Players = 105936 KB 20.9% CPU
DOM-Core with 7 Players = 105968 KB 12.0% CPU
DOM-Outrigger with 9 Players = 110128 KB 15.4% CPU
DOM-Outrigger with 10 Players = 110312 KB 24.3% CPU
DOM-Ruination with 4 Players = 113676 KB 18.4% CPU
DOM-Ruination with 18 Players = 113708 KB 48.1% CPU
DOM-ScorchedEarth with 13 Players = 120832 KB 20.4% CPU
DOM-ScorchedEarth with 16 Players = 120840 KB 36.7% CPU
DOM-SepukkuGorge with 7 Players = 108808 KB 20.1% CPU
DOM-SepukkuGorge with 12 Players = 108808 KB 18.8% CPU
DOM-Suntemple with 14 Players = 109408 KB 15.5% CPU
DOM-Suntemple with 10 Players = 109408 KB 15.4% CPU
DOM-Core with 8 Players = 109968 KB 15.5% CPU
DOM-Core with 1 Players = 109968 KB 5.2% CPU
DOM-Outrigger with 2 Players = 113972 KB 5.4% CPU
DOM-Outrigger with 2 Players = 113996 KB 5.3% CPU
DOM-Ruination with 1 Players = 116576 KB 11.0% CPU
DOM-Ruination with 0 Players = 116576 KB 0.6% CPU
DOM-ScorchedEarth with 0 Players = 124056 KB 6.0% CPU
My other results that where sent in email for reference:
Ok looks like its slightly better but still not a dent in the main leak:
CTF-Citadel with 19 Players = 91584 KB 44.1% CPU
CTF-Citadel with 19 Players = 111520 KB 49.3% CPU
CTF-Citadel with 19 Players = 122704 KB 46.5% CPU
BR-Anubis with 13 Players = 106188 KB 27.5% CPU
BR-Anubis with 14 Players = 123324 KB 30.5% CPU
BR-Anubis with 14 Players = 124328 KB 33.7% CPU
DM-Phobos2 with 11 Players = 109136 KB 15.3% CPU
DM-Phobos2 with 11 Players = 127164 KB 17.6% CPU
Each entry above is after 1 complete cycle of the map list ( all maps )
since I started the servers this morning. I've picked entries to that
have as consistent amount of players as possible to give a fair comparison.
Created attachment 28 [details] Some stats Here's some statistics showing the swapfile grow over time on two of our servers. it doesnt seem to grow at regular intervals or with regular steps. I've included CPU and bandwidth usage to give an impression of the busyness of the server. Neither one seems to have any influence.