Issue Starting Monitor
Posted: Wed Apr 05, 2017 1:32 pm
So last night we had a minor/major issue on one of our services. An error log filled up 300gb of storage causing the box to max out it's space, in the process taking down the UGCC Monitor.
When trying to start it this morning, nothing we tried seemed to want to work.
Here's /var/log/syslog
There's no ugccmon.log in the log folder. If I start via sudo/root it works fine (but then my servers run as root so that's not happening.)
Last time it successfully ran:
Now we're just stuck with the error I mentioned above in syslog, and no other rhyme or reason. Help would be appreciated! I even copied the UGCC from another server to this one and edited configs. It's working there but not here. Guessing something screwed up when we ran out of space?
Here's start.sh:
When trying to start it this morning, nothing we tried seemed to want to work.
Here's /var/log/syslog
Code: Select all
Apr 5 14:20:13 usdedi2 mono: ugccmonsvc.exe: Exception has been thrown by the target of an invocation.
Apr 5 14:20:13 usdedi2 mono: ugccmonsvc.exe: Document element did not appear. Line 1, position 1.
Apr 5 14:20:21 usdedi2 mono: ugccmonsvc.exe: Exception has been thrown by the target of an invocation.
Apr 5 14:20:21 usdedi2 mono: ugccmonsvc.exe: Document element did not appear. Line 1, position 1.
Apr 5 14:20:31 usdedi2 mono: ugccmonsvc.exe: Exception has been thrown by the target of an invocation.
Apr 5 14:20:31 usdedi2 mono: ugccmonsvc.exe: Document element did not appear. Line 1, position 1.
Last time it successfully ran:
Code: Select all
tangoww@usdedi2:~/ugcc/logs$ tail ugccmon.log
4/5/2017 11:06:48 AM UGCCMon Started (1.88) | Dbtype=mysql | Interval=60000ms | OS: Linux | Config file: ugccmon.cfg | User: tangoww
4/5/2017 11:06:49 AM Remote mode enabled. ID=5, Port=3432
4/5/2017 11:06:50 AM Starting CPU/Memory Monitor thread
4/5/2017 11:06:50 AM Starting process crash/query monitoring thread, firstbootpause: 10
4/5/2017 11:06:50 AM Starting timed restarts based on process runtime thread
4/5/2017 11:06:50 AM Starting timed restarts thread
4/5/2017 11:06:50 AM Starting auto updater thread
4/5/2017 11:06:50 AM Starting FTP Server on port 8081. Using PASV data ports: 2000-65535
4/5/2017 11:06:50 AM Home path: /home/tangoww | LD_LIBRARY_PATH path: .:./bin:
4/5/2017 11:06:50 AM -+-+-| Monitor Initialized |-+-+-
Now we're just stuck with the error I mentioned above in syslog, and no other rhyme or reason. Help would be appreciated! I even copied the UGCC from another server to this one and edited configs. It's working there but not here. Guessing something screwed up when we ran out of space?
Here's start.sh:
Code: Select all
#!/bin/sh
LD_LIBRARY_PATH=".:bin:$LD_LIBRARY_PATH"
mono-service2 ugccmonsvc.exe