• linux mis cpu usage

    From esc@21:2/142 to All on Thursday, December 21, 2017 19:25:48
    Hey guys,

    I noticed I couldn't ssh into my bbs today (I prefer to not use telnet basically ever) but telnet was working fine. I ssh'd into the server itself
    and noticed in 'top' that mis was eating up 100% of my CPU. I bounced the mis service and now it's hovering around 5ish percent of CPU. There's nothing in error.log and nothing interesting in mis.log. Any ideas?

    --- Mystic BBS v1.12 A36 2017/12/03 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From Hawk Hubbard@21:1/188 to esc on Thursday, December 21, 2017 15:40:35
    Hey guys,
    I noticed I couldn't ssh into my bbs today (I prefer to not use telnet basically ever) but telnet was working fine. I ssh'd into the server itself and noticed in 'top' that mis was eating up 100% of my CPU. I bounced the mis service and now it's hovering around 5ish percent of
    CPU. There's nothing in error.log and nothing interesting in mis.log.
    Any ideas?

    I dont't have any idea but nothing escapes your watching eye.
    The sentinel of the White Tower.

    ²ß±ß° {tHE.pIRATE.kING} ÜßÛßÜ
    ß²Ü°ß HOIST THE COLOURS ßÛÜÛß
    þþþ NEVER SHALL WE DIE. ÝÝÞ

    --- Mystic BBS v1.12 A37 2017/12/13 (Raspberry Pi/32)
    * Origin: ACiD Underworld // Victor Ludorum (21:1/188)
  • From g00r00@21:1/108 to esc on Thursday, December 21, 2017 16:42:28
    itself and noticed in 'top' that mis was eating up 100% of my CPU. I bounced the mis service and now it's hovering around 5ish percent of
    CPU. There's nothing in error.log and nothing interesting in mis.log.
    Any ideas?

    If this is an ongoing problem you could consider upgrading to the latest
    test versions and we can go from there if the problem persists.

    Were any users "stuck" online?

    --- Mystic BBS v1.12 A37 2017/12/21 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)
  • From esc@21:2/142 to g00r00 on Friday, December 22, 2017 01:44:33
    itself and noticed in 'top' that mis was eating up 100% of my CPU. I bounced the mis service and now it's hovering around 5ish percent of CPU. There's nothing in error.log and nothing interesting in mis.log. Any ideas?

    If this is an ongoing problem you could consider upgrading to the latest test versions and we can go from there if the problem persists.

    Were any users "stuck" online?

    Hi g00r00,

    Yeah, it's back up to 100% again for some reason, though I'm able to use the BBS for now. Nothing in any mystic logs or in syslog, for that matter.

    No users stuck online. I'm on alpha 36, 32 bit linux. So I should upgrade,
    huh? I can give that a shot.

    --- Mystic BBS v1.12 A36 2017/12/03 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From Pequito@21:1/126 to esc on Thursday, December 21, 2017 19:51:25
    On 12/22/17, esc said the following...

    itself and noticed in 'top' that mis was eating up 100% of my CP bounced the mis service and now it's hovering around 5ish percen CPU. There's nothing in error.log and nothing interesting in mis Any ideas?

    If this is an ongoing problem you could consider upgrading to the lat test versions and we can go from there if the problem persists.

    Were any users "stuck" online?

    Hi g00r00,

    Yeah, it's back up to 100% again for some reason, though I'm able to use the BBS for now. Nothing in any mystic logs or in syslog, for that
    matter.

    No users stuck online. I'm on alpha 36, 32 bit linux. So I should
    upgrade, huh? I can give that a shot.


    Curious you are only loading MIS now with A37 I think the late A36. g00r00 changed the name of mis2 to mis and removed mis v1 all together. I ask this due to the possibility of you using both mis/mis2 still and the events had
    been transfered to the new mis so only one instance was needed. :)

    Cheers!
    Pequito

    --- Mystic BBS v1.12 A37 2017/12/16 (Linux/64)
    * Origin: Twinkle BBS # (21:1/126)
  • From esc@21:2/142 to Pequito on Friday, December 22, 2017 03:39:54
    Curious you are only loading MIS now with A37 I think the late A36. g00r00 changed the name of mis2 to mis and removed mis v1 all together.
    I ask this due to the possibility of you using both mis/mis2 still and
    the events had been transfered to the new mis so only one instance was needed. :)

    Hi Pequito, thanks :) I actually completely killed the vm slice where the mis/mis2 configuration issues lived. I then moved to install a fresh a36. I just now upgraded to a37, and I'm keeping an eye open for the .bsy file issue
    I ran into before. Fingers crossed...

    --- Mystic BBS v1.12 A37 2017/12/21 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From g00r00@21:1/108 to esc on Friday, December 22, 2017 15:01:05
    Hi Pequito, thanks :) I actually completely killed the vm slice where the mis/mis2 configuration issues lived. I then moved to install a fresh
    a36. I just now upgraded to a37, and I'm keeping an eye open for the
    .bsy file issue I ran into before. Fingers crossed...

    Now that you have the latest version of A37 you will see a lot of extra logging...

    There will be a "busylog.txt" in the root Mystic directory (or should be) and that will allow us to match up every create/remove of a BSY file so we can at least see where it happens, if it happens.

    There will also be .mem files created so if you have a crash it can (hopefully) tell me an estimated line in the code where it happened. I'll also add in any additional logging I can surrounding any issue you're having if needed and I can update the prealpha site instantly.

    The downsides with the latest prealphas are that I accidentally compiled it with BINKP debug logging so those BINKP logs get pretty cluttered. The binaries are also up to 10 times the size they normally are, and things in general will run a little slower... but it should go a long way to help me resolve your issues.

    --- Mystic BBS v1.12 A37 2017/12/21 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)
  • From esc@21:2/142 to g00r00 on Tuesday, December 26, 2017 06:30:00
    There will also be .mem files created so if you have a crash it can (hopefully) tell me an estimated line in the code where it happened.
    I'll also add in any additional logging I can surrounding any issue
    you're having if needed and I can update the prealpha site instantly.
    The downsides with the latest prealphas are that I accidentally compiled it with BINKP debug logging so those BINKP logs get pretty cluttered.
    The binaries are also up to 10 times the size they normally are, and things in general will run a little slower... but it should go a long
    way to help me resolve your issues.

    Hi g00r00,

    Awesome. So far so good with the .bsy files, no issues.

    One thing that was weird was that a file, /mystic/logs/eventsignal.txt was being created and filling up faster than I could kill it. It was filling up like hundreds of gigs a day. My only workaround thus far was to actually shut down mis, kill eventsignal.txt, create an empty eventsignal.txt, and chmod -w the file (so it couldn't be written to).

    However now I do keep running into the issue of mis taking up 100% of CPU.
    Any ideas where to start troubleshooting?

    --- Mystic BBS v1.12 A37 2017/12/21 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From g00r00@21:1/108 to esc on Tuesday, December 26, 2017 12:30:15
    One thing that was weird was that a file, /mystic/logs/eventsignal.txt
    was being created and filling up faster than I could kill it. It was filling up like hundreds of gigs a day. My only workaround thus far was
    to actually shut down mis, kill eventsignal.txt, create an empty eventsignal.txt, and chmod -w the file (so it couldn't be written to).

    Any idea of what events were in it that was filling it up?

    However now I do keep running into the issue of mis taking up 100% of
    CPU. Any ideas where to start troubleshooting?

    I am not able to reproduce any issue like this so far, but I haven't given it enough time because the past couple of days have been with friends and family for the most part. The little I had for Mystic I've been working on things easy to pick up and put down like the text/ANSI editors (which are almost done).

    If you know what event(s) were filling up the debug event file that might be a clue.

    Someone else was suggesting that it might be related to SSH - do you think that could be possible? Are you able to disable SSH to see if it stops the problem? Does it happen immediately every time or just randomly after running for a while?

    I'll try to see if I can get it to happen soon. I am going to put my BBS up via SSH on Linux/64 for a while and work on any stability issues.

    --- Mystic BBS v1.12 A37 2017/12/26 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)
  • From Nugax@21:1/107 to All on Tuesday, December 26, 2017 14:59:13
    Might want to set a file size limit on log files too... just a thought

    On 06:30 26/12 , g00r00 wrote:
    One thing that was weird was that a file, /mystic/logs/eventsignal.txt was being created and filling up faster than I could kill it. It was filling up like hundreds of gigs a day. My only workaround thus far was to actually shut down mis, kill eventsignal.txt, create an empty eventsignal.txt, and chmod -w the file (so it couldn't be written to).

    Any idea of what events were in it that was filling it up?

    However now I do keep running into the issue of mis taking up 100% of CPU. Any ideas where to start troubleshooting?

    I am not able to reproduce any issue like this so far, but I haven't given it >enough time because the past couple of days have been with friends and family >for the most part. The little I had for Mystic I've been working on things >easy to pick up and put down like the text/ANSI editors (which are almost >done).

    If you know what event(s) were filling up the debug event file that might be a >clue.

    Someone else was suggesting that it might be related to SSH - do you think that
    could be possible? Are you able to disable SSH to see if it stops the problem?
    Does it happen immediately every time or just randomly after running for a >while?

    I'll try to see if I can get it to happen soon. I am going to put my BBS up >via SSH on Linux/64 for a while and work on any stability issues.

    --- Mystic BBS v1.12 A37 2017/12/26 (Windows/32)
    * Origin: Sector 7 [Mystic BBS WHQ] (21:1/108)


    --
    yrNews Usenet Reader for iOS
    http://appstore.com/yrNewsUsenetReader

    --- Mystic BBS/NNTP v1.12 A37 2017/12/16 (Linux/64)
    * Origin: -=The ByteXchange BBS : bbs.thebytexchange.com=- (21:1/107)
  • From Tony Langdon@21:1/143 to g00r00 on Tuesday, December 26, 2017 22:22:48
    g00r00 wrote to esc <=-

    One thing that was weird was that a file, /mystic/logs/eventsignal.txt
    was being created and filling up faster than I could kill it. It was filling up like hundreds of gigs a day. My only workaround thus far was
    to actually shut down mis, kill eventsignal.txt, create an empty eventsignal.txt, and chmod -w the file (so it couldn't be written to).

    Any idea of what events were in it that was filling it up?

    Sounds like the time my eventsignal.txt was filled with signal 11 events (basically filled up my SD). That particular issue hasn't re-occured so far *touches wood*. :)


    ... DOS never says EXCELLENT command or filename...
    ___ MultiMail/Win32 v0.49

    --- Mystic BBS/QWK v1.12 A36 2017/12/04 (Raspberry Pi/32)
    * Origin: The Bridge - bridge.vkradio.com (21:1/143)
  • From Tony Langdon@21:1/143 to Nugax on Tuesday, December 26, 2017 22:22:48
    Nugax wrote to All <=-

    Might want to set a file size limit on log files too... just a thought

    Yep. Can be helpful. One open source package I use has a 10M size limit (in normal operation with even monthly log rotation, this is never reached), so certain hardware issues didn't cause the log to grow and fill the disk.


    ... THE fIRST sTEP iS tO tAKE oFF tHE cAPS lOCK.
    ___ MultiMail/Win32 v0.49

    --- Mystic BBS/QWK v1.12 A36 2017/12/04 (Raspberry Pi/32)
    * Origin: The Bridge - bridge.vkradio.com (21:1/143)
  • From esc@21:2/142 to g00r00 on Wednesday, December 27, 2017 01:17:48
    Any idea of what events were in it that was filling it up?

    Sadly, no. I suppose hypothetically I could remove the temporary non-write
    file I created and see...

    I am not able to reproduce any issue like this so far, but I haven't
    given it enough time because the past couple of days have been with friends and family for the most part. The little I had for Mystic I've been working on things easy to pick up and put down like the text/ANSI editors (which are almost done).

    Dude! I'm not even paying you. You owe me nothing. Please don't ever feel the need to explain :)

    If you know what event(s) were filling up the debug event file that
    might be a clue.

    This being eventsignal.txt?

    I should mention I have the a37 with the rampant .mem file creation. I'd be happy to poke aroun in there if you thought it was useful.

    Someone else was suggesting that it might be related to SSH - do you
    think that could be possible? Are you able to disable SSH to see if it stops the problem? Does it happen immediately every time or just
    randomly after running for a while?

    It seems to take a good day or so for this to happen. It's pretty bizarre.
    I wonder if I'm being portscanned or someone's able to ddos mis somehow. Interestingly enough, when it's using 100% cpu, ssh doesn't work at all, but telnet still does.

    I _could_ disable ssh and use telnet but not for a couple of weeks. I'm
    camping at various RV parks with crap open guest wifi and am not a huge fan
    of telnetting in the clear. Sadly, wrapping stuff in my vpn is tough, because it totally crushes network perf.

    --- Mystic BBS v1.12 A37 2017/12/21 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From Tony Langdon@21:1/143 to esc on Wednesday, December 27, 2017 02:01:17
    esc wrote to g00r00 <=-

    I _could_ disable ssh and use telnet but not for a couple of weeks. I'm camping at various RV parks with crap open guest wifi and am not a huge fan of telnetting in the clear. Sadly, wrapping stuff in my vpn is
    tough, because it totally crushes network perf.

    If you have a Linux box on your etwork, why not use SSH forwarding to securely carry a telnet session over the Internet? I do this on my Windows XP laptop that won't run SyncTerm. I run PuTTY with the port forwarding setup to the BBS. That PuTTY session connects to a Linux box inside the network. Once the SSH session is active, I point Netrunner at localhost on the appropriate port for the session ! want. Bingo! all BBS communication is inside SSH encryption. :)


    ... Between two evils, I always pick the one I never tried before.
    ___ MultiMail/Win32 v0.49

    --- Mystic BBS/QWK v1.12 A36 2017/12/04 (Raspberry Pi/32)
    * Origin: The Bridge - bridge.vkradio.com (21:1/143)
  • From Pequito@21:1/126 to esc on Tuesday, December 26, 2017 20:15:58
    On 12/27/17, esc said the following...

    Any idea of what events were in it that was filling it up?

    Sadly, no. I suppose hypothetically I could remove the temporary
    non-write file I created and see...

    Think he meant check mystic/cfg/events and see what is active/running there. If you run in daemon maybe try out in server mode for a day so could also monitor it and see what happens? :)

    I am not able to reproduce any issue like this so far, but I haven't given it enough time because the past couple of days have been with friends and family for the most part. The little I had for Mystic I' been working on things easy to pick up and put down like the text/ANS editors (which are almost done).

    Dude! I'm not even paying you. You owe me nothing. Please don't ever
    feel the need to explain :)


    Yeah this is one way back but since the new a35-a37 I never had any 100% cpu issues but when I did in earlier versions it could either be a door program/dosemu exit and or an event that is hung.

    It seems to take a good day or so for this to happen. It's pretty
    bizarre. I wonder if I'm being portscanned or someone's able to ddos mis somehow. Interestingly enough, when it's using 100% cpu, ssh doesn't
    work at all, but telnet still does.

    I can concur here ssh still crashes the user and closes the connection after
    a short time, telnet/rlogin are working without issues here.

    I _could_ disable ssh and use telnet but not for a couple of weeks. I'm camping at various RV parks with crap open guest wifi and am not a huge fan of telnetting in the clear. Sadly, wrapping stuff in my vpn is
    tough, because it totally crushes network perf.

    Other thing I can think of close out mis and in the cfg re-create the telnet server and restart mis and see if this might help? Also when you upgraded
    from a?? to a37 did you run the upgrade at least once? :)

    Cheers!
    Pequito

    --- Mystic BBS v1.12 A37 2017/12/23 (Linux/64)
    * Origin: Twinkle BBS # (21:1/126)
  • From esc@21:2/142 to Tony Langdon on Wednesday, December 27, 2017 06:58:28
    If you have a Linux box on your etwork, why not use SSH forwarding to securely carry a telnet session over the Internet? I do this on my Windows XP laptop that won't run SyncTerm. I run PuTTY with the port forwarding setup to the BBS. That PuTTY session connects to a Linux box inside the network. Once the SSH session is active, I point Netrunner
    at localhost on the appropriate port for the session ! want. Bingo!
    all BBS communication is inside SSH encryption. :)

    Yeah, I suppose I could port forward from putty. I haven't yet switched off
    ssh in mis, cpu usage is still hovering under 5% :)

    --- Mystic BBS v1.12 A37 2017/12/21 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From esc@21:2/142 to Pequito on Wednesday, December 27, 2017 07:01:49
    Think he meant check mystic/cfg/events and see what is active/running there. If you run in daemon maybe try out in server mode for a day so could also monitor it and see what happens? :)

    Ah, gotcha. The only events that run are the standard mutil/fidotoss hourly jobs as well as semaphore jobs for mail. There is a nightly script to run at midnight, but that doesn't appear to have ever gotten stuck. Hypothetically I could just cronjob that one.

    Other thing I can think of close out mis and in the cfg re-create the telnet server and restart mis and see if this might help? Also when you upgraded from a?? to a37 did you run the upgrade at least once? :)

    Ok, I can putz around in mystic -cfg and flip things on/off and relaunch mis. Certainly can't hurt anything.

    I did run the upgrade task, it was a simple upgrade from a36 to a37. So, not sure...

    --- Mystic BBS v1.12 A37 2017/12/21 (Linux/32)
    * Origin: lo fidelity bbs (21:2/142)
  • From Tony Langdon@21:1/143 to esc on Thursday, December 28, 2017 02:41:08
    esc wrote to vk3jed <=-

    Yeah, I suppose I could port forward from putty. I haven't yet switched off ssh in mis, cpu usage is still hovering under 5% :)

    SSH port forwarding is often overlooked, but very handy for tight spots. :)


    ... Patriotism is not who can leak the most Secret documents to the NY Times... ___ MultiMail/Win32 v0.49

    --- Mystic BBS/QWK v1.12 A36 2017/12/04 (Raspberry Pi/32)
    * Origin: The Bridge - bridge.vkradio.com (21:1/143)