You are not logged in.

#1 2013-04-26 14:50:30

catch
New member
Registered: 2013-04-20
Posts: 7

Trouble diagnosing "soft crash"

Hi all,

TL;DR Server unresponsive can move cursor anywhere on console including over the login prompt and switch consoles but that's about it, no access via ssh and ZFS RAID seems unresponsive too.

I've had my Funtoo server running almost happily for a week or so now. I'm hoping for some help with a fairly major problem that's happened twice now. My setup is a HP Microserver n40l running Funtoo with zfs and a seperate raidz pool which I currently have to manually import (not sure why it doesn't get manually imported normally).

It's now become mostly unresponsive twice and I don't know how fix, or even diagnose the issue.

Symptoms are:
1. Server stops responding on ssh
2. I go check what the state of the server is on the console and there are error messages about applications that have crashed in the console (I have no x installed) and console won't give me a login prompt (one of the exceptions seemed to be from  program that was complaining about the raidz pool dropping out from under it.)
3. I'm able to move the cursor around anywhere on the screen with the arrow keys.
4. I'm able to switch to different consoles and each has the same weird cursor behavior.

After the first time this happened I realised I had no syslog-ng installed (d'oh) so I installed it following this guide and config verbatim : http://en.gentoo-wiki.com/wiki/Syslog-ng (down to the options section), I didn't have time to tweak anything.

I've now had this issue occur the second time. Checking the logs there's not much in /var/log/messages or /var/log/genkernel.log, I was surprised there's no /var/log/system or /var/log/syslog.

Has anyone come across this issue or know what might be causing it?

Thanks for reading smile

Offline

#2 2013-04-26 18:42:54

catch
New member
Registered: 2013-04-20
Posts: 7

Re: Trouble diagnosing "soft crash"

Ok, so I think the ability to move the cursors on the console is a red herring. The whole system became unresponsive to most things again for the second time today. I rebooted (ctrl-alt-del worked!) and noticed that the same thing happens after a fresh restart, I was able to move the cursor anywhere on the screen when the login prompt is up and type over stuff. I'm also able to do this when I first log out as a user, but not after I've logged in or put a password in incorrectly.

So, I'm now stuck. I need to invest some time figuring out how to make syslog-ng give me more details about what it's up to. If anyone could share a decent working syslog-ng config with me I'd be very great-full.

Thanks

Offline

#3 2013-04-26 20:17:46

leprosys
Member
From: El Salvador
Registered: 2010-10-07
Posts: 54

Re: Trouble diagnosing "soft crash"

I have this config:

@version: 3.3
#
# /etc/syslog-ng/syslog-ng.conf
#

options {
  stats_freq (0);
  flush_lines (0);
  time_reopen (10);
  log_fifo_size (10000);
  chain_hostnames (off);
  use_dns (no);
  use_fqdn (no);
  create_dirs (no);
  keep_hostname (yes);
  perm(0640);
};

source src {
  unix-dgram("/dev/log");
  internal();
  file("/proc/kmsg");
};

destination d_authlog { file("/var/log/auth.log"); };
destination d_syslog { file("/var/log/syslog.log"); };
destination d_cron { file("/var/log/crond.log"); };
destination d_daemon { file("/var/log/daemon.log"); };
destination d_kernel { file("/var/log/kernel.log"); };
destination d_lpr { file("/var/log/lpr.log"); };
destination d_user { file("/var/log/user.log"); };
destination d_uucp { file("/var/log/uucp.log"); };
destination d_mail { file("/var/log/mail.log"); };
destination d_news { file("/var/log/news.log"); };
destination d_ppp { file("/var/log/ppp.log"); };
destination d_debug { file("/var/log/debug.log"); };
destination d_messages { file("/var/log/messages.log"); };
destination d_errors { file("/var/log/errors.log"); };
destination d_everything { file("/var/log/everything.log"); };
destination d_iptables { file("/var/log/iptables.log"); };
destination d_acpid { file("/var/log/acpid.log"); };
destination d_console { usertty("root"); };

# Log everything to tty12
destination console_all { file("/dev/tty12"); };

filter f_auth { facility(auth); };
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { program(syslog-ng); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kernel { facility(kern) and not filter(f_iptables); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };
filter f_ppp { facility(local2); };
filter f_debug { not facility(auth, authpriv, news, mail); };
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news, cron) and not program(syslog-ng) and not filter(f_iptables); };
filter f_everything { level(debug..emerg) and not facility(auth, authpriv); };
filter f_emergency { level(emerg); };
filter f_info { level(info); };
filter f_notice { level(notice); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };
filter f_iptables { match("IN=" value("MESSAGE")) and match("OUT=" value("MESSAGE")); };
filter f_acpid { program("acpid"); };

log { source(src); filter(f_acpid); destination(d_acpid); };
log { source(src); filter(f_authpriv); destination(d_authlog); };
log { source(src); filter(f_syslog); destination(d_syslog); };
log { source(src); filter(f_cron); destination(d_cron); };
log { source(src); filter(f_daemon); destination(d_daemon); };
log { source(src); filter(f_kernel); destination(d_kernel); };
log { source(src); filter(f_lpr); destination(d_lpr); };
log { source(src); filter(f_mail); destination(d_mail); };
log { source(src); filter(f_news); destination(d_news); };
log { source(src); filter(f_ppp); destination(d_ppp); };
log { source(src); filter(f_user); destination(d_user); };
log { source(src); filter(f_uucp); destination(d_uucp); };
#log { source(src); filter(f_debug); destination(d_debug); };
log { source(src); filter(f_messages); destination(d_messages); };
log { source(src); filter(f_err); destination(d_errors); };
log { source(src); filter(f_emergency); destination(d_console); };
log { source(src); filter(f_everything); destination(d_everything); };
log { source(src); filter(f_iptables); destination(d_iptables); };

# Log everything to tty12
#log { source(src); destination(console_all); }

Offline

Board footer

Powered by FluxBB