Rebooting linux machine without disk access

This is a weird situation which I found myself in recently. One of the ubuntu servers were having issues and was unable to launch any process for unknown reason (kernel issue/disk or filesystem failure/etc). Puzzled, I tried to reboot the machine using shutdown and reboot, but the system did not allow me to. It also failed to run utilities such as ping, ifconfig, etc…

After some googling, here is the magic (“magic SysRq key”):

echo 1 >/proc/sys/kernel/sysrq
echo s > /proc/sysrq-trigger (optional, sync mounted filesystems)
echo u > /proc/sysrq-trigger (optional, remount filesystems as read-only)
echo b > /proc/sysrq-trigger

Aha! echo is a bash shell builtin command and does not have to be a new process. Reboot now!

Linux Magic System Request Key Hacks

Advertisements

apt-get stuff

1. Add redundant repo

In /etc/apt/sources.list, add:

deb http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse
deb http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ trusty-backports main restricted universe multiverse

 

2. Set timeout

Create /etc/apt/apt.conf.d/99timeout, add:

Acquire::http::Timeout "10";

 

apt-get away!

 

apt – How to lower wait time for repository updates – Ask Ubuntu

Realtek 8168 module issue

My cute little dual-core atom system (D945GCLF2) had a small hiccup with Debian. The kernel kept loading on boot the r8169 (incorrect) module for the onboard network device, which caused all sorts of weird phenomenon including, but not limited to, strange tasting coffee, distracted cats, ifconfig showing seemingly random numbers for dropped frames every time and not obtaining IP from DHCP servers occasionally.

The high-level solution is to replace the incorrect module with the appropriate one. Here goes:

  • Obtain the correct module (r8168)

Go to Realtek website and download the source for the latest driver.

Extract the archive:

tar -jxf r8168-x.y.z.tar.bz2

Prepare build environment:

apt-get install build-essential linux-headers-`uname -r`

Compile the driver source:

cd r8168-x.y.z
make clean modules && make install
  • Replace using the correct module

Before moving on, take note that removing the module for networking will disable the network interface. Which means any remote session running on the interface will be terminated.

Unload the current (incorrect) module:

rmmod r8169

Generate module dependency:

depmod

Load the correct module:

modprobe r8168

Update existing initramfs:

update-initramfs -u

If the kernel insists on loading the r8169 module, add the line blacklist r8169 into /etc/modprobe.d/blacklist:

echo blacklist r8169 >> /etc/modprobe.d/blacklist

(edited 18-07-2012, thanks to x in comments)
or, for newer distros:

echo blacklist r8169 >> /etc/modprobe.d/blacklist.conf

Hopefully all is good after going through the troubles.

p/s: My r8168.ko (Compiled for Lenny, 2.6.26-2-686) can be found under Resources.

S.M.A.R.T. stuff

RAID setup is nice – it provides redundancy and makes you buy more hard disks (and drives the economy). But the advantages are somehow retrospective; it is only realized when a disk gives up and calls it a day and you suddenly feel so smart (no pun here) that you had a RAID setup. For us to take preemptive measures, we can use smartmontools to monitor the vitals of the disks. So we know when the disk is reaching the end of its life and plan ahead, like getting a replacement, migrate to another system, etc.

apt-get install smartmontools                          # or

apt-get install smartmontools --no-install-recommends  # to install w/o the recommended mail server

Under /etc/default/smartmontools, uncomment a line to have smartd start on system startup:

start_smartd=yes

Configurations are found in /etc/smartd.conf. I chose to combine 2 examples to come up with the following:

DEVICESCAN -m root -M exec /usr/share/smartmontools/smartd-runner -R 194 -R 231 -I 9

It means to scan for all devices, monitor them using default settings, send an email to root on failures, show raw Temperature_Celsius readings in /var/log/syslog and ignore Power_On_Hours. Now I drive the economy before the disk fails on me 🙂

Manpage of SMARTD.CONF

Install unstable packages on lenny

I run lenny on my home server. That said, there are times when I want new packages which are not available on lenny backports. One solution is apt pinning (another is compile from source :p).

First, add the unstable repo entry in /etc/apt/source.list:

deb http://ftp.us.debian.org/debian unstable main

Then, add the following in /etc/apt/preferences:

Package: *
Pin: release a=unstable
Pin-Priority: 800

While this works generally, please refer to apt_preferences manpage for more details. Finally, run

apt-get update && apt-get install -t unstable <your-new-package>

to update and install the package from unstable. Depending on individual requirements, you may choose to substitute testing for unstable. Enjoy!

AptPinning – Debian Wiki