ProxmoxVE 4.x OpenVZ to LXC migration

At Onlime Webhosting we chose ProxmoxVE as our favorite virtualization platform and are running a bunch of OpenVZ containers for many years now, with almost zero issues. We very much welcome the small overhead and simplicity of container based virtualization and wouldn’t want to move to anything else. ProxmoxVE added ZFS support by integrating ZFSonLinux back in Feb 2015 with the great ProxmoxVE 3.4 release – which actually would have deserved to bump its major version because of this killer feature.

Previously we have been running our OpenVZ containers on plain Debian Linux boxes. But with Debian Wheezy (7.0) the Debian community decided to no longer include a kernel which has been patched with the OpenVZ extensions. Switching to ProxmoxVE at that time was a piece of cake as ProxmoxVE was just a plain Debian Linux with a RHEL kernel, OpenVZ and KVM support, and some nice web interface to manage the containers and VMs. Actually, we never really needed the ProxmoxVE web interface (GUIs always suck!), but that one is quite lightweight and very well integrated. If you don’t like it, use the provided CLI tools – ProxmoxVE does not force you to use the web interface at all.

In 2014 I did a review for Proxmox High Availability (by PACKT publishing). But at Onlime, we never employed any cluster technologies to date. I was always looking for simple solutions without unnecessary complexity. ProxmoxVE with OpenVZ on ZFS pools was the perfect match. We are replicating our containers from one host node to another every 10mins with simple and super fast zfs send|receive via a modified version of ZREP.

In ProxmoxVE 4, OpenVZ support was removed in favor of LXC. Both, KVM and LXC are built into any newer Linux kernel, so in the long term (and due to the nature of OpenVZ as a huge kernel patch which is hard to maintain) it was clear we had to give up OpenVZ. Even though the OpenVZ developers are currently focusing on merging the OpenVZ and Virtuozzo source codebase and have just released OpenVZ 7.0 I don’t give OpenVZ a long future any more. In comparison to LXC, the tools for OpenVZ are still much more mature and powerful, but lately LXC made a lot of progress and having it perfectly integrated into ProxmoxVE is a great step forward. So, we have decided to migrate to LXC.

(Sorry about the long preface. You probably don’t care about history reading this blog post. I am getting to the point now…)

What I cannot believe and what prevented me to look into LXC on ProxmoxVE up to now is the clumsy migration path that ProxmoxVE suggests on ProxmoxVE Wiki: Convert OpenVZ to LXC. Basically they suggest to dump and restore a full container as follows:

Srsly? Do you guys only have containers with sizes below 1GB and don’t care about downtime at all? Real world looks different and I would not even go this migration path with a 5GB container. I don’t even want to think about those containers with ~ 1TB of data.

If you are running OpenVZ containers on ProxmoxVE, sure you have some easy solution to migrate your containers from one host node to another. As I said, we are using ZREP for container replication and have written a wrapper script that migrates a container to its failover host node. That’s what I suggest and please forget about live migration – it never really was working in OpenVZ (well, it was, but I am talking about a really stable solution that never fails) and LXC lacks any kind of hot/live migration. Doing a ZREP presync, stopping the container on the source host node, doing the main ZREP sync, finally starting the container on the destination host node is just a matter of seconds, usually causes a downtime of 10s up to 30s max (on a larger container).

Read More

sendmail-wrapper for PHP

Years ago I wrote about the extensive sendmail wrapper with sender throttling, a pretty simple Perl script. It reliably provided throttling of the email volume per day by the sender’s original UID (user id). It also logged the pathes of scripts that sent emails directly via sendmail (e.g. via PHP’s mail() function). The main flaw in the original sendmail wrapper was security, though. As in Linux, every executable script must be readable by the user that calls it, the throttle table in MySQL was basically open and every customer could manipulate it. Every customer could raise his own throttling limit and circumvent it.

Today, I’m publishing my new sendmail-wrapper that is going to fix all the flaws of the previous version and add some nice extras.
The new sendmail-wrapper is written entirely in PHP and does not require any external libraries. It is a complete rewrite and has pretty much nothing in common with the old Perl version.

This project is hosted on Github

Features:

  • Lets you monitor any mail traffic from PHP scripts
  • Allows throttling (limiting) emails sent by PHP’s mail() function
  • Throttle by sent email and/or recipient count per day
  • Logs both to syslog and database with message metadata
  • Logs common mail headers like From, To, Cc, Bcc, Subject
  • Fixes Return-Path header on the fly for users who did not correctly set it
  • Highly secured setup, customers cannot access the logging/throttling database
  • Standalone PHP application without any external library dependencies
  • Built for shared webhosting environment where PHP runs as cgi/FastCGI/suPHP
  • No cronjobs required, sendmail-wrapper will reset counters automatically every day

Read More

Process hiding: hidepid capabilities of procfs

Five years ago I wrote about kernel based process hiding in Linux (see articles Simple process hiding kernel patch, Process hiding Kernel patch for 2.6.24.x, RSBAC – Kernel based process hiding). It got time to continue the story and finally present you a real solution without the hassle of a self-compiled kernel.

How can I prevent users from seeing processes that do not belong to them?

In January 2012, Vasiliy Kulikov came up with a kernel patch that solved the problem nicely by adding a hidepid mount option for procfs. The patch landed in Linux kernel 3.3.

In the meantime, this patch luckily also landed in the 3.2 kernel of Debian Wheezy (see backport request in Debian bug report #669028). This feature has been also pushed back into the kernel of Red Hat Enterprise Linux 6.3 (see RHEL 6.3 Release Notes), and from there to CentOS 6.3 and Scientific Linux 6.3. Recently, this feature was even backported to the 2.6.18 kernel in RHEL 5.9.

As Proxmox VE currently runs on a RHEL based 2.6.32 kernel, it’s also supported in my favorite OpenVZ/KVM virtualization platform. Great!

Read More

Proxmox VE: Restricting Web UI access

With the release of Proxmox VE 3.0 back in May 2013, the Proxmox VE web interface does no longer require Apache. Instead, they’re using now a new event driven API server called pveproxy. That was actually a great step ahead, as we all know Apache get’s bulkier every day and the new pveproxy is a much more lightweight solution. But the question arose:

How do I protect my Proxmox VE WebUI with basic user authentication?

 

Basically, we do not trust any web application out there so we better double protect the whole WebUI with plain old basic auth – previously done in Apache by .htaccess.

The main idea:

  • Restrict access to the pveproxy (= Web UI) to localhost
  • Install a local Nginx web proxy server that forwards requests from port 443 to pveproxy’s port 8006 and restrict access to it using HTTP BASIC AUTH

Read More

Fix Snow Leopard issues with SMB

Apple’s OS X Leopard (10.5) had problems with accessing Samba shares. Most of them got fixed in late Leopard releases, 10.5.4/10.5.5. During the last Leopard releases the hope arose, that finally OS X will make it into office environments sharing Windows networks.

Then, Apple released Snow Leopard (10.6). Another step into the right direction by providing built-in support for Exchange Server and Google services. But what happened with SMB? The whole story started all over again. While Samba-shares worked pretty well on Leopard, in Snow Leopard all seemed to be messed up again.

Read More

Extensive sendmail wrapper with sender throttling

In this tutorial I’d like to describe how to create an extensive sendmail wrapper for a web server to monitor all sent emails and throttle daily sent email volume by the senders original UID (user id). This is useful if you e.g. run PHP in CGI-mode with SuExec, that is: all customers run their scripts under their own UID. The wrapper described here is not just a PHP-only wrapper (as described in my Simple PHP mail wrapper tutorial) – it directly replaces /usr/sbin/sendmail so we are able track all sent email of the whole system.

Read More

Simple PHP mail wrapper

If you run a webserver with several hundreds of virtual hosts running PHP, you definitely need to monitor or log the access to PHP’s mail() function. I describe in a short tutorial how to painlessly setup a simple sendmail wrapper to accomplish this.
This has been tested on a Debian Lenny 5.0 system running PHP 5.2.8 and Postfix.

Read More

Process hiding Kernel patch for 2.6.24.x

Currently all Linux kernel security patch projects seem to be sleeping. There is no useful kernel patch that provides us with a decent patch set allowing us to strengthen the Linux kernel. Some years ago I was using Grsecurity, a wonderful solution to enforce security on 2.4.x kernels at that time. The project seems to be pretty dead by now.

During the last months I was using RSBAC, a great set of security enhancements to the 2.6.x kernels. RSBAC seems to be a great project and I like the way they provide pre-patched vanilla kernels. But again, reaction time is way too slow. Root exploits for Linux kernels seem to appear all the time and force a server administrator to react fast. The lately published vmsplice root exploit made me give up on RSBAC as it’s just always a step behind. I decided to switch back to self compiled vanilla kernels from kernel.org.

Read More

ProFTPd xferlog via MySQL

Logging your FTP transfers to xferlog with ProFTPd is a nice thing. This can easily be done by a one-liner in /etc/proftpd/proftpd.conf:

This generates a nice transfer log which we could then parse for transfer statistics. But there is a much better way to accomplish this: MySQL. Let’s use MySQL for everything!
It’s pretty straightforwarded to get ProFTPd to log into a MySQL table.

Read More

RSBAC – Kernel based process hiding

A webserver usually is the primary target to intrude into any network. If you provide web hosting services for your customers you have to provide them with a lot of features to make them happy. The main requirement for any hosting provider is PHP, probably the widest spread web scripting language out there.

Some customers only start to get happy if you give them PHP without any safe_mode restrictions, if you provide them with custom CGI scripting next to the basic good old SSI features (which in my eyes no one really needs since we got PHP) by Apache HTTP Server, if you give them FTP access and let them manage their account by themselves.

Rule Set Based Access ControlIn every feature there is always a hidden security risk. We cannot give all this to our customers without thinking about security and its consequences if a user gets hold of data which does not belong to himself or even breaks into the whole system. So, let’s start at the basics: No customer should be able to see any other running processes on the system except the ones that belong to himself. We want to hide all processes that the given customer is not allowed to see. That’s process hiding. And because on a Linux box it’s always smart to implement something from bottom up, we name it kernel based.

There is no simple solution for this problem. Some rootkits simply overwrite the ‚ps‘ command. But we want something more trustworthy, somehow deeper anchored in the system (got that?). The only kernel patch I found was the one from RSBAC.org (Rule Set Based Access Control), a full blown kernel security patch. The only feature we actually need is „CAP process hiding“.

Read More

css.php