I am used to have my own server on the Internet. Ever since I first got in touch with the Net in mid-90’s I have had at least one server of my own serving my emails, domains, websites, etc. I used to have the machine my server was running on in a colocation facility (aka ISP server housing) in Prague, but then in 2005 I moved to New Zealand, leaving my server some 20.000km behind. That soon became a problem. Whenever the hardware experienced issues or the OS needed an upgrade I was inconveniently too far. Luckily the ISP staff was very helpful and they often tried hard to help me out but the distance between me and the server was a real issue. Finally, in June 2008, I undertook the big step and shifted from my very own hardware to a hosted Virtual Private Server (VPS).

My first VPS

I played with VMware and Xen before, but didn’t have any prior experience with commercial VPS offerings. I pretty much randomly picked one of the providers and paid for a mid-sized VPS. Yet before I could move my data across I had to accept some new limits of my new OpenVZ-based VPS:

  • I used to use POSIX ACLs for access control on the multiuser server. That was not possible anymore, the VPS provider insisted OpenVZ didn’t support POSIX ACL.
  • I used to be connected to a native IPv6 network — now I couldn’t even get a tunnel since the VPS kernel didn’t have IPv6 compiled in.
  • OpenVPN setup was a bit of a challenge as the kernel didn’t have TUN driver enabled by default.
  • I used to have Apache chrooted in noexec,nodev mounted partition for security reasons. Again, impossible.
  • … and so on

Such a VPS is probably a reasonable choice for a website hosting, less so for a multiuser multipurpose internet server. Sadly I was in a bit of a pressure and didn’t have time to look for something better. Anyway, few days later I had CentOS 5 set up, all the required services — web, mail, dns, mysql, … — were running and I didn’t have to care about the hardware anymore. All good.
However after some time I started experiencing more and more problems ranging from occasional poor performance and unplanned outages right to lost files and directories after a host-server crash. Then they raised their prices by 20% and I decided to move on. But where?


IPv6 readyMy next VPS must be Linux, if possible CentOS 5 since that’s what I have now. Migration in such a case would be a matter of copying 99% of files across, leaving alone just the network config. I also want to get back all the features from the list above. While most of them are fairly common, especially on Xen-powered hostings, the native IPv6 requirement was a tough nut to crack. After going through many websites and contacting some 30 VPS companies I compiled a list of 5 (yes, only five) native IPv6 enabled VPS hostings. Increible! Here they are:

VPS Provider Country Note
goscomb.net United Kingdom
rapidxen.net USA
serveraxis.com USA
xencon.net Germany
verio.com USA FreeBSD only,
no Linux

I’m sure there are more but I couldn’t find them. Perhaps all those missing ones treat IPv6 as an established technology not worth mentioning on their websites. I wish. But I doubt …
Let me know if you’re aware of anyone else with native IPv6, I’ll add them to the table.

My next VPS

I decided to go ahead with Goscomb — for start I’ve got a small VPS for testing and am pretty impressed. It’s a fully virtualised Xen host which means it behaves like a real machine — there is GRUB boot loader, it could run my own kernel if I wanted to, I can insert any kernel modules I need, have a total control over the (virtual) disk, etc. Just like a real system. It even has an out-of-band console via VNC which means I can play with a firewall setup without worrying that I cut my access short. And it does have native IPv6, yay! Most of the Europe is just a few hops away and traffic to US is reasonably fast as well. From New Zealand everything is slow so it doesn’t really matter if the server is in US or UK.
There are only two downsides I discovered so far — Goscomb doesn’t accept PayPal payments and doesn’t provide remote VPS reset. Both are pretty minor — for payments I can use my VISA card and for forced restart, if I ever need one, will open a support ticket — so far they were responding reasonably fast.
I’ll post an update once my server — and this blog — is moved to its new home and IPv6 enabled :-)

Untrusted SSL certificate in Citrix ICAclient on Linux

Today I was asked to perform some Linux server maintenance for an important client. They use Citrix Access Gateway™ (CAG) for remote access to their systems. The plan was to login to the Citrix Access Gateway web interface from Firefox, click the PuTTY icon, let CAG server execute the PuTTY SSH client and display its interface on my Linux desktop via a locally installed ICA client. Once it’s up enter the remote Linux server’s IP and do my job. Unfortunately when I clicked the PuTTY icon things went wrong instantly:

You have not chosen to trust 'Thawte Server CA', the issuer of the server's security certificate.

You have not chosen to trust ‘Thawte Server CA’, the issuer of the server’s security certificate.

Aha, now what? Apparently Citrix ICAclient comes with its own set of trusted Certification Authorities. Both Firefox and my system-wide CA list are set to trust Thawte but ICAclient didn’t. After a bit of research I have found the ICAclient’s trusted certificates are stored in ~/ICAClient/linuxx86/keystore/cacerts/ directory:

~/ICAClient/linuxx86/keystore/cacerts $ ls -l
-r--r--r-- 1 mludvig users 891 2009-06-07 12:00 BTCTRoot.crt
-r--r--r-- 1 mludvig users 774 2009-06-07 12:00 Class3PCA_G2_v2.crt
-r--r--r-- 1 mludvig users 774 2009-06-07 12:00 Class4PCA_G2_v2.crt
-r--r--r-- 1 mludvig users 606 2009-06-07 12:00 GTECTGlobalRoot.crt
-r--r--r-- 1 mludvig users 576 2009-06-07 12:00 Pcs3ss_v4.crt
-r--r--r-- 1 mludvig users 568 2009-06-07 12:00 SecureServer.crt

That’s a good start. Now find out in what format these certs are — two most common formats are DER (binary format) and PEM (ASCII encoded). Quick check reveals that these files are binary, therefore most likely in DER format. Verify the assumption with openssl:

.../cacerts $ openssl x509 -inform der -text -noout -in BTCTRoot.crt
        Version: 3 (0x2)
        Serial Number: 33554617 (0x20000b9)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root

Very good — openssl was told to open it as DER and gave us a reasonable output, so it’s DER! Now we need to get the Thawte Server CA certificate from somewhere, convert it to DER format and save into this directory. It is almost certain that the Citrix Access Gateway web interface uses the very same SSL certificate that the ICA client complains about. So … grab it from there!

Right click somewhere on the page and select View Page Info — a Page info dialog should pop up. Select the last tab — Security — and then View Certificate

Page info — View Certificate

Page info — View Certificate

Certificate Viewer will pop up. Select the second tab — Details. There, in the Certificate Hierarchy tree select to top-most item, Thawte Server CA in our case. Click the Export button at the bottom and save the certificate for example as ~/ThawteServerCA.pem.

Certificate Viewer

Export “Thawte Server CA” certificate

Now the last step on our quest — convert the certificate from PEM format to DER with the help of openssl and verify that it worked out. Note that we’re still in the ICAclient’s keystore/cacerts directory:

.../cacerts $ openssl x509 -inform pem -outform der -in ~/ThawteServerCA.pem -out ThawteServerCA.crt
.../cacerts $ openssl x509 -inform der -in ThawteServerCA.crt -text -noout
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: md5WithRSAEncryption
        Issuer: C=ZA, ST=Western Cape, L=Cape Town, O=Thawte Consulting cc, 
                   OU=Certification Services Division, 
                   CN=Thawte Server CA
        Subject: C=ZA, ST=Western Cape, L=Cape Town, O=Thawte Consulting cc, 
                   OU=Certification Services Division, 
                   CN=Thawte Server CA

All right, we’re set to go. Click on the PuTTY icon again and see how far we get.

Starting PuTTY

PuTTY is now starting

Voilà, things look good. A while later PuTTY is up, running on the remote Citrix Access Gateway server, ready to open a SSH connection to the Linux box in the company’s internal network.