HintShop

Let’s put two things together today…

1. The BusPirate is the ultimate tinkerers’ tool that makes the SPI, I2C, 1-Wire, UART, JTAG and some other low-level protocols available to an ordinary PC equipped with a serial or USB port. That way a PC can talk to many digital electronic components, from temperature sensors through microcontrollers and memories to I/O devices like LCD displays or ethernet interfaces.
    
2. Uploading a firmware to an Atmel AVR microcontroller (MCU) is possible through many different ways, with the most common and universal probably being ICSP — In Circuit System Programming. In essence ICSP is an SPI-based protocol, where the programmer sends special programming commands to the AVR chip along with data to be written to the flash memory. In most cases the MCU doesn’t even need to be removed from its circuit first.

• So the AVR can be programmed using SPI and the BusPirate can talk SPI, therefore the BusPirate can program AVRs, correct?

Yes, of course it can! All you need is a recent enough avrdude — either the current SVN checkout or avrdude 5.7 once it is released.

BusPirate with 2 independent AVRs.

AVRdude with BusPirate

Programming one AVR is a breeze, let’s have an example of ATtiny2313. Connect BusPirate to the IC like this:

It is possible, although not required, to power the chip from the BusPirate during programming. Even if the BP is not used as a Vcc supply for AVR their GNDs should still be interconnected.

All right, the ultracomplex “circuit” is ready to go. It’s time to test if it works…

.../attiny2313/test $ avrdude -p attiny2313 -c buspirate
Detecting BusPirate...
**
**  Bus Pirate v1a
**  Firmware v3.0
**  DEVID:0x0447 REVID:0x3003 (A3)
**  http://dangerousprototypes.com
**
BusPirate: using BINARY mode
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.02s

avrdude: Device signature = 0x1e910a
avrdude: safemode: Fuses OK
avrdude done.  Thank you.

avrdude was able to read the device signature so the connection apparently works. Now it’s time to run avrdude with -U flash:w:project.hex to actually flash the firmware in.

One … two … three AVRs!

Any ICSP programmer can program one chip at a time. However the BusPirate goes a step further — we can connect up to 3 independent AVRs to a single BP and without any re-wiring program them. Each with a different firmware of course.

How come? The programmer grabs AVR’s attention by pulling its RESET pin Low. Check again the table above: BP’s CS (Chip Select) is connected to AVR’s RESET — once CS->RESET signal goes down (and stays down) the AVR is ready to accept SPI programming commands. However the BusPirate also has an AUX output pin that is independent on CS. If we add one more AVR to the picture with the same MISO, MOSI and CLK setup, but with its RESET connected to BP’s AUX instead of CS we’re instantly able to program either of the two chips because avrdude can be ordered to pull down AUX instead of CS using -x reset=aux parameter.

.../at90usb162/test $ avrdude -p at90usb162 -c buspirate -x reset=aux
Detecting BusPirate...
[...]
avrdude: Device signature = 0x1e9482
avrdude: safemode: Fuses OK

Talking to the other AVR works as well and its signature (different to the first one) has been read.

I also have an older BusPirate “clone” from Fundamental Logic — the hardware revision is v1a which is the only one that has one additional pin — AUX2. Obviously you can now add a 3^rd AVR into the circuit and use avrdude -x reset=aux2 for its programming.

… four … five … six … seven AVRs!

What? Is there a BusPirate with 7 AUX’es?! No, there is not. However we can command avrdude not only to pull down either CS or AUX or AUX2, we can tell it to pull down, say CS and AUX2, while leaving AUX high: avrdude -x reset=cs,aux2. So what? Well then we can use a simple 3-bits to 1-of-8 decoder (for example 74HCT138), connect CS, AUX and AUX2 to its address inputs and use the 1-of-8-is-Low outputs for selecting the AVR to talk to. That would give us access to up to 7 independent AVRs in a single circuit with a single BusPirate (v1a only though). Why 7 and not all 8? Figure that out for a homework ;-)

BusPirate with 3-bits to 1-of-8 decoder.
Only one output [Y0..Y7] is low at any time.

The same approach can be used to program three AVRs from a newer BusPirates without AUX2 output. Simply get a 2-bits to 1-of-4 decoder (for example 74HCT155) and, again, as a homework think of why we can only attach 3 AVRs and not all four ;-)

Decoupling

There is a little catch though. If all the AVRs don’t belong to the same design they should not be interconnected through SPI lines either. For example I’ve got ATtiny2313 and AT90USB162 on my breadboard but they do different things — the ATtiny is a temperature display with a SPI-connected sensor, while the AT90 has an ENC28J60 ethernet interface attached to its SPI pins. These two SPI buses are totally independent and we can’t simply connect all the MISOs and MOSIs together. We need to decouple them from each other and only pass the BusPirate SPI through to a single chip at a time. How? Using a 3-state bus buffer, for example 74HCT244. Since it’s got 3-state outputs it is virtually non-existent on the SPI bus unless active. A RESET Low signal coming from BusPirate’s CS or AUX or from the 1-of-N decoder activates the outputs of one of the 4-line gates and enables passing the SPI traffic between the BusPirate and the selected AVR. No other AVR will notice that one of these little silicon guys is being programmed. Huh? All right, here’s the schema ;-)

Decoupling the BusPirate from AVRs
to prevent interference with normal SPI traffic.

Etc…

A few days ago Ian, the author of BusPirate, announced the availability of STK500 AVR programmer firmware for the BusPirate hardware. I haven’t had a time to give it a try yet. I expect it may be somewhat faster comparing to avrdude -c buspirate. On the other hand it probably doesn’t support more than one concurrently connected chip — a feature for which I’m keen to sacrifice some programming speed.

The rumours go that there is one more BusPirate firmware on the way — a Microchip PIC programmer. I can’t wait to see that in action. And don’t forget the original BP firmware that can do a lot more than just programming microcontrollers. Sadly there doesn’t seem to be a dedicated, well structured BusPirate project website — you’re left to gather the information scattered throughout the BusPirate blog to find out all the details.

BusPirate is truly one of the must-have tools for anyone working with microcontrollers and related low-level communication protocols. It’s incredible how much functionality can be found in a $30 gadget.

Some HP ProLiant systems come with a stripped down version of ILO (Integrated Lights-Out) management card that, without an additional license, supports only text mode remote console. While this is indeed a big problem for Windows users who are therefore forced to purchase the Advanced ILO license, Linux can happily run in a text mode. With one exception though: installation disks of most current distributions boot up into a graphical mode first and only then allow selecting a text-only installation. That’s true at least for RedHat Enterprise, CentOS, OpenSUSE and Ubuntu.

Useless CentOS boot-time graphical splash screen.

Since the boot-time graphics is totally useless let’s build a custom installation CD with a text-only boot loader. In the following how-to we are going to rebuild the first CD of CentOS 5.3 in x86_64 version. However the general concept should be applicable to the installation CDs or DVDs of any current distribution as long as they use isolinux boot loader. Majority of them do.

Text-only CentOS

1. Download CentOS-5.3-x86_64-bin-1of7.iso from your favourite CentOS mirror.
2. Mount it as a loop-back device so that we can access its content:
   

   ~# mount -oloop,ro /data/iso/CentOS/CentOS-5.3-x86_64-bin-1of7.iso /mnt
   ~# cd /mnt
   

3. Copy isolinux subdirectory to some writable location, for instance to /tmp:
   

   ~# cp -a /mnt/isolinux /tmp
   ~# cd /tmp/isolinux
   /tmp/isolinux# ls -l
   total 8616
   -r--r--r-- 1 root root    2048 Mar 22 02:14 boot.cat
   -rw-r--r-- 1 root root     292 Mar 22 02:14 boot.msg
   -rw-r--r-- 1 root root     919 Mar 22 02:14 general.msg
   -rw-r--r-- 1 root root 6640927 Mar 22 02:13 initrd.img
   -r--r--r-- 1 root root   10648 Mar 22 02:13 isolinux.bin
   -r-xr-xr-x 1 root root     364 Mar 22 02:14 isolinux.cfg*
   -r--r--r-- 1 root root   94600 Mar 22 02:14 memtest
   -rw-r--r-- 1 root root     817 Mar 22 02:14 options.msg
   -rw-r--r-- 1 root root     517 Mar 22 02:14 param.msg
   -rw-r--r-- 1 root root     490 Mar 22 02:14 rescue.msg
   -rw-r--r-- 1 root root   63803 Mar 22 02:14 splash.lss
   -r--r--r-- 1 root root    2659 Mar 22 02:14 TRANS.TBL
   -rw-r--r-- 1 root root 1889308 Mar 22 02:13 vmlinuz
   

4. Remove boot.cat file—it will be re-created later. Also remove splash.lss as that’s the boot-loader image we don’t like.
   

   /tmp/isolinux# rm -f boot.cat splash.lss
   

5. Edit boot.msg file and remove the reference to splash.lss (it’s on the 2^nd line, remove the whole line). Eventually replace it with a custom message saying it’s a CentOS 5.3 installation. Also change the first hint message (you’ll see later why):
   

   ^L
   ^Xsplash.lss[[CentOS 5.3 x86-64]]
    —  To install or upgrade in graphical mode, boot from the hard drive press the ^O0b^O07 key.
    —  To install or upgrade in text mode, type: ^O0blinux text ^O07. [...]
   

6. Optionally edit isolinux.cfg and change the first line from default linux to default local
   

   default linuxlocal
   prompt 1
   timeout 600
   display boot.msg
   

   This change will make the CD boot from the harddrive by default. That means you can leave the CD in the drive and the server will come up properly even after reboot. Original CD starts the graphical installation by default which is IMHO a bad idea.
7. Now with all the changes done we need to put the modified files back into the CD tree and recreate the ISO image.
   • One way is to copy everything off /mnt and place the changes there. That's a needlessly space-consuming approach, especially with DVDs.
   • Better way is to bind-mount /tmp/isolinux back to /mnt/isolinux and effectively replace the old directory with our modified one:

   ~# mount --bind /tmp/isolinux /mnt/isolinux

8. Now compile the new text-only bootable ISO image using mkisofs:
   

   ~# mkisofs -R -J -T -v -V "CentOS 5.3 x86-64 text" \
           -no-emul-boot -boot-load-size 4 -boot-info-table \
           -b isolinux/isolinux.bin -c isolinux/boot.cat \
           -o /data/iso/CentOS/CentOS-5.3-x86_64-bin-1of7-text.iso /mnt
   mkisofs 2.01 (cpu-pc-linux-gnu)
   Scanning /mnt
   Scanning /mnt/CentOS
   Excluded: /mnt/CentOS/TRANS.TBL
   Scanning /mnt/images
   [...]
    97.59% done, estimate finish Thu Jul  2 13:03:10 2009
    99.16% done, estimate finish Thu Jul  2 13:03:10 2009
   Total translation table size: 197177
   Total rockridge attributes bytes: 83484
   Total directory bytes: 129024
   Path table size(bytes): 112
   Done with: The File(s)                             Block(s)    317394
   Writing:   Ending Padblock                         Start Block 317533
   Done with: Ending Padblock                         Block(s)    150
   Max brk space used c6000
   317683 extents written (620 MB)
   

9. That's it. Now burn the ISO onto a CD and enjoy a pure text-only CentOS installation!
   

New text-only CentOS boot loader

Here we go, this disk will work nicely even on a ProLiant with a crippled ILO :-)

Oh, by the way, you may wonder how did I make the purple colour for the [[ CentOS 5.3 x86-64 ]] title... The trick is: isolinux can interpret some special character sequences as colour-codes and I simply wrapped the title with these. Have a look here for a list of these codes or download IsoLinux Mate if you want a user-friendly tool for creating custom isolinux boot screens.

Today I was asked to perform some Linux server maintenance for an important client. They use Citrix Access Gateway™ (CAG) for remote access to their systems. The plan was to login to the Citrix Access Gateway web interface from Firefox, click the PuTTY icon, let CAG server execute the PuTTY SSH client and display its interface on my Linux desktop via a locally installed ICA client. Once it’s up enter the remote Linux server’s IP and do my job. Unfortunately when I clicked the PuTTY icon things went wrong instantly:

You have not chosen to trust ‘Thawte Server CA’, the issuer of the server’s security certificate.

Aha, now what? Apparently Citrix ICAclient comes with its own set of trusted Certification Authorities. Both Firefox and my system-wide CA list are set to trust Thawte but ICAclient didn’t. After a bit of research I have found the ICAclient’s trusted certificates are stored in ~/ICAClient/linuxx86/keystore/cacerts/ directory:

~/ICAClient/linuxx86/keystore/cacerts $ ls -l
-r--r--r-- 1 mludvig users 891 2009-06-07 12:00 BTCTRoot.crt
-r--r--r-- 1 mludvig users 774 2009-06-07 12:00 Class3PCA_G2_v2.crt
-r--r--r-- 1 mludvig users 774 2009-06-07 12:00 Class4PCA_G2_v2.crt
-r--r--r-- 1 mludvig users 606 2009-06-07 12:00 GTECTGlobalRoot.crt
-r--r--r-- 1 mludvig users 576 2009-06-07 12:00 Pcs3ss_v4.crt
-r--r--r-- 1 mludvig users 568 2009-06-07 12:00 SecureServer.crt

That’s a good start. Now find out in what format these certs are — two most common formats are DER (binary format) and PEM (ASCII encoded). Quick check reveals that these files are binary, therefore most likely in DER format. Verify the assumption with openssl:

.../cacerts $ openssl x509 -inform der -text -noout -in BTCTRoot.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 33554617 (0x20000b9)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
[...]

Very good — openssl was told to open it as DER and gave us a reasonable output, so it’s DER! Now we need to get the Thawte Server CA certificate from somewhere, convert it to DER format and save into this directory. It is almost certain that the Citrix Access Gateway web interface uses the very same SSL certificate that the ICA client complains about. So … grab it from there!

Right click somewhere on the page and select View Page Info — a Page info dialog should pop up. Select the last tab — Security — and then View Certificate

Page info — View Certificate

Certificate Viewer will pop up. Select the second tab — Details. There, in the Certificate Hierarchy tree select to top-most item, Thawte Server CA in our case. Click the Export button at the bottom and save the certificate for example as ~/ThawteServerCA.pem.

Export “Thawte Server CA” certificate

Now the last step on our quest — convert the certificate from PEM format to DER with the help of openssl and verify that it worked out. Note that we’re still in the ICAclient’s keystore/cacerts directory:

.../cacerts $ openssl x509 -inform pem -outform der -in ~/ThawteServerCA.pem -out ThawteServerCA.crt
.../cacerts $ openssl x509 -inform der -in ThawteServerCA.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: md5WithRSAEncryption
        Issuer: C=ZA, ST=Western Cape, L=Cape Town, O=Thawte Consulting cc, 
                   OU=Certification Services Division, 
                   CN=Thawte Server CA
        Subject: C=ZA, ST=Western Cape, L=Cape Town, O=Thawte Consulting cc, 
                   OU=Certification Services Division, 
                   CN=Thawte Server CA
[...]

All right, we’re set to go. Click on the PuTTY icon again and see how far we get.

PuTTY is now starting

Voilà, things look good. A while later PuTTY is up, running on the remote Citrix Access Gateway server, ready to open a SSH connection to the Linux box in the company’s internal network.

Earlier today I worked on migrating a D³ database server to a VMware ESX environment. The tool that we used for migration did a good job in converting the RHEL3 operating system and all the Linux filesystems, but failed to copy the D3’s raw data partition:

old-server:~# fdisk -l /dev/sda
[...]
/dev/sda12        10965     17816  55038658+  d3  Unknown

Now what? I didn’t have enough space neither on the source server nor on the destination VM to dump the partition contents to a file, copy across and reload back to /dev/sda12 on the VM. It had to be done online.

Gladly, SSH has the ability to run a command remotely and feed its standard output to some other program running locally. Using that feature it’s easy to copy the raw partition — simply run dd if=/dev/sda12 on the source server and dd of=/dev/sda12 on the destination VM. The first dd without any other parameters will print whatever it reads from /dev/sda12 on the old, source server to its standard output. The second dd, inversely, will write whatever it reads from the standard input down to /dev/sda12 on the new virtual machine. Glue it together with this ssh command:

new-server:~# ssh old-server "dd if=/dev/sda12" | dd of=/dev/sda12

That’s all sweet, but … dd doesn’t provide any progress tracking. In my case I had to transfer over 50GB of data and had only a vague idea how fast it goes. Should I wait? Or leave it overnight? Hmm, hard to tell.

Finally I came up with a simple solution for checking progress of dd — get a sample data, say 1kB, from a given offset on both the source and destination partition and compare their checksums:

old-server:~# dd if=/dev/sda12 bs=1k count=1 skip=5M | md5sum
7b10e9e1029c4c0f3901ee13db18a927  —

new-server:~# dd if=/dev/sda12 bs=1k count=1 skip=5M | md5sum
0f343b0931126a20f133d67c2b018a3b  —

OK, the checksums didn’t match. Yet. I kept re-running the command on the new server and as soon as it returned the same checksum as on the old one I knew it just copied 20GBs.

A side note here: I used skip=5MB and claim it copied 5GB — why’s that? Because skip= skips a given amount of ibs-sized records. In our example ibs=bs=1kB and therefore skip=5M skips 5 millions records, which means it skips to an offset 5GB in /dev/sda12. And reads one kilobyte from there.

Re-running the hashing command every minute or so manually is boring. Instead I wrote a little shell script to record the timestamp when the hashes at a given offset match:

#!/bin/sh
## check-progress.sh from http://hintshop.ludvig.co.nz/show/copy-raw-partition-over-net/
DEVICE=$1
OFFSET=$2
REQ_HASH=$3

if [ "${REQ_HASH}" = "" ]; then
   echo "Usage: $0 {device} {offset} {required-hash}"
   exit 1
fi

while (true) do
   HASH=$(dd if=${DEVICE} count=1 bs=1k skip=${OFFSET} 2>/dev/null | md5sum)
   HASH=${HASH:0:32}
   if [ ${HASH} = ${REQ_HASH} ]; then
      echo "Hashes match: $(date)"
      exit 0
   fi
   echo "Not yet..."
   sleep 30
done

To use it I needed a hash value from a given offset on the old-server and then run the script on the new-server:

new-server:~# ./check-progress.sh /dev/sda12 5M 7b10e9e1029c4c0f3901ee13db18a927
Not yet...
Not yet...
...
Hashes match: Sat Feb 14 11:22:33 NZDT 2009

Once it recorded the timestamp I set it up again with a new offset, say 6M, and waited. That way I was able to track the progress and approximately measure the transfer speed. Precise enough to realise it did about 8GB per 15 minutes. Then I knew I’ve had enough time to go and get some lunch.