Debian's "Push Mirroring" Technique

Overview

Push mirroring is a form of mirroring using rsync that minimizes the time it takes for changes to the main archive to reach mirrors. The server mirror uses a triggering mechanism to inform the client mirror it needs to be updated.

Push mirroring takes a little more effort to set up since the maintainers of the upstream and downstream mirror must exchange information. The benefit is that the upstream mirror initiates the mirror process immediately after its archive has been updated. This allows changes to the archive to propagate extremely quickly.

Explanation

First some background on ssh. Ssh allows people to connect to accounts on different machines in a secure way. Not only are passwords never passed in the clear, once you connect to a machine you are basically guaranteed that future connections will be to the same machine. This prevents many man-in-the-middle attacks.

One capability ssh has is the ability for a user to take the public identity key for a user on another machine and add it to a file of authorized keys on your machine. By default, the user on the other machine (who has the private identity key associated with the public identity key given to you) then has login privileges to your account. It is possible, though, to add text to an authorized key restricting the type of access a person accessing your account using that key has.

So to protect the downstream mirror, the key provided by the upstream mirror has text added to it to limit it to only give the person accessing your account permission to do one thing - start the program on your machine that updates your mirror. Even if someone (an evil third party) was able to break the key, the most they could do is to start the mirror program on your machine. You do not even have to worry about multiple copies of the program being started as a lockfile is used.

On the upstream end, rsync can be configured to restrict who can mirror a given area by username and password. These are totally separate from /etc/passwd so a push server doesn't have to worry about giving others access to their machine. As it is set up, the username and password are passed in the clear. This shouldn't be a problem though, as the worst that can happen is that a third party gains the ability to mirror the Debian pages from that site.

Setting up a push server consists of two basic tasks: setting up rsync access (for normal, "pull" mirroring) and setting up ssh trigger mechanism (for "pushing" the pull mirroring).

Configure rsync

Install rsync 2.1.1 or greater.

Create rsyncd.conf file and put something similar to this in it:

 uid = nobody
 gid = nogroup
 max connections = 25
 syslog facility = daemon
 socket options = SO_KEEPALIVE

 [debian]
   path = /org/ftp.debian.org/ftp
   comment = Debian FTP Archive (~24 GB)
   auth users = authorized_account1,authorized_account2,authorized_accountN
   read only = true
   secrets file = /etc/rsyncd/debian.secrets

 [debian-web]
   path = /org/www.debian.org/debian.org
   comment = Debian Web Site (~400 MB)
   auth users = authorized_account1,authorized_account2,authorized_accountN
   read only = true
   secrets file = /etc/rsyncd/debian.secrets

Add an entry for each site you are pushing to in the /etc/rsyncd/debian.secrets file:

 authorized_account1:a_password
 authorized_account2:another_password
 authorized_accountN:password

You have now given the downstream mirrors access to the archive on your machine.

You will probably want to start the rsync daemon from inetd. To do this, you have to add rsync service in /etc/services file (if it isn't already there), like this:

 rsync           873/tcp

To enable the daemon from inetd, add the following to your /etc/inetd.conf file:

 rsync      stream      tcp         nowait      root /usr/bin/rsync rsyncd --daemon

(Remember to send inetd a HUP signal to tell it to reread its config file after modifying the file.)

SSH trigger mechanism

Create a new ssh key for the account that you use to mirror Debian. Make sure you don't overwrite your original ssh key by using the -f option, for example: [=ssh-keygen -f ~/.ssh/identity.mysite]

Make sure that the new public key ([=~/.ssh/identity.mysite.pub]) contains this at the beginning:

 no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="~/websync &"

(replace "websync" with "ftpsync", or "ftpsync-non-US", or whatever's the command to start the mirroring called)

You need to set up a script that will contact the downstream mirrors. Create a file called signal, containing this:

 #!/bin/sh

 # This script is called to signal the remote host that it is time to
 # mirror the archive.
 
 echo Signalling $1
 ssh -o"BatchMode yes" -o"user $2" "$1" -i $HOME/.ssh/identity.mysite sleep 1

This script will login to a remote host using the special ssh key you created above. The script itself will not do anything useful remotely, the ~/websync (or ~/ftpsync, or ~/ftpsync-non-US) command will be run from the key.

To actually signal the mirrors, you need to add ./signal <site> <username> lines at the end of either the websync script, or if it's more convenient for you, in a new script, and then run that script from websync.

This new script, runmirrors, would contain something like this:

 #!/bin/sh
 
 # This script is called by websync to signal the downstream mirrors.

 ./signal some.other.site archvsync
 ./signal and.another.site othersiteaccount

Thus, as soon as your site is finished mirroring from the upstream site, you will start pushing to those downstream from you.

---

Source: http://www.debian.org/mirror/push_server