This commit is contained in:
pierreozoux 2015-01-05 16:45:04 +01:00
parent 7b39ee0ca9
commit a450375ddc
30 changed files with 1 additions and 293 deletions

View file

@ -1,10 +1,7 @@
## IndieHosters
This repository contains the configuration and scripts we use to control our servers.
It can run inside Vagrant or
[deploy to a server](doc/getting-started-as-a-hoster.md) (FIXME: update those instructions to
prescribe less folder structure, explain static https+smtp hosting, and check if they currently
work).
It can run inside Vagrant or [deploy to a server](doc/getting-started-as-a-hoster.md)
## Prerequisites to running this code with Vagrant:
- [vagrant](http://www.vagrantup.com/)

View file

@ -1,83 +0,0 @@
-----BEGIN CERTIFICATE-----
MIIFljCCA34CCQDXgLjASWHpmDANBgkqhkiG9w0BAQUFADCBjDELMAkGA1UEBhMC
UFQxETAPBgNVBAgTCFBvcnR1Z2FsMQ8wDQYDVQQHEwZMaXNib24xFTATBgNVBAoT
DEluZGllSG9zdGVyczEZMBcGA1UEAxMQaW5kaWVob3N0ZXJzLmRldjEnMCUGCSqG
SIb3DQEJARYYY29udGFjdEBpbmRpZWhvc3RlcnMubmV0MB4XDTE0MTAxMDE0MzY1
NVoXDTE1MTAxMDE0MzY1NVowgYwxCzAJBgNVBAYTAlBUMREwDwYDVQQIEwhQb3J0
dWdhbDEPMA0GA1UEBxMGTGlzYm9uMRUwEwYDVQQKEwxJbmRpZUhvc3RlcnMxGTAX
BgNVBAMTEGluZGllaG9zdGVycy5kZXYxJzAlBgkqhkiG9w0BCQEWGGNvbnRhY3RA
aW5kaWVob3N0ZXJzLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIB
AKBOylYEoL1P3q7skTJsRA8yQj6fVHWHS3kPg6tcVavZawc6tRxIiDc41/EWjL7i
Owb6io2UbKaD/g8695CFER9FvcW1iukrC/tUV5/AVd0SDcvS3RnGUndKh82HCNrM
rUDU/XH8smEpfjuXrq0YPuiGbY1zSLQKirjYTiasJODfGkxSbobNfjdL7aEo+3HX
BQq5mGIj9A4PYmeyFGkHCN8tRvf4lY1KfPJoWtDL4kmO4SFNZ4FAehH9AJ6vTN8y
MFcHtFzpp2636TYTBQsLu48nrKs6MqOOyU0R/Ufw9QjiWDLo3Co6pcCTmVf16skO
odg9BNdEhMXefpiEE1NOL6ZOkSUG5WSY0Q5Il649QcJOYzw2A0Nk3IOxoIexXat4
siCgSlNfgyRmBn5HNcZo5aEDf9+3gEqFzEFSyH3ClIApC7RePbpPvsCAgpagBOXC
PgO2w2VW9HfNHkwpF3Yqn7cqw0FQKwKREufVdnSvs9fgFlMZnqA3sMym8o99Fcvq
WBaTuh54ePfNGmawPt1N8vUZUYXXOasWKmnjfan3S1rsNAf5M2ntLqEJRDwihdSm
ZSO+B51hDO5jzHoqxHwA71CwUAp4hRO83xR6ziB1KR2834I/7LBzbpZ0EWm9adez
8V+dwgBhTt0LYEUGLJN22XRi9d4RPhnRJpSLPV/h0Fa/AgMBAAEwDQYJKoZIhvcN
AQEFBQADggIBAFzYeGiomhKZW//aUM4V4RLMVIf0B4uixSMxZGQIUWVtYckmyG2N
t8qNBHAQ3gl811NqnqestIQ4DpGkNQRCv/iDa5OwdLJHTOQUxajUE/1xmidHtpzR
ReBZvW48k0dLEM2gmIrt7qQwqqecjlWjvSQlvJxYWrn6TBAkFL6Quu8gfoPK9/cE
HG/aRQ0PCywGV20LSZ+J03LN7MlACClgVTB7dJuWIN0dNi7TsqpIupk11ZQ3ybBY
WPQmLnIiCAijL69kBmBynLvJT5XDy2C4ChyzZ5Y73CXhgJwCqOZJwbO7Doig9PZQ
yVLtui18W3uVQ7ZlIxCAQUeFzSkZf3/XNlr2FkP+efw4LLGH8kiKMsyKuoLuthO1
1YrXvI0sjuDOxQwrlNQ2CLVANLBpUMH2U1aiYbA6iICSHr8ORAc84StgG9mFLeyN
w32/04MGPvZfset8gRCOuvA2sLTjylqh0IpaPWlnT77neqOFtETtzJ+3UuOcdfnN
t2bxqimHT8WhBB823WajWlLdXcc902e9LLhe9M1/bwOqFIIlKDqtCndjyXpe/qhA
s0YB8TqJLxJQqvdnmYiBFfGrDTgNBpjt6AKJHRGd4xgsYsmQ3zLJ0Z8mNNQhlLf/
osGXa2s/ZX7ernfvSDQIOB70gohCLFtBok0unyBJhtHxXmZ7UmpuIanx
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIJKAIBAAKCAgEAoE7KVgSgvU/eruyRMmxEDzJCPp9UdYdLeQ+Dq1xVq9lrBzq1
HEiINzjX8RaMvuI7BvqKjZRspoP+Dzr3kIURH0W9xbWK6SsL+1RXn8BV3RINy9Ld
GcZSd0qHzYcI2sytQNT9cfyyYSl+O5eurRg+6IZtjXNItAqKuNhOJqwk4N8aTFJu
hs1+N0vtoSj7cdcFCrmYYiP0Dg9iZ7IUaQcI3y1G9/iVjUp88mha0MviSY7hIU1n
gUB6Ef0Anq9M3zIwVwe0XOmnbrfpNhMFCwu7jyesqzoyo47JTRH9R/D1COJYMujc
KjqlwJOZV/XqyQ6h2D0E10SExd5+mIQTU04vpk6RJQblZJjRDkiXrj1Bwk5jPDYD
Q2Tcg7Ggh7Fdq3iyIKBKU1+DJGYGfkc1xmjloQN/37eASoXMQVLIfcKUgCkLtF49
uk++wICClqAE5cI+A7bDZVb0d80eTCkXdiqftyrDQVArApES59V2dK+z1+AWUxme
oDewzKbyj30Vy+pYFpO6Hnh4980aZrA+3U3y9RlRhdc5qxYqaeN9qfdLWuw0B/kz
ae0uoQlEPCKF1KZlI74HnWEM7mPMeirEfADvULBQCniFE7zfFHrOIHUpHbzfgj/s
sHNulnQRab1p17PxX53CAGFO3QtgRQYsk3bZdGL13hE+GdEmlIs9X+HQVr8CAwEA
AQKCAgEAgDpF8sRE5ukqUHV+Nv0O+7DR+FFuN4x/PFjCk6GKDaodyGyXTgZenv1j
Db9h2ZYQbSafCVy+A/v0jq42NG2cIo2gnLL4aEY8kU8HwAsTI4A7dNw4a1ONx0ng
ku/+jzXFJ+S2ziS5cqrEBFryKBcKyugsXUbn0svT5sNuz9RGs3ECEialrkJVQVoE
vDKR3p+Fsux+DZKAt3Zq2lNBrDkqSYpoCBXZWmlIxIXgjr9nRDt7rS3DK0ot2pGr
m0LRlH8K17Kb/O4RNaj6bHyOPiWmY33yygwFUXr3XiSTmqYM+oxCzIYjBcxfpUjr
EcbthOGlZ9h3NNHj+npcfRa4dpxF09c8gW2AVG+nXVhciZpcnLDZ5z/Nd/510axU
0m0PlCPfh+3L5tiia9k7zlRxjyzER/GofNiJ6v8oo8YZFvhVdbBBQoGs8aadSLH9
5Kf3fPwm8ZhmmOTVWbFJZul/3o0Ho3yFxMVMq86Qu8Pm+h6Q1Pn7yZsXMg/ECXP/
/ErBaWA+zuBZkgCSbdZk58cxkN45PGWGkoHHACVUvCbG8IuYQ989JeCy5w01FgFV
IXm4squNtWgyhLZgvkhl2Hnc4pR+iYJRgh+ouyv7nELQde7hpM6YJLLUpMfjo7r5
lJyWasZtb9E4iEl4/JrdQYMJCDEyBfDN6sTKr1Ai2txjzQA4uOECggEBAM9LDpJ+
RR+b1rdYgtS6VL5OR1bWUHSi1W9L8Xz20wSQGbRxfEJfWmSslOU0COXvA01eOxQ9
OvHcWxISiHdiM3QxpYNtbsgATCQbsSgegMHpbaEgJPadEkUWxdWejbtpA1ypKmGg
iFB5H5IIcz65wWNFC3g29wrXyBsRevi+K/PTbwOzOlad7AAcbuuHiv73wxi5xo1P
i6IZfjgQMKzD9AJbACAAqyvg70XT+3vlIo5ABKOw1kLuejbNBaXd1af7OfVXReL7
BGGJmG6IzI0qP9q7fX3Iq4Gx34Sf0TSomSyW4kxtsDMPXVURMU4ssxeshh0zYFsZ
GQgsr36mOW5cvbkCggEBAMX5gJTrAW47GgObnQWtYIHRvYO0g7Ge1fN12VzHLiap
3a3RfhEDTVKkiugO1GxRC1NY0tcDUwrUzS/00ovDZ/8dVqMHITFj6zfA8aX6vnzA
TnoUWINawPxFBB6FrEuXyGIVbykinuvFyk+z/DzgKzL8X5MaLymYSV+eT+9jjLHO
pJ37S86evkljq24Ow6KB1rKb8mMsk8GDZB4JalDdGWzlG1qJkHMg7ULkEHx2lDTW
mcuHwRtMimFPCBGqH0i+p3O1IUkodJPNYbldrEfAkzRdD4lH9B+DNYBgxP4FWhY2
d9DTHAGCa9ZV0HjnGgPOILRmV69+9yQhNhu5010qNDcCggEABq1VP9S/Z0A+z1MT
i8SgvCyLUbm/h7JDC723fp34uBnoKg7JwN2PbNS+Sw+9BaMISTKy1nkOcAH4EQH1
0Vqha6m5uh0JR3ny+erGbxNkdFqPhHQjnKn8j6snHjVoPVQpno94ZQKlwWnVYX/S
LoAPQaJUtz+V/4xpzq1md6Kwib8SwVzBkU6u7mX8EKwiBwp2B1LcmWqphcQqc6XZ
24bIUlcaDu3Wlag+LNKiNCByV4CqZZdpn2hNGXzLJMebfTizajqwbppFTtr+xPi1
Fgr5WZNWfHm9RIU1PPFk7LxNisklau7RkSN6jyXpn6oC7s1I2KHyBZ0uWDwQPxUd
nndwSQKCAQA/gmrdWwZ6djtCLQmSaKws+TvypFYbBPldwNCaEsubW6Lhv/LRQl3r
xR1KlHdQyC757eS1VTuundW1LLTeYTFbhe3lHsRnM8ahfCQJOwcgvhBu2VgLy3Fd
fEZ2BCvhlC+UR4wBhjm1KR5dsz+Xx9IT6SI/7oZysYfYRNEf2q+n2sK0a4lGH2ar
5G16QQJBf6WAZsa7SfGcgqn7eMnCZytg456CzN6qEEYMz1z6kI+6450yzboFJ+i8
jr3n7Mtcas0NMW4cKf477AcNkB9UZVLT2YbCY3LNKSpgpKqNUuozdgW51/+D/HLb
r2vRXVHbJqUXOj2m7vQZgw34lwRXPtLBAoIBAChJgVltpcWKUWqltYXCQsdPPbb4
DQMb4bb2vV2iON2kl+UlcCdhr0f5yWoAyKjs49lcHBN2Ny4zVR0vIu/IDeX47Fx7
n0OfcFgcnqiqiFhXkWGcfU2JHq/q5tmk5M04aCgkFM8IyEsG6ZLoi849Km9r8quu
VfclpJ6SsMGnWo/A2eIVP9GsfqRys9ZWKJ9inZRP5Lmx6pCZa12Mn6ey0h/kxOqh
ruJQDdV0O4PsvZhTQFhahSVyNmSKnLguq3zsyBwKRsNI9TVXMv/hs0nnwfFgtBK1
K61c7AL4+9dtAWEnuwqy/1srZEeBr/jgTqyFyr+GQFYUMuE/uXNKCDWlIRI=
-----END RSA PRIVATE KEY-----

View file

@ -1,6 +0,0 @@
/etc/haproxy/approved-certs/michielbdejong.com.pem michielbdejong.com
/etc/haproxy/approved-certs/michielbdejong.com.pem www.michielbdejong.com
/etc/haproxy/approved-certs/otherdomain.com.pem otherdomain.com
/etc/haproxy/approved-certs/otherdomain.com.pem www.otherdomain.com

View file

@ -1,44 +0,0 @@
global
maxconn 4096
user haproxy
group haproxy
defaults
mode http
option forwardfor
option httpclose
option httplog
option dontlognull
retries 3
timeout connect 5000
timeout client 50000
timeout server 50000
frontend https-in
mode http
bind *:443 ssl crt-list /etc/haproxy/crt-list crt /etc/haproxy/approved-certs/default.pem
reqadd X-Forwarded-Proto:\ https
# otherdomain.com:
acl https_otherdomain.com hdr(host) -i otherdomain.com
acl https_otherdomain.com hdr(host) -i www.otherdomain.com
use_backend otherdomain.com if https_otherdomain.com
# michielbdejong.com:
acl https_michielbdejong.com hdr(host) -i michielbdejong.com
acl https_michielbdejong.com hdr(host) -i www.michielbdejong.com
use_backend michielbdejong.com if https_michielbdejong.com
frontend http-in
bind *:80
redirect scheme https code 301
# michielbdejong.com:
backend michielbdejong.com
cookie SERVERID insert nocache indirect
server Server michielbdejong.com-backend:80 cookie Server
# otherdomain.com:
backend otherdomain.com
cookie SERVERID insert nocache indirect
server Server otherdomain.com-backend:80 cookie Server

View file

@ -1,5 +0,0 @@
docker rm haproxy
docker run -d --restart=always --name haproxy -v $(pwd):/etc/haproxy -p 80:80 -p 443:443 \
--link michielbdejong.com:michielbdejong.com-backend \
--link otherdomain.com:otherdomain.com-backend \
indiehosters/haproxy

View file

@ -1 +0,0 @@
k2.michiel.indiehosters.net, indiehosters.net, 3pp.io

View file

@ -1,2 +0,0 @@
/.*@somedomain.com/ person1@gmail.com, person2@hotmail.com
/michiel@somewhere-else.net/ me@forward.net

View file

@ -1 +0,0 @@
k2.michiel.indiehosters.net

View file

@ -1,2 +0,0 @@
docker rm postfix
docker run -d --name postfix -v $(pwd):/data -p 25:25 indiehosters/postfix-forwarder

View file

@ -1,20 +0,0 @@
#!/bin/sh
if [ $# -ge 3 ]; then
SERVER=$1
DOMAIN=$2
PEMFILE=$3
else
echo "Usage: sh ./deploy/add-placeholder-site.sh server domain pemfile [user]"
exit 1
fi
if [ $# -ge 4 ]; then
USER=$4
else
USER="core"
fi
echo "Adding $DOMAIN to $SERVER with cert from $PEMFILE"
echo "Remote user is $USER"
ssh $USER@$SERVER sudo mkdir -p /data/domains/$DOMAIN/static
scp $PEMFILE $USER@$SERVER:/data/runtime/haproxy/approved-certs/$DOMAIN.pem
ssh $USER@$SERVER sudo sh /data/indiehosters/scripts/activate-user.sh $DOMAIN static

View file

@ -1,22 +0,0 @@
#!/bin/sh
if [ $# -ge 5 ]; then
SERVER=$1
DOMAIN=$2
PEMFILE=$3
IMAGE=$4
GITREPO=$5
else
echo "Usage: sh ./deploy/add-site.sh server domain pemfile image gitrepo [user]"
exit 1
fi
if [ $# -ge 6 ]; then
USER=$6
else
USER="core"
fi
echo "Adding $DOMAIN to $SERVER, running $IMAGE behind $PEMFILE and pulling from $GITREPO"
echo "Remote user is $USER"
ssh $USER@$SERVER sudo mkdir -p /data/domains/$DOMAIN/$IMAGE/data
scp $PEMFILE $USER@$SERVER:/data/runtime/haproxy/approved-certs/$DOMAIN.pem
ssh $USER@$SERVER sudo sh /data/indiehosters/scripts/activate-user.sh $DOMAIN $IMAGE $GITREPO

View file

@ -1,6 +0,0 @@
#!/bin/sh
for i in `deploy/list-sites.sh $1`; do
echo "Approving combined cert for $i";
cp ../orchestration/TLS/combined/$i.pem ../orchestration/TLS/approved-certs/$i.pem;
scp ../orchestration/TLS/approved-certs/$i.pem root@$1:/data/runtime/haproxy/approved-certs/
done

View file

@ -1,21 +0,0 @@
#!/bin/sh
if [ $# -eq 2 ]; then
CA=$2
else
CA="startssl"
fi
echo "CA is $CA"
echo Some information about cert ../orchestration/TLS/cert/$1.cert:
openssl x509 -text -in ../orchestration/TLS/cert/$1.cert | head -50 | grep -v ^\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
#echo Some information about chain cert ../orchestration/TLS/chain/$2.pem:
#openssl x509 -text -in ../orchestration/TLS/chain/$2.pem
#echo Some information about key ../orchestration/TLS/key/$1.key:
#openssl rsa -text -in ../orchestration/TLS/key/$1.key
cat ../orchestration/TLS/cert/$1.cert ../orchestration/TLS/chain/$CA.pem ../orchestration/TLS/key/$1.key > ../orchestration/TLS/combined/$1.pem
echo Running a test server on port 4433 on this server now \(please use your browser to check\):
openssl s_server -cert ../orchestration/TLS/combined/$1.pem -www

View file

@ -1 +0,0 @@
startssl

View file

@ -1,5 +0,0 @@
#!/bin/sh
cd ../orchestration/per-server/$1/sites
for i in *; do
echo $i
done

View file

@ -1,70 +0,0 @@
# Using just Docker and bash
## WARNING: Still a work-in-process
Given that CoreOS is not available everywhere, and the Ubuntu 14.10 setup with etcdctl inside a Docker instance still has some problems,
I'll try if I can run all our services (postfix-forwarder, haproxy, and the various web backend containers) on an off-the-shelf Ubuntu server.
Note that not all servers support Docker, because of kernel modules etcetera; of the images I tried at Gandi, only the Ubuntu 12.04-64 one
allowed me to actually run `docker ps`, and even on there, I was not able to run `docker run debian apt-get update` because from the looks
of it, containers are not allowed to contact the outside world. In the end I got a Debian 7.6 server at Linode.
Note that about 5GB of disk space will be used, and when running multiple sites, 2Gb of RAM is probably also well-spent.
Here's what I did to prepare the server:
````bash
apt-get update && apt-get -y upgrade
apt-get -y install unattended-upgrades git
dpkg-reconfigure -plow unattended-upgrades
# set unattended upgrades to 'Yes'
ssh-keygen -t rsa
# select all the defaults by hitting <enter> repeatedly
# Install Docker:
curl -sSL https://get.docker.com/ | sh
# Install docker-enter ([recommended](https://github.com/jpetazzo/nsenter#nsenter-in-a-can)):
docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter
# In this case I also had to:
/etc/init.d/exim stop
````
Test your server by running `docker run debian apt-get update` (there should be no 'could not resolve' errors).
Then I added the .ssh/id_rsa.pub to .ssh/authorized_keys at both backup server accounts, and ran:
````
git clone git@bu25:postfix
cd postfix; ./runme.sh; cd ..
git clone git@bu25:haproxy
cd haproxy; ./runme.sh; cd ..
````
These 'postfix' and 'haproxy' repos with their `runme.sh` files look like the samples in the `data/` folder of this repo.
And then for each domain I host:
````
git clone git@bu25:michielbdejong.com
cd michielbdejong.com; ./runme.sh; cd ..
````
These domain repos look quite similar to the IndieHosters migration format, except that there is a `runme.sh` file in there,
as follow:
````
docker rm michielbdejong.com
docker run -d --restart=always --name michielbdejong.com -v $(pwd):/data indiehosters/lamp-git
docker inspect -f {{.NetworkSettings.IPAddress}} michielbdejong.com
````
And furthermore, the dump.sql file includes the 'mysql' database as well as the 'known' database, so that the local 'root' user
is created, and the php files of Known 0.6.5 are included (will iron out that difference when switching from generic lamp-git
image to specific Known image).
Now the only thing you need to be careful with is to start haproxy *after* starting the backends, otherwise the container linking
doesn't work. This means reboots are not actually safe, and also, at the time of writing, hourly backups are not working yet in the
lamp-git image, so be aware that this is a work in progress. :)
The idea of this setup would be to eventually support the migration format in a programmatic way.