Skip to content
Snippets Groups Projects
Commit a57dc627 authored by sabrinezerrad's avatar sabrinezerrad
Browse files

modif README

parent ca2e7a60
No related branches found
No related tags found
No related merge requests found
# ansible-polytech-2024
# TP ASR - Ansible
- Ce TP a pour but d'automatiser l'ensemble des configurations d'une machine en utilisant **Ansible** , en minimisant au maximum les interactions "human to computer".
This is the repo that will serve as the support for the ASR practical work session of january. You should probably work in this directory and add code to it, it is going to be easier than creating a new ansible repository from scratch. You are also welcome to commit to this repository to checkpoint your work, as well as push it into any repository that polytech gives you access to.
- Ce fichier README va servir comme un résumé ou un "summary" qui regroupe les grandes étapes qu'on a pu achevé pendant ce TP.
The goal here is to deploy a Synapse Matrix server, an opensource decentralised messaging app, using industry standard tools such as ansible and docker.
- L'ensemble des commandes et outils utilisés sont présents dans le repository ci-dessus.
By the end of this lab you'll be able to message each other in an end to end encrypted fashion using your own servers!
## I. Pré-configuration
### 1. Virtualenv
Il s'agit d'un environnement virtuel de python, séparé du système de fichiers de la 'zabeth' pour faire les choses proprement.
## How it's going to work
* Log into https://gitea.plil.fr with your school credentials
* Then _fork_ [this repository](https://gitea.plil.fr/tmaurice/polytech-ansible) (you need to do that only once per group)
* Modify the repo's code as you see fit and remember to *commit and push often*. This is how Xavier and myself are going to assess progress and be able to grade your work.
### 2. Ansible
Ansible est un outil de gestion de configuration et d'automatisation qui permet de déployer, configurer et orchestrer des infrastructures informatiques de manière efficace et reproductible.
Optionally I would recomment you to add this repository as a remote so you can pull fixes should I have to push emergency fixes to the instructions (it happens more or less every year):
### 3. Vérification de la configuration d'un playbook
- `ansible-playbook` utilisée pour exécuter des scripts YAML, appelés playbooks, afin de définir et d'automatiser des tâches de gestion de configuration sur des machines distantes.
A chaque configuration d'un playbook, on a toujours intérêt à la vérifier avec `ansible-playbook`.
```
$ git remote add upstream https://git.plil.fr/tmaurice/polytech-ansible
# If you want to get the latest code merged with your branch
$ git pull upstream master
```
## II. Configuration du serveur DATABASE
- Il s'agit d'une configuration d'un serveur de base de données avec Ansible incluant l'automatisation de l'installation et de la configuration de `PostgreSQL`.
:warning: Please use explicit and descriptive commit messages when committing to git.
- Cela nous a permit de garantir une configuration cohérente et reproductible, ainsi que la gestion des utilisateurs, des autorisations, des sauvegardes et de la sécurité, le tout dans un environnement scalable et facilement maintenable.
### Optional setup: a gitea runner for continous integration
## III. Configuration du serveur web/Synapse
- Automatiser l'installation et la configuration de Nginx pour servir notre page web statique.
You can if you so wish deploy a [gitea runner](https://gitea.com/gitea/act_runner) somewhere, preferably on your zabeth. I would recommend you set the runner up for your user, and enable the "actions" in the settings of your newly created repo.
- Assurer la sécurisation du serveur en configurant des certificats avec l'utilitaire `certbot` pour sécuriter les communications HTTPS.
### 1. Synapse
Le logiciel synapse avait pour but dans ce TP de :
- Fournir une plateforme de messagerie instantanée décentralisée et sécurisée (riot).
This way every time you push your code, the checks in `.gitea/workflows` will be ran and you will have a live preview of the state of your code (how is the linting doing, what is the code quality, etc etc).
- Permettre la communication en temps réel entre les utilisateurs via des salons de discussion, des messages directs... (Dernière étape du TP)
If the consensus is "fuck that ain't nobody got time for that" I can also deploy one from my own server somewhere and you'll be able to use the runner for the entire length of the lab.
- Assurer la synchronisation des données entre les différents serveurs Matrix pour une expérience utilisateur cohérente.
:warning: you can run your gitea actions locally by running `make action` after installing the [`act` binary](https://github.com/nektos/act)
### 2. Nginx reverse proxy
- Le reverse proxy avait pour but d'utiliser la certificat TLS (générée avec certbot) qui écoute sur le port 443 et sur notre servername `synapse.nyala.website` (voir dans le fichier `playbooks/matrix.conf.j2`).
The default provided CI pipelines only check for YAML and Ansible correctness, but you could imagine workflows where after these tests are run and are successful, we trigger an ansible run directly from the CI and the runner itself. Feel free to try :)
- Pour le rôle du reverse proxy en sécurité, on cite :
**La gestion de la sécurité** : Il agit comme une barrière entre les clients et les serveurs back-end, offrant une couche de sécurité supplémentaire en masquant les détails de l'infrastructure interne et en fournissant des fonctionnalités de protection contre les attaques DDoS, les injections SQL, etc.
## Setup
:warning: This setup has to be done on your Zabeth host.
You will need to install a few things to get started, buckle up.
----
Ce projet a été développé par **EL HASNAOUI Bilal** & **CHAOUNI Ayoub**, sous l'encadrement de **M. MAURICE Thomas** et **M. REDON Xavier**.
Firstly you need to export some environment variables in your working terminal to be able to talk to the outside world using the school's proxy:
```
export https_proxy=http://proxy.polytech-lille.fr:3128
```
Also, depending on the operating system installed on your virtual machines (Ubuntu server should not have the issue, Debian might), you should install python 3:
```
# apt install python3
```
### The virtualenv
Since Ansible is written in python and we don't want to install it in the system, you will need to create a virtual environment. These are used to have your python stuff installed, without making them available system-wide, we are doing this to avoid polluting your lab machine with things that won't be used after today.
To create the virtualenv you need to run the following:
```
$ python3 -m venv ~/.ansible-venv
# then you want to "activate" the venv, you will need to do this for every new terminal you open
$ . ~/.ansible-venv/bin/activate
```
:warning: you need to run `~/.ansible-venv/bin/activate` every time you want to open a new terminal and use ansible in it, otherwise it just won't work because the ansible binary won't be found.
### Install ansible
Install ansible via pip _after entering the virtualenv_
```
$ pip install -r requirements.txt
```
At this point you should have ansible installed.
### Install the docker role
Install the docker role using ansible galaxy (ansible galaxy is a sort of package manager for ansible).
```
$ ansible-galaxy install -r requirements.yml
```
At this point you should be good to go!
### Generate an SSH key if you don't have one already
You'll need an SSH key if you don't have one already
If `ssh-keygen` complains about the key already existing, just reuse the existing key in case someone else needs it.
```
$ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -P ''
```
:warning: In real life, don't use `-P ''` because it creates your SSH key without a passphrase, it is ok for this lab, not for real life.
:warning: Never, ever, EVER commit an unencrypted SSH keys to a git repository. If you need to add the key to the repository for some reason I would recommend you look up something like [sops](https://github.com/getsops/sops) to store an encrypted version of the secret in the repository.
Lastly, look into `group_vars/all.yml`, go at the end of the file and add the created _public_ key in the root_user.default_root_keys (from `~/.ssh/id_ed25519.pub`, or any other keys you created before hand).
### One more thing, update your inventory
You can now update your `inventory` file by modifying it with your new values (hostnames and ip addresses for the machines you'll be working with).
The IP addresses you should put in there are the IPs that you have set in the VLAN50 (or the DNS names you have associated with it, if applicable)
### Read the roles to understand how everything works !
Ansible runs `playbooks`, which are collections of `roles` that in turn are a collection of `tasks`. `tasks` are instructions like "install this package", "copy this file", "create this directory", "install this service", "create this container" and so on and so forth.
I have very much documented the example roles in `./roles` and I would _greatly_ encourage you to read them to understand how to do basic stuff in ansible such as copying a file, starting a service and so on. If you do not do that, you will be lost and won't understand anything that is coming at you. The playbooks on the other hand are stored in the `./playbooks/` directory, feel free to look at them to inspire your own playbooks for this lab.
### Check everything works properly
You should now be able to actually `run` ansible to execute the `base.yml` playbook.
```
$ ansible-playbook -vi inventory -l all playbooks/base.yml
```
The `-i` flag specifies the inventory file to use, the `-l` file limits which hosts it applies to, either by hostname or group name, here we apply it to all the hosts.
If you cannot connect to the host with a `permission denied` error, rerun the command using the `-k` flag, to make ansible ask you for the host password.
```
# make sure the sshpass program is installed on the Zabeth
$ sudo apt install -y sshpass
$ ansible-playbook -vi inventory -k -l all playbooks/base.yml
```
If that does not work, add it to the `~/.ssh/authorized_keys` in the `root` home folder on every one of your virtual machines, make sure the `authorized_keys` file has `600` permissions. If the permissions are too wide then sshd will refuse to use it.
While you are at it I would recommend you install docker as well using the `docker.yml` playbook
Good, now you are good to go !
## Install a database server
To deploy Synapse and Mastodon, you need to deploy a database server. We are going to use Postgres in this lab. You will for this need to use the `community.postgres_db` module for this. The `community.*` modules are modules written by the community and available to everyone, you will encounter similar modules when you will want to start deploying docker container!
For more details to do this, I refer you to [the official ansible postgres documentation](https://docs.ansible.com/ansible/latest/collections/community/postgresql/index.html).
Note that you will need to install (with ansible !)
- `postgresql`
- `python3-psycopg2`
If you are feeling adventurous you can install postgresql using [docker](https://github.com/levonet/ansible-docker-postgres).
Either way, you need to install this on your DB machine, which should be the one with the least amount of RAM.
Go check the configuration of `/etc/postgresql/15/main/pg_hba.conf` and `/etc/postgresql/15/main/postgresql.conf` to make sure you can connect from `0.0.0.0`.
On the `pg_hba.conf` file add a line like that at the end:
```
host all all <the netmask you want to add> scram-sha-256
```
You can (and should) use the [postgresql_pg_hba](https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_pg_hba_module.html#ansible-collections-community-postgresql-postgresql-pg-hba-module) module for that. This module controles how clients can connect/authorise themselves to postgres.
The netmask should look like `A.B.C.D/E` like `0.0.0.0/0` for everyone, for example.
Also change the `listen_addresses` instruction to `'*'` in `postgresql.conf`.
Then create the [user](https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_user_module.html#ansible-collections-community-postgresql-postgresql-user-module) and the [database](https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_db_module.html#ansible-collections-community-postgresql-postgresql-db-module). Please note that you need to pass the following parameters to the database you are creating, otherwise Synapse will most likely complain about encoding issues and refuse to start:
```yaml
encoding: "UTF-8"
lc_collate: "C"
lc_ctype: "C"
```
Bonus point if you store the username in an encrypted fashion in your repository with either [ansible vault](https://docs.ansible.com/ansible/latest/vault_guide/vault_managing_passwords.html) or [sops](https://github.com/getsops/sops).
If you end up using sops, please also encrypt your secret with the following age public key so I am able to decrypt them later on `age18rkuwwpzl3az5gr093uhvk7cwg348eajxsm9fjansur5qa97csfs597zh6`, this can be achieved running `sops <file> --add-age <my key>`
## Install Nginx
You need to install nginx (or apache2, or traefik, whichever you are more comfortable with, but be aware I'll only be able to help with nginx and traefik as I have no idea how apache2 works). Use ansible for this, and install the `nginx-extras` package.
## Create a certificate for our deployments
While you are at it you might as well create a DNS `A` fields to point at OpenWRT (which will forward it to nginx).
We need to use a certificate to secure HTTPS communication, the Matrix protocols require it. This can be done manually for the moment and automated later, as it is not super straightforward. I refer you to the [documentation on certbot + nginx](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-20-04) I would recommend that you create a `matrix.<yourdomain>` certificate, then back them up somewhere safe as Letsencrypt has pretty aggressive rate limiting with regard to certificate creations.
## Let's install Synapse
Synapse is the messaging app I told you about.
### Preconfigure synapse
I would recommend you read quickly through this [synapse install guide](https://hub.docker.com/r/matrixdotorg/synapse/) so you understand what you are doing before touching anyting!
In essence you need to run the following command in your lab machine:
```
$ docker run -it --rm \
-v $(pwd)/synapse-data:/data \
-e SYNAPSE_SERVER_NAME=matrix.<yourdomain> \
-e SYNAPSE_REPORT_STATS=no \
matrixdotorg/synapse:latest generate
```
:warning: careful this is not the same command as in the documentation above, this will create the files you need in a `synapse-data` directory in your working directory, which is probably what you want in order to send them to your host via ansible.
Create a new role and copy over these files to `roles/synapse/files`.
Next customise the configuration file. If you feel like using postgres (you already set it up so why not, right ?) check the [docs](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#database).
Another approach that would be cleaner, but potentially harder, would be to tell ansible to run the configuration bootstrap command on the synapse machine if the `homeserver.yaml` file cannot be found, and then potentially customise the file.
## Create the synapse docker container
Normally your `synapse` module does not have any tasks yet, so go ahead and create a `roles/synapse/tasks/main.yml` file. You need to use the [`community.docker.docker_container`](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_container_module.html#examples) module for this. In essence you need to make sure of a few things:
- you want your container to port forward port 8008 in the host
to 8008 in the container either that or you can just use the
`network_mode: host` in the definition
- you want to mount the directory in which you copied the
`homeserver.yaml`, `matrix.<yourdomain>.log.config` and
`matrix.<yourdomain>.signing.key` to `/data` in the
container.
For example, if I wanted to start a `redis` container that forwards 6379 and mounts `/tmp` I would do something like
```yaml
---
- name: creates the redis container
community.docker.docker_container:
name: "redis"
image: "{{ redis_image }}:{{ redis_image_tag }}"
state: started
recreate: yes
ports:
- 6379:6379/tcp
volumes:
- /tmp:/tmp
restart_policy: "unless-stopped"
```
Note `restart_policy: "unless-stopped"` that ensures that your container is going to:
- Be started on boot
- Restart in case it crashes
:information_source: Note that I used `{{ redis_image }}` in the manifest above. These are variables that are available in the same fashion as they are for [templates](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template_module.html) which use the [jinja2](https://jinja.palletsprojects.com/en/3.1.x/templates/) markup language. If you want to create a default variable for your module, you can create a `roles/<rolename>/defaults/main.yml` that will contain something like this for example
```yaml
---
redis_image: redis
redis_image_tag: latest
```
You can override this variable if you want in any of the following:
- `vars` section of a playbook file
- `host_vars/<hostname>.yml` file for host specific overrides
- `group_vars/<group>.yml` file for group specific variables
Anyways back to synapse, create a playbook real quick just containing the synapse role
```yaml
---
- hosts: all
roles:
- synapse
```
Then apply it like we have seen above (`ansible-playbook -vi inventory -l <dockerhost> playbooks/synapse.yml`)
You should now have a running container by now, that has a directory mounted with your configuration. Check your container works with the `docker ps` command, you should see something like
```
root synapse.lil.maurice.fr ~ # docker ps | grep synapse
95aba77cc320 matrixdotorg/synapse:v1.74.0 "/start.py" 2 weeks ago Up 6 days (healthy) 8009/tcp, 0.0.0.0:8008->8008/tcp, 8448/tcp synapse
```
### Configure the nginx reverse proxy
You can see from the configuration that matrix listens on port `8008` for federation and client connexions.
This means you will need to create one reverse proxy in your Nginx configuration, that will listen on port 443, that will utilise the TLS certificate you have generated earlier, and listens on the servername `matrix.<yourhost>`. To do that, update the configuration of your Nginx role to upload a file to `/etc/nginx/sites-enabled/matrix` looks like something like
```nginx
server {
listen 443 ssl;
server_name matrix.<yourdomain>;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_certificate <your certificate>.crt;
ssl_certificate_key <your key>.key;
location / {
proxy_pass http://127.0.0.1:8008;
}
}
```
After a reload of nginx it should work properly.
### Configure openwrt
To make you able to talk to synapse, you need to configure openwrt to send all packets from ports 443 and 8448 (the default federation port) to your docker host on port 443. You can do that either with the UI or with the command line config
### Create an admin user
Note about the matrix addresses, a matrix address looks like this:
```
@<localpart>:<homeserver>
```
For example my address is `@thomas:matrix.maurice.fr`, the localpart is `thomas` and the homeserver is `matrix.maurice.fr`
In the following step you can create one or two users (for your team) to connect to the homeserver, pick the localpart carefully because that will be your username.
Get into your synapse host and run
```
$ docker exec -it synapse register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml
```
Fill in the localpart, the password and you should be good to go.
### Test it works
#### Check you can login
The Matrix people wrote a nice web based application for using the protocol, called [Riot](https://riot.im/app/). Go to Riot, log in by specifying the homeserver URL, username and password, and verify the login goes smoothly. If it does not, check the logs.
#### Test the federation works
The federation is required to work for you to be able to speak to users on other servers !
Enter your hostname into the [matrix federation tester](https://federationtester.matrix.org/) and check the federation works
#### Test by sending a message to another address
Fire a message to `@thomas:matrix.maurice.fr`, or better yet, send a message to one of your colleague on another homeserver.
### Congratulations you reached the end !
This is pretty cool, however there is more you can do in the optional objectives of the lab, I would suggest you have a look at:
* Setup a Gitea runner and make sure your tests pass
* Add more tests ?
* Using a mechanism (sops/ansible vault/hashicorp vault) to store secrets in the repository and make it accessible to the gitea runner
* Finally, run ansible _from the CI_ so your playbooks get applied when you push new code to the repository
\ No newline at end of file
File deleted
- hosts: all
tasks:
# Install nginx et assurer la dernière version.
- name: ensure nginx is at the latest version
apt: name=nginx state=latest
# Start nginx on the machine
- name: start nginx
service:
name: nginx
state: started
# Push the nginx server configuration file
- name: Upload the file of configuration to the nginx server.
copy:
src: matrix.conf.j2
dest: /etc/nginx/sites-enabled/matrix
# Restart nginx server
- name: Restart nginx server.
service:
name: nginx
......
---
# Cette playbook c'est pour installer la base de donner PostgreSQL sur la machine db
- name: "Install postgresql"
hosts: db
roles:
......
---
- name: Install postgresql
ansible.builtin.apt:
name: postgresql
state: present
- name : Ajouter une adresse au fichier pg_hba.conf
community.postgresql.postgresql_pg_hba:
dest: /etc/postgresql/15/main/pg_hba.conf
contype: host
source: 10.69.0.0/24
method: scram-sha-256
- name : Changement de listen adress
lineinfile:
path: /etc/postgresql/15/main/postgresql.conf
regexp: '^#?listen_addresses = .*'
line: "listen_addresses = '*'"
- name: Create a new database with name "g7-db"
community.postgresql.postgresql_db:
name: g7-db
template : template0
encoding: "UTF-8"
lc_collate: "C"
lc_ctype: "C"
become : true
become_user : postgres
- name: Connect to g7-db database, create g7-belhasna, and grant access to database
community.postgresql.postgresql_user:
db: g7-db
name: g7-belhasna
password: glopglop
become : true
become_user : postgres
- name: Connect to g7-db database, create g7-achaouni user, and grant access to database
community.postgresql.postgresql_user:
db: g7-db
name: g7-achaouni
password: glopglop
become : true
become_user : postgres
---
# Install postgreSQL server on the db machine
- name: Install postgresql
ansible.builtin.apt:
name: postgresql
state: present
# Add the addresses to the posgresql configuration file pg_hba.conf
- name : Ajouter une adresse au fichier pg_hba.conf
community.postgresql.postgresql_pg_hba:
dest: /etc/postgresql/15/main/pg_hba.conf
contype: host
source: 10.0.69.0/24
source: 10.69.0.0/24
method: scram-sha-256
# Changing the addresses so that postgreSQL listens on all the addresses.
- name : Changement de listen adress
lineinfile:
path: /etc/postgresql/15/main/postgresql.conf
regexp: '^#?listen_addresses = .*'
line: "listen_addresses = '*'"
# Create the database for g7 and spicifying the encoding to UTF-8
- name: Create a new database with name "g7-db"
community.postgresql.postgresql_db:
name: g7-db
template : template0
encoding: "UTF-8"
lc_collate: "C"
lc_ctype: "C"
become : true
# Change the user to postgres
become_user : postgres
# Create the user g7-belhasna in the database and garantee access to the user.
- name: Connect to g7-db database, create g7-belhasna, and grant access to database
community.postgresql.postgresql_user:
db: g7-db
name: g7-belhasna
password: glopglop
# Use the encrypted password, already encrypted with Ansible-vault and set as a var in /postgresql/vars/main.yml
password: "{{ postgresql_password }}"
become : true
# Change the user to postgreS
become_user : postgres
# Create the second user g7-achaouni and garantee access.
- name: Connect to g7-db database, create g7-achaouni user, and grant access to database
community.postgresql.postgresql_user:
db: g7-db
# The name of the user to create
name: g7-achaouni
password: glopglop
# Use the encrypted password, already encrypted with Ansible-vault and set as a var in /postgresql/vars/main.yml
password: "{{ postgresql_password }}"
become : true
# Change the user to postgreS
become_user : postgres
- name: Create a new database with name "g7-db"
community.postgresql.postgresql_db:
name: g7-db
template : template0
encoding: "UTF-8"
lc_collate: "C"
lc_ctype: "C"
owner: g7-belhasna
become : true
become_user : postgres
- name: Reload Postgres
ansible.builtin.service:
name: postgresql
state: reloaded
postgresql_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
66613832383534356465346565336436316639616333373333613330316564316139626234393135
3463386561626263613735613039353366323765643339330a363534356132333461303330633563
32363131326438653866383165353863313434333032666637386531346231306666333263343236
3261336530316265370a363566636261336638656265633661666163383862313338356437356239
6438
# Task description: Copies files to the Synapse server
- name: Copy files to Synapse server
# Ansible module used for copying files
ansible.builtin.copy:
# Source file or directory to be copied
src: "{{ item }}"
# Destination directory on the Synapse server
dest: "/synapse/"
# Loop through each item in the list
with_items:
- homeserver.yaml
- matrix.synapse.nyala.website.signing.key
- matrix.synapse.nyala.website.log.config
# Task description: Creates a directory
- name: Creates directory
# Ansible module used for file operations
ansible.builtin.file:
# Path of the directory to be created
path: "/synapse/media_store"
# Permissions for the directory (in octal format)
mode: 0700
state: directory
# Task description: Creates a container for Synapse
- name: Creates the synapse container
# Ansible module used for managing Docker containers
community.docker.docker_container:
name: "synapse"
# Docker image to use for the container
image: "matrixdotorg/synapse:latest"
# Desired state of the container (started)
state: started
# Indicates that the container should be recreated if it already exists
recreate: yes
# Maps port 8008 on the host to port 8008 on the container
ports:
- 8008:8008
# Mounts volumes from the host system into the container
volumes:
- /synapse:/data
# Defines the container's restart policy to be "unless-stopped"
restart_policy: "unless-stopped"
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment