diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/backups.org | 197 | ||||
| -rw-r--r-- | docs/desktop.org | 19 | ||||
| -rw-r--r-- | docs/gcloud.org | 21 | ||||
| -rw-r--r-- | docs/gnome-keyring.org | 66 | ||||
| -rw-r--r-- | docs/install.org | 153 | ||||
| -rw-r--r-- | docs/tools.org | 128 | ||||
| -rw-r--r-- | docs/wireguard.org | 23 |
7 files changed, 607 insertions, 0 deletions
diff --git a/docs/backups.org b/docs/backups.org new file mode 100644 index 0000000..0b0d25a --- /dev/null +++ b/docs/backups.org @@ -0,0 +1,197 @@ +#+TITLE: Backups + +There's a number of backups that are managed by the NAS. + +In order for the backup to work, there's two files that need to be provisioned: +- =/etc/restic/password= this contains the password for restic. It's currently stored in 1password (named *backup on nas*). +- =/etc/restic/google.json= this contains the authn/authz information to store our data in various GCS. This is stored in 1password, with restic's password. + +* restic +For backups I'm using [[https://restic.readthedocs.io/][restic]]. + +On the NAS itself, we backup the git repositories to =/data/backups=. + +The password is stored in =/etc/restic/password= (this is not managed by puppet for now, and the password is stored within 1password). +** List the snapshots +To get a list of snapshots: +#+BEGIN_SRC sh :dir /ssh:nas: :results verbatim +sudo restic -r /data/backups/ -p /etc/restic/password snapshots +#+END_SRC + +#+RESULTS: +#+begin_example +repository a37cfab5 opened successfully, password is correct +ID Time Host Tags Paths +--------------------------------------------------------------------------------- +e36e9100 2020-02-29 08:43:37 nas /home/git/repositories +603a46a7 2020-03-31 08:39:03 nas /home/git/repositories +e890453b 2020-04-30 08:22:37 nas /home/git/repositories +0affa4d9 2020-05-10 08:47:18 nas /home/git/repositories +a01d8be4 2020-07-31 08:41:25 nas /home/git/repositories +78afb27a 2020-08-31 08:23:52 nas /home/git/repositories +68a417b1 2020-09-30 08:44:49 nas /home/git/repositories +ac6701b4 2020-10-18 06:00:00 nas git /home/git/repositories +4f183431 2020-10-25 06:00:00 nas git /home/git/repositories +aec0b472 2020-10-25 07:24:10 aptos home /home/fcuny +3e98a872 2020-10-30 06:00:00 nas git /home/git/repositories +0268f733 2020-10-31 06:00:00 nas git /home/git/repositories +1b840de3 2020-11-01 06:00:00 nas git /home/git/repositories +2d224944 2020-11-02 06:00:00 nas git /home/git/repositories +fa0107dd 2020-11-03 06:00:00 nas git /home/git/repositories +1165032b 2020-11-04 06:00:00 nas git /home/git/repositories +612b66e3 2020-11-05 06:00:00 nas git /home/git/repositories +2de6fb79 2020-12-31 06:01:19 nas gitea /data/containers/gitea +ece08207 2020-12-31 06:01:41 nas traefik /data/containers/traefik +d59bd75a 2020-12-31 06:06:19 nas grafana /data/containers/grafana +168c0ddf 2020-12-31 06:07:24 nas unifi /data/containers/unifi +5882ffe4 2021-01-27 18:58:06 aptos home /home/fcuny +3565b23b 2021-01-31 06:05:18 nas traefik /data/containers/traefik +653d4411 2021-01-31 06:14:12 nas gitea /data/containers/gitea +38a3e50e 2021-01-31 06:15:13 nas unifi /data/containers/unifi +542e2c80 2021-01-31 06:15:13 nas grafana /data/containers/grafana +8c804805 2021-02-06 19:13:24 aptos home /home/fcuny +3f38d369 2021-02-28 06:03:28 nas grafana /data/containers/grafana +ef2042e2 2021-02-28 06:11:50 nas unifi /data/containers/unifi +b429ef99 2021-02-28 06:18:02 nas gitea /data/containers/gitea +b73f5128 2021-02-28 06:18:04 nas traefik /data/containers/traefik +7a7e3e06 2021-03-28 09:05:35 aptos home /home/fcuny +3a0c790f 2021-03-30 06:12:20 nas grafana /data/containers/grafana +58179a2f 2021-03-31 06:05:04 nas gitea /data/containers/gitea +fc4ede5d 2021-03-31 06:08:18 nas unifi /data/containers/unifi +5eaa5148 2021-03-31 06:17:13 nas traefik /data/containers/traefik +d7c95e53 2021-04-27 18:10:36 aptos home /home/fcuny +4c702501 2021-04-30 06:02:11 nas gitea /data/containers/gitea +8de29c3c 2021-04-30 06:04:42 nas unifi /data/containers/unifi +66664254 2021-04-30 06:08:25 nas traefik /data/containers/traefik +9a3ad896 2021-04-30 06:15:15 nas grafana /data/containers/grafana +344ef4c3 2021-05-15 14:22:05 aptos home /home/fcuny +6141b888 2021-05-30 06:14:37 nas traefik /data/containers/traefik +106c4819 2021-05-31 06:04:56 nas grafana /data/containers/grafana +8e0ba4c3 2021-05-31 06:12:37 nas gitea /data/containers/gitea +8cba7fbf 2021-05-31 06:17:26 nas unifi /data/containers/unifi +2cc04ad6 2021-06-28 17:08:25 aptos home /home/fcuny +8b04e195 2021-06-30 06:03:56 nas grafana /data/containers/grafana +d21a464f 2021-06-30 06:09:56 nas unifi /data/containers/unifi +f180e1a0 2021-06-30 06:10:20 nas gitea /data/containers/gitea +b9e0ce43 2021-06-30 06:11:50 nas traefik /data/containers/traefik +512e80fb 2021-07-23 17:25:45 aptos home /home/fcuny +28b32d1f 2021-07-31 06:03:50 nas gitea /data/containers/gitea +884574c8 2021-07-31 06:11:13 nas unifi /data/containers/unifi +a61cd90f 2021-07-31 06:16:50 nas grafana /data/containers/grafana +614f9123 2021-07-31 06:19:38 nas traefik /data/containers/traefik +17698a8a 2021-08-14 06:05:34 nas git /data/containers/git +b5674e76 2021-08-16 13:47:52 aptos home /home/fcuny +d7c251f6 2021-08-31 06:16:07 nas gitea /data/containers/gitea +ef20f101 2021-08-31 06:16:11 nas unifi /data/containers/unifi +b7cd0d5c 2021-08-31 06:16:16 nas grafana /data/containers/grafana +facffc9a 2021-08-31 06:16:19 nas traefik /data/containers/traefik +b2d31938 2021-08-31 06:16:22 nas syncthing /data/containers/syncthing +8ab3bee2 2021-09-27 10:35:27 aptos home /home/fcuny +1559f48c 2021-09-30 04:11:21 nas gitea /data/containers/gitea +353d202d 2021-09-30 04:11:25 nas unifi /data/containers/unifi +b567fec1 2021-09-30 04:11:30 nas grafana /data/containers/grafana +d7b239c1 2021-09-30 04:11:33 nas traefik /data/containers/traefik +4890d748 2021-09-30 04:11:35 nas syncthing /data/containers/syncthing +4d6b6646 2021-10-31 04:11:55 nas gitea /data/containers/gitea +b2820465 2021-10-31 04:12:01 nas unifi /data/containers/unifi +cd2230ff 2021-10-31 04:12:07 nas grafana /data/containers/grafana +807f1bb3 2021-10-31 04:12:12 nas traefik /data/containers/traefik +5d9c2314 2021-10-31 04:12:15 nas syncthing /data/containers/syncthing +5f1a2de0 2021-10-31 12:38:40 carmel home /home/fcuny +89f6bbec 2021-10-31 14:53:27 aptos home /home/fcuny +5bb120c9 2021-11-05 15:54:28 aptos home /home/fcuny +5fb31f63 2021-11-06 16:05:30 aptos home /home/fcuny +9bfd32e2 2021-11-07 18:02:06 aptos home /home/fcuny +d4dd252f 2021-11-17 13:40:16 aptos home /home/fcuny +b072a3a1 2021-11-21 04:18:17 nas gitea /data/containers/gitea +6ba6bff3 2021-11-21 04:18:32 nas unifi /data/containers/unifi +bb697aae 2021-11-21 04:18:38 nas grafana /data/containers/grafana +33ba0e83 2021-11-21 04:18:41 nas traefik /data/containers/traefik +e2cae3b5 2021-11-21 04:18:43 nas syncthing /data/containers/syncthing +1caaca88 2021-11-21 13:35:29 carmel home /home/fcuny +97d034ce 2021-11-27 19:16:12 aptos home /home/fcuny +5fa6b510 2021-11-28 04:11:27 nas gitea /data/containers/gitea +6670d391 2021-11-28 04:11:32 nas unifi /data/containers/unifi +77d11ce4 2021-11-28 04:11:38 nas grafana /data/containers/grafana +04ee74c6 2021-11-28 04:11:40 nas traefik /data/containers/traefik +1371d8d2 2021-11-28 04:11:43 nas syncthing /data/containers/syncthing +3b2a45ee 2021-11-28 09:19:13 aptos home /home/fcuny +b19902e6 2021-11-28 15:25:29 carmel home /home/fcuny +02fb34d8 2021-11-30 04:05:15 nas gitea /data/containers/gitea +1ac8f79f 2021-11-30 04:05:21 nas unifi /data/containers/unifi +848505be 2021-11-30 04:05:26 nas grafana /data/containers/grafana +2e48e232 2021-11-30 04:05:29 nas traefik /data/containers/traefik +47732732 2021-11-30 04:05:34 nas syncthing /data/containers/syncthing +dd141856 2021-11-30 12:06:56 carmel home /home/fcuny +00e5429b 2021-12-03 18:31:51 aptos home /home/fcuny +31b849ad 2021-12-05 04:06:10 nas gitea /data/containers/gitea +8cc78932 2021-12-05 04:06:26 nas unifi /data/containers/unifi +b7364a55 2021-12-05 04:06:38 nas grafana /data/containers/grafana +043c4b36 2021-12-05 04:06:43 nas traefik /data/containers/traefik +2e415963 2021-12-05 04:06:48 nas syncthing /data/containers/syncthing +1ef944db 2021-12-05 11:14:51 carmel home /home/fcuny +e58a2421 2021-12-06 04:02:44 nas gitea /data/containers/gitea +907bb839 2021-12-06 04:02:50 nas unifi /data/containers/unifi +050dcff3 2021-12-06 04:02:55 nas grafana /data/containers/grafana +72092444 2021-12-06 04:03:00 nas traefik /data/containers/traefik +d04b79bb 2021-12-06 04:03:03 nas syncthing /data/containers/syncthing +2ef060ec 2021-12-06 11:36:51 carmel home /home/fcuny +a3036320 2021-12-07 04:19:42 nas gitea /data/containers/gitea +18af7ba5 2021-12-07 04:19:48 nas unifi /data/containers/unifi +ba7adae4 2021-12-07 04:19:53 nas grafana /data/containers/grafana +b71283de 2021-12-07 04:19:57 nas traefik /data/containers/traefik +d1918837 2021-12-07 04:19:59 nas syncthing /data/containers/syncthing +ec06c179 2021-12-07 17:24:07 carmel home /home/fcuny +49722319 2021-12-08 04:11:10 nas gitea /data/containers/gitea +b7cfa0d8 2021-12-08 04:11:18 nas unifi /data/containers/unifi +64e98ec2 2021-12-08 04:11:25 nas grafana /data/containers/grafana +d5f848fd 2021-12-08 04:11:30 nas traefik /data/containers/traefik +ce58becc 2021-12-08 04:11:33 nas syncthing /data/containers/syncthing +8342e5b7 2021-12-08 17:45:07 carmel home /home/fcuny +93584f9e 2021-12-09 04:06:27 nas gitea /data/containers/gitea +fb0e6073 2021-12-09 04:06:33 nas unifi /data/containers/unifi +68d354c2 2021-12-09 04:06:39 nas grafana /data/containers/grafana +73e199bd 2021-12-09 04:06:46 nas traefik /data/containers/traefik +47e0e0a6 2021-12-09 04:06:49 nas syncthing /data/containers/syncthing +9d7bcb97 2021-12-09 11:53:49 carmel home /home/fcuny +c2130706 2021-12-10 04:00:56 nas gitea /data/containers/gitea +29af7e4f 2021-12-10 04:01:03 nas unifi /data/containers/unifi +393b006b 2021-12-10 04:01:08 nas grafana /data/containers/grafana +433a00d1 2021-12-10 04:01:13 nas traefik /data/containers/traefik +d4949919 2021-12-10 04:01:18 nas syncthing /data/containers/syncthing +ce2a8a73 2021-12-10 12:10:49 carmel home /home/fcuny +c8d56977 2021-12-11 04:11:20 nas gitea /data/containers/gitea +40f3c6d8 2021-12-11 04:11:25 nas unifi /data/containers/unifi +f24178f5 2021-12-11 04:11:30 nas grafana /data/containers/grafana +3ca4553f 2021-12-11 04:11:33 nas traefik /data/containers/traefik +ca41fe42 2021-12-11 04:11:35 nas syncthing /data/containers/syncthing +b2643ef9 2021-12-11 12:40:49 carmel home /home/fcuny +50cb9254 2021-12-12 04:10:34 nas gitea /data/containers/gitea +85de9005 2021-12-12 04:10:40 nas unifi /data/containers/unifi +0fd36196 2021-12-12 04:10:46 nas grafana /data/containers/grafana +bd8f14dd 2021-12-12 04:10:50 nas traefik /data/containers/traefik +ee0735e3 2021-12-12 04:10:53 nas syncthing /data/containers/syncthing +--------------------------------------------------------------------------------- +148 snapshots +#+end_example + +** How to configure a backup +All daily backups are added to the [[file:~/workspace/infrastructure/puppet/site-modules/backup/files/etc/systemd/system/backups.service][unit file]]. Each backup needs a tag (to make it easier to filter/search). + +This will run once a day. The backups will be stored in =/data/backups= and then be exported to GCS. +** How to restore the backup +First, this is the [[https://restic.readthedocs.io/en/latest/050_restore.html][documentation]] to read. Here's an example: +#+begin_src sh +$ sudo restic -r /data/backups/ -p /etc/restic/password restore 8dbaaf98 --target /tmp/this-is-a-test +repository a37cfab5 opened successfully, password is correct +restoring <Snapshot 8dbaaf98 of [/data/containers/traefik] at 2021-08-14 06:05:49.547829076 -0700 PDT by restic@nas> to /tmp/this-is-a-test +$ sudo ls -l /tmp/this-is-a-test/data/containers/traefik +total 4 +drwxrwxr-x 2 root root 4096 Nov 6 2020 config +#+end_src +* rclone / GCP +Backups are exported off-site to some GCS buckets, using [[https://rclone.org/][rclone]]. + +=restic= snapshots are exported to this [[https://console.cloud.google.com/storage/browser/fcuny-restic;tab=objects?forceOnBucketsSortingFiltering=false&project=fcuny-backups][bucket]], while our music collection is stored in this [[https://console.cloud.google.com/storage/browser/fcuny-music;tab=objects?forceOnBucketsSortingFiltering=false&project=fcuny-backups&prefix=&forceOnObjectsSortingFiltering=false][one]]. + +The timer for the backup can be found in [[file:~/workspace/infrastructure/puppet/site-modules/backup/manifests/service.pp][service.pp]]. All the configuration bits for =rclone= are parts of the unit file for the backups. diff --git a/docs/desktop.org b/docs/desktop.org new file mode 100644 index 0000000..a52fc53 --- /dev/null +++ b/docs/desktop.org @@ -0,0 +1,19 @@ +* Next build +** Requirements +- Future proof (PCIe 5, DDR5) +- Re-use the nr200p case +- 2 NVMe drive would be nice +- not have to use a GPU would be nice +** Hardware selection + +| component | model | price | note | +|-------------+-----------------------------------------------+-------+------| +| CPU | Intel Core i7-12700K | 380 | | +| CPU cooler | Noctua NH-U9S chromax.black | 0 | | +| motherboard | Asus ROG STRIX B660-I GAMING | 220 | | +| memory | Corsair Vengeance 32 GB (2 x 16 GB) DDR5-5200 | 309 | | +| boot drive | Western Digital Black SN850 | 160 | | +| case | nr200p | 0 | | +|-------------+-----------------------------------------------+-------+------| +| | | 1069 | | +#+TBLFM: @8$3=vsum(@2..@-1) diff --git a/docs/gcloud.org b/docs/gcloud.org new file mode 100644 index 0000000..95e7531 --- /dev/null +++ b/docs/gcloud.org @@ -0,0 +1,21 @@ +#+TITLE: Gcloud + +* Initial setup +First we need to create a service account, with: +#+begin_src sh +gcloud --project fcuny-homelab iam service-accounts create world-nix +#+end_src + +Next we need to bind the new policy: +#+begin_src sh +gcloud projects add-iam-policy-binding fcuny-homelab --member="serviceAccount:world-nix@fcuny-homelab.iam.gserviceaccount.com" --role="roles/accessapproval.configEditor" +#+end_src + +Note: I had to add DNS administrator in the console, I don't know what I need to add to this command. + +Finally we need the key: +#+begin_src sh +gcloud iam service-accounts keys create world-nix.json --iam-account=world-nix@fcuny-homelab.iam.gserviceaccount.com +#+end_src + +This will create a file name =world-nix.json=. It's best to encrypt it with =age= and move it under the =secrets= directory for a host. diff --git a/docs/gnome-keyring.org b/docs/gnome-keyring.org new file mode 100644 index 0000000..35480e5 --- /dev/null +++ b/docs/gnome-keyring.org @@ -0,0 +1,66 @@ +#+TITLE: gnome-keyring-daemon setup + +It seems that there's a lot of hate for the =gnome-keyring-daemon= online, so I might be missing something. But on my end, it seems to simplifies a few things and there are no more prompt when I log into my session about various keys. + +* gnome-keyring-daemon +It looks like we need to install a few packages: +- =gnome-keyring= +- =seahorse= + +There is a [[file:~/workspace/linux-desktop/systemd/gnome-keyring.service][unit]] that ensure it starts when we log in a session. + +Using =seahorse=, we can see which secrets / keys are managed by it. + +Additional documentations: +- [[https://wiki.archlinux.org/title/GNOME/Keyring][arch wiki]] +* PGP +** Unlocking the key +The keyring daemon unlocks the key for us. +** Backup the key +To backup the key, do +#+begin_src sh +gpg --export-secret-keys --armor franck@fcuny.net > ~/documents/backups/gpg-secret-key-backup.asc +#+end_src + +To see the list of keys: +#+begin_src sh :results verbatim raw +gpg --list-secret-keys +#+end_src + +#+RESULTS: +/home/fcuny/.gnupg/pubring.kbx +------------------------------ +sec rsa4096 2021-09-13 [SC] + 23348B57F01D4234B5CFBA0923208AC01EB6EEA1 +uid [ultimate] Franck Cuny <franck@fcuny.net> +ssb rsa4096 2021-09-13 [E] + +To export the trusted keys: +#+begin_src sh +gpg --export-ownertrust > ~/documents/backups/gpg-trusteddb-backup.txt +#+end_src + +** Restore the key +To restore the key from the backup +#+begin_src sh +gpg --import ~/documents/backups/gpg-secret-key-backup.asc +#+end_src + +To restore the trusted db: +#+begin_src sh +gpg --import-ownertrust < ~/documents/backups/gpg-trusteddb-backup.txt +#+end_src + +If you don't import the trusted db you need to set your key as trusted +#+begin_src +gpg --edit-key franck@fcuny.net +gpg> trust +gpg> save +#+end_src +** Configuration for the agent +In =$HOME/.gnupg/gpg-agent.conf= +#+begin_src conf +pinentry-program /usr/bin/pinentry-gnome3 +#+end_src +* SSH +As the keyring daemon manages our ssh key, all we need to do is to export =SSH_AUTH_SOCK= to where the socket started by the daemon is. This is done in [[file:~/workspace/linux-desktop/dotfiles/pam_environment][pam_environment]]. diff --git a/docs/install.org b/docs/install.org new file mode 100644 index 0000000..40ba5a8 --- /dev/null +++ b/docs/install.org @@ -0,0 +1,153 @@ +#+TITLE: Installation +#+AUTHOR: Franck Cuny +#+EMAIL: franck@fcuny.net + +* Prepare the USB stick +Download the most recent image from https://nixos.org/download.html then put it on a stick: +#+begin_src sh +sudo cp ~/downloads/nixos-minimal-21.11.336020.2128d0aa28e-x86_64-linux.iso /dev/sda +#+end_src +* Partitioning +** For the workstation (desktop/laptop) +All hosts have the same partitioning for the boot drive: +- /boot partition for UEFI +- / encrypted with btrfs +- a 8GB swap + +If we assume the boot drive to be =nvme0n1=, we will do the following: +#+begin_src sh +parted /dev/nvme0n1 -- mklabel gpt +parted /dev/nvme0n1 -- mkpart primary 512MiB -8GiB +parted /dev/nvme0n1 -- mkpart primary linux-swap -8GiB 100% +parted /dev/nvme0n1 -- mkpart ESP fat32 1MiB 512MiB +parted /dev/nvme0n1 -- set 3 esp on +#+end_src + +Running =lsbkl= should give the following output: +#+begin_src sh +[root@nixos:~]# lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +loop0 7:0 0 709M 1 loop /nix/.ro-store +sda 8:0 1 29.9G 0 disk +├─sda1 8:1 1 784M 0 part /iso +└─sda2 8:2 1 37M 0 part +nvme0n1 259:0 0 465.8G 0 disk +├─nvme0n1p1 259:1 0 457.3G 0 part +├─nvme0n1p2 259:2 0 8G 0 part +└─nvme0n1p3 259:3 0 511M 0 part +#+end_src + +Then we create the LUKS device: +#+begin_src sh +cryptsetup --verify-passphrase -v luksFormat /dev/nvme0n1p1 +cryptsetup open /dev/nvme0n1p1 system +#+end_src + +We can create the partition for the boot drive and activate the swap: +#+begin_src sh +mkswap -L swap /dev/nvme0n1p2 +swapon /dev/nvme0n1p2 +mkfs.fat -F 32 -n nixos-boot /dev/nvme0n1p3 +#+end_src +#+begin_src sh +mkfs.btrfs /dev/mapper/system + +mount -t btrfs /dev/mapper/system /mnt + +btrfs subvolume create /mnt/nixos +btrfs subvolume create /mnt/home +btrfs subvolume create /mnt/snapshots + +umount /mnt +#+end_src + +Now we can re-mount the partitions with the proper options: +#+begin_src sh +mount -o subvol=nixos,compress=zstd,noatime,autodefrag /dev/mapper/system /mnt + +mkdir /mnt/{home,boot,.snapshots} + +mount -o subvol=home,compress=zstd,noatime,autodefrag /dev/mapper/system /mnt/home +mount -o subvol=snapshots,compress=zstd,noatime /dev/mapper/system /mnt/.snapshots +mount /dev/nvme0n1p3 /mnt/boot +#+end_src + +Once the installation is completed: +#+begin_src sh +CUSTOMIZE_TIMESTAMP=$(date -u +%Y%m%dT%H%M%S) +btrfs subvolume snapshot /mnt /mnt/.snapshots/$CUSTOMIZE_TIMESTAMP +#+end_src +** Partitions for the NAS +Create the RAIDs: +#+begin_src sh +mdadm --create /dev/md/fast --level=mirror --raid-devices=2 /dev/sda /dev/sdb +mdadm --create /dev/md/slow --level=mirror --raid-devices=2 /dev/sdc /dev/sde +#+end_src + +Encrypt the RAIDs: +#+begin_src sh +cryptsetup --verify-passphrase -v luksFormat /dev/md/slow +cryptsetup --verify-passphrase -v luksFormat /dev/md/fast +#+end_src + +Then open them: +#+begin_src sh +cryptsetup open /dev/md/fast raid-fast +cryptsetup open /dev/md/slow raid-slow +#+end_src + +Create the filesystem: +#+begin_src sh +mkfs.btrfs /dev/mapper/raid-fast +mkfs.btrfs /dev/mapper/raid-slow +#+end_src + +Then we can mount them to generate the host configuration +#+begin_src sh +btrfs subvolume create /mnt/media +btrfs subvolume create /mnt/containers +umount /mnt + +mount -t btrfs /dev/mapper/raid-slow /mnt/ +btrfs subvolume create /mnt/backups +mkdir /mnt/data/{backups,containers,media} +mount -o subvol=media,compress=zstd,noatime,autodefrag /dev/mapper/raid-fast /mnt/data/media +mount -o subvol=media,compress=zstd,noatime,autodefrag /dev/mapper/raid-fast /mnt/data/media +mount -o subvol=containers,compress=zstd,noatime,autodefrag /dev/mapper/raid-fast /mnt/data/containers +mount -o subvol=backups,compress=zstd,noatime,autodefrag /dev/mapper/raid-slow /mnt/data/backups +#+end_src +* Installing the system +Let's add git and nixFlakes: +#+begin_src sh +nix-shell -p git nixFlakes +#+end_src + +#+begin_src sh +nixos-generate-config --root /mnt +mkdir /mnt/root +git clone https://git.fcuny.net/fcuny/world.git /mnt/root/world +mkdir /mnt/root/world/hosts/<host name> +cp /mnt/etc/nixos/hardware-configuration.nix /mnt/root/world/hosts/<host name>/ +cp /mnt/root/world/hosts/aptos/default.nix /mnt/root/world/hosts/<host name>/ +vim /mnt/root/world/hosts/<host name>/default.nix +cd /mnt/root/world +git add hosts/tahoe +cd / +nixos-install --root /mnt --flake /mnt/root/world#<host name> +#+end_src + +Create another snapshot +#+begin_src sh +CUSTOMIZE_TIMESTAMP=$(date -u +%Y%m%dT%H%M%S) +btrfs subvolume snapshot /mnt /mnt/.snapshots/$CUSTOMIZE_TIMESTAMP +#+end_src + +And a =reboot= should be enough. +* home-manager initial install +After a reboot, as root: +#+begin_src sh +nix-channel --add https://github.com/nix-community/home-manager/archive/release-21.11.tar.gz home-manager +nix-channel --update +nix-shell '<home-manager>' -A install +home-manager build --flake .#fcuny@<host name> +#+end_src diff --git a/docs/tools.org b/docs/tools.org new file mode 100644 index 0000000..8dfebe1 --- /dev/null +++ b/docs/tools.org @@ -0,0 +1,128 @@ +#+TITLE: Collection of recipes for various tools + +* syncthing +** connection to the remote UI +The web UI for syncthing is binded to localhost. To access the UI of a remote host, create a SSH tunnel: +#+begin_src sh +ssh -L 1235:localhost:8384 -N -f 192.168.0.106 +#+end_src +* yt-dlp +- use =--merge-output-format=mkv= +- check what's the best audio and video for a video +- prefer =mp4= for the audio over =webm= + +** List of supported formats +#+begin_src sh :results verbatim +yt-dlp --list-formats https://www.youtube.com/watch?v=igH-NgcuW2M +#+end_src + +#+RESULTS: +#+begin_example +[youtube] igH-NgcuW2M: Downloading webpage +[youtube] igH-NgcuW2M: Downloading android player API JSON +[info] Available formats for igH-NgcuW2M: +ID EXT RESOLUTION FPS | FILESIZE TBR PROTO | VCODEC VBR ACODEC ABR ASR MORE INFO +--- ---- ---------- --- - ---------- ----- ----- - ----------- ----- --------- ---- ------- ----------------- +139 m4a audio only | 15.00MiB 47k https | mp4a.40.5 47k 22050Hz low, m4a_dash +249 webm audio only | 15.28MiB 48k https | opus 48k 48000Hz low, webm_dash +250 webm audio only | 19.58MiB 62k https | opus 62k 48000Hz low, webm_dash +140 m4a audio only | 40.06MiB 127k https | mp4a.40.2 127k 44100Hz medium, m4a_dash +251 webm audio only | 39.20MiB 124k https | opus 124k 48000Hz medium, webm_dash +17 3gp 176x144 12 | 24.81MiB 78k https | mp4v.20.3 78k mp4a.40.2 0k 22050Hz 144p +160 mp4 256x144 12 | 34.44MiB 109k https | avc1.4d400c 109k 144p, mp4_dash +278 webm 256x144 12 | 28.61MiB 90k https | vp9 90k 144p, webm_dash +133 mp4 426x240 24 | 77.23MiB 244k https | avc1.4d4015 244k 240p, mp4_dash +242 webm 426x240 24 | 72.41MiB 229k https | vp9 229k 240p, webm_dash +134 mp4 640x360 24 | 178.23MiB 565k https | avc1.4d401e 565k 360p, mp4_dash +18 mp4 640x360 24 | 231.71MiB 734k https | avc1.42001E 734k mp4a.40.2 0k 44100Hz 360p +243 webm 640x360 24 | 137.73MiB 436k https | vp9 436k 360p, webm_dash +135 mp4 854x480 24 | 329.98MiB 1046k https | avc1.4d401e 1046k 480p, mp4_dash +244 webm 854x480 24 | 244.94MiB 776k https | vp9 776k 480p, webm_dash +136 mp4 1280x720 24 | 638.05MiB 2023k https | avc1.4d401f 2023k 720p, mp4_dash +22 mp4 1280x720 24 | 2150k https | avc1.64001F 2150k mp4a.40.2 0k 44100Hz 720p +247 webm 1280x720 24 | 490.14MiB 1554k https | vp9 1554k 720p, webm_dash +137 mp4 1920x1080 24 | 1.13GiB 3685k https | avc1.640028 3685k 1080p, mp4_dash +248 webm 1920x1080 24 | 893.45MiB 2833k https | vp9 2833k 1080p, webm_dash +#+end_example +** Best audio and video +#+begin_src sh +yt-dlp -f 'bv*+ba' https://www.youtube.com/watch?v=igH-NgcuW2M -o '%(id)s.%(ext)s' +#+end_src +** Download a playlist +Save into =channel_id/playlist_id= directory with the video added to an archive text file: +#+begin_src sh +yt-dlp -f 'bv*[height=1080]+ba' --download-archive videos.txt https://www.youtube.com/playlist?list=PLlVlyGVtvuVnUjA4d6gHKCSrLAAm2n1e6 -o '%(channel_id)s/%(playlist_id)s/%(id)s.%(ext)s' +#+end_src +** Download a channel +#+begin_src sh +yt-dlp -f 'bv*[height=720]+ba' --download-archive videos.txt https://www.youtube.com/c/FootheFlowerhorn/videos -o '%(channel)s/%(title)s.%(ext)s' +#+end_src +* exiftool +** Copy media based on the creation date +#+begin_src sh +exiftool -v -o . '-Directory<CreateDate' -d /data/photos/%Y/%Y-%m-%d/ . +#+end_src +** Move media based on the creation date +#+begin_src sh +exiftool -v '-Directory<CreateDate' -d /data/photos/%Y/%Y-%m-%d/ . +#+end_src + +Alternatively, in case the creation date is incorrect: +#+begin_src sh +exiftool -v '-Directory<DateTimeOriginal' -d /data/photos/%Y/%Y-%m-%d/ +#+end_src +** Move pdf to a directory +To move papers (for example) using the title and date of creation to a specific destination: +#+begin_src sh +exiftool '-filename<${Title;}.%e' '-directory<CreateDate' -d ~/documents/papers/%Y/ . +#+end_src +** Edit metadata from a google takeout +This [[https://github.com/kaytat/exiftool-scripts-for-takeout][repository]] as a few scripts for =exiftools= that are interesting. In case this repository were to disappear in the future, here is the script to update the metadata from the JSON files: +#+begin_src sh :filename use_json.args +# Fill in from Google's JSON + +# Look at all media files and ignore JSON +--ext +json + +# Recursive +-r + +# Show processed filenames +-v0 + +# Check if the corresponding JSON exists +-if +(-e "${Directory}/${Filename}".".json") + +# Attempt to modify media only if the info doesn't already exist +-if +($Filetype eq "MP4" and not $quicktime:TrackCreateDate) or ($Filetype eq "MP4" and $quicktime:TrackCreateDate eq "0000:00:00 00:00:00") or ($Filetype eq "JPEG" and not $exif:DateTimeOriginal) or ($Filetype eq "PNG" and not $PNG:CreationTime) + +# Attempt to read in the JSON +-tagsfromfile +%d%F.json + +# +# Write out the tags. Use ConvertUnixTime to try and convert the UTC timestamp +# to a reasonable local EXIF string. +# + +# EXIF for regular JPG photos +-AllDates<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,1)} + +# PNG-specific +-XMP-Exif:DateTimeOriginal<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,1)} +-PNG:CreationTime<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,1)} + +# Quicktime / MP4. Assume that timestamp is in UTC. +-QuickTime:TrackCreateDate<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,0)} +-QuickTime:TrackModifyDate<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,0)} +-QuickTime:MediaCreateDate<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,0)} +-QuickTime:MediaModifyDate<${PhotoTakenTimeTimestamp;$_=ConvertUnixTime($_,0)} + +# Clobber everything +-overwrite_original +#+end_src + +and to run it: =exiftool -@ use_json.args <takeout_dir>= diff --git a/docs/wireguard.org b/docs/wireguard.org new file mode 100644 index 0000000..456205f --- /dev/null +++ b/docs/wireguard.org @@ -0,0 +1,23 @@ +#+TITLE: Configuration for wireguard + +* Creating the keys +Create a directory with the hostname under =secrets/network/=. + +We need a key for the host: +#+begin_src sh +(umask 0077; wg genkey > peer_A.key) +#+end_src + +Next we create the public key: +#+begin_src sh +wg pubkey < peer_A.key > peer_A.pub +#+end_src + +Now we need to add the private key to the list of secrets: +#+begin_src sh +nix run github:ryantm/agenix -- -e secrets/network/<host name>/wireguard_privatekey.age +#+end_src + +Once this is done, update [[file:~/workspace/world/configs/wireguard.toml][wireguard.toml]] to add the new peer with the public key. + +Once this is completed, we can delete the files =peer_A.key= and =peer_A.pub=. |