docs: backups!

1 files changed, 197 insertions, 0 deletions

diff --git a/docs/backups.org b/docs/backups.org
new file mode 100644
index 0000000..0b0d25a
--- /dev/null
+++ b/docs/backups.org
@@ -0,0 +1,197 @@
+#+TITLE: Backups
+
+There's a number of backups that are managed by the NAS.
+
+In order for the backup to work, there's two files that need to be provisioned:
+- =/etc/restic/password= this contains the password for restic. It's currently stored in 1password (named *backup on nas*).
+- =/etc/restic/google.json= this contains the authn/authz information to store our data in various GCS. This is stored in 1password, with restic's password.
+
+* restic
+For backups I'm using [[https://restic.readthedocs.io/][restic]].
+
+On the NAS itself, we backup the git repositories to =/data/backups=.
+
+The password is stored in =/etc/restic/password= (this is not managed by puppet for now, and the password is stored within 1password).
+** List the snapshots
+To get a list of snapshots:
+#+BEGIN_SRC sh :dir /ssh:nas: :results verbatim
+sudo restic -r /data/backups/ -p /etc/restic/password snapshots
+#+END_SRC
+
+#+RESULTS:
+#+begin_example
+repository a37cfab5 opened successfully, password is correct
+ID        Time                 Host        Tags        Paths
+---------------------------------------------------------------------------------
+e36e9100  2020-02-29 08:43:37  nas                     /home/git/repositories
+603a46a7  2020-03-31 08:39:03  nas                     /home/git/repositories
+e890453b  2020-04-30 08:22:37  nas                     /home/git/repositories
+0affa4d9  2020-05-10 08:47:18  nas                     /home/git/repositories
+a01d8be4  2020-07-31 08:41:25  nas                     /home/git/repositories
+78afb27a  2020-08-31 08:23:52  nas                     /home/git/repositories
+68a417b1  2020-09-30 08:44:49  nas                     /home/git/repositories
+ac6701b4  2020-10-18 06:00:00  nas         git         /home/git/repositories
+4f183431  2020-10-25 06:00:00  nas         git         /home/git/repositories
+aec0b472  2020-10-25 07:24:10  aptos       home        /home/fcuny
+3e98a872  2020-10-30 06:00:00  nas         git         /home/git/repositories
+0268f733  2020-10-31 06:00:00  nas         git         /home/git/repositories
+1b840de3  2020-11-01 06:00:00  nas         git         /home/git/repositories
+2d224944  2020-11-02 06:00:00  nas         git         /home/git/repositories
+fa0107dd  2020-11-03 06:00:00  nas         git         /home/git/repositories
+1165032b  2020-11-04 06:00:00  nas         git         /home/git/repositories
+612b66e3  2020-11-05 06:00:00  nas         git         /home/git/repositories
+2de6fb79  2020-12-31 06:01:19  nas         gitea       /data/containers/gitea
+ece08207  2020-12-31 06:01:41  nas         traefik     /data/containers/traefik
+d59bd75a  2020-12-31 06:06:19  nas         grafana     /data/containers/grafana
+168c0ddf  2020-12-31 06:07:24  nas         unifi       /data/containers/unifi
+5882ffe4  2021-01-27 18:58:06  aptos       home        /home/fcuny
+3565b23b  2021-01-31 06:05:18  nas         traefik     /data/containers/traefik
+653d4411  2021-01-31 06:14:12  nas         gitea       /data/containers/gitea
+38a3e50e  2021-01-31 06:15:13  nas         unifi       /data/containers/unifi
+542e2c80  2021-01-31 06:15:13  nas         grafana     /data/containers/grafana
+8c804805  2021-02-06 19:13:24  aptos       home        /home/fcuny
+3f38d369  2021-02-28 06:03:28  nas         grafana     /data/containers/grafana
+ef2042e2  2021-02-28 06:11:50  nas         unifi       /data/containers/unifi
+b429ef99  2021-02-28 06:18:02  nas         gitea       /data/containers/gitea
+b73f5128  2021-02-28 06:18:04  nas         traefik     /data/containers/traefik
+7a7e3e06  2021-03-28 09:05:35  aptos       home        /home/fcuny
+3a0c790f  2021-03-30 06:12:20  nas         grafana     /data/containers/grafana
+58179a2f  2021-03-31 06:05:04  nas         gitea       /data/containers/gitea
+fc4ede5d  2021-03-31 06:08:18  nas         unifi       /data/containers/unifi
+5eaa5148  2021-03-31 06:17:13  nas         traefik     /data/containers/traefik
+d7c95e53  2021-04-27 18:10:36  aptos       home        /home/fcuny
+4c702501  2021-04-30 06:02:11  nas         gitea       /data/containers/gitea
+8de29c3c  2021-04-30 06:04:42  nas         unifi       /data/containers/unifi
+66664254  2021-04-30 06:08:25  nas         traefik     /data/containers/traefik
+9a3ad896  2021-04-30 06:15:15  nas         grafana     /data/containers/grafana
+344ef4c3  2021-05-15 14:22:05  aptos       home        /home/fcuny
+6141b888  2021-05-30 06:14:37  nas         traefik     /data/containers/traefik
+106c4819  2021-05-31 06:04:56  nas         grafana     /data/containers/grafana
+8e0ba4c3  2021-05-31 06:12:37  nas         gitea       /data/containers/gitea
+8cba7fbf  2021-05-31 06:17:26  nas         unifi       /data/containers/unifi
+2cc04ad6  2021-06-28 17:08:25  aptos       home        /home/fcuny
+8b04e195  2021-06-30 06:03:56  nas         grafana     /data/containers/grafana
+d21a464f  2021-06-30 06:09:56  nas         unifi       /data/containers/unifi
+f180e1a0  2021-06-30 06:10:20  nas         gitea       /data/containers/gitea
+b9e0ce43  2021-06-30 06:11:50  nas         traefik     /data/containers/traefik
+512e80fb  2021-07-23 17:25:45  aptos       home        /home/fcuny
+28b32d1f  2021-07-31 06:03:50  nas         gitea       /data/containers/gitea
+884574c8  2021-07-31 06:11:13  nas         unifi       /data/containers/unifi
+a61cd90f  2021-07-31 06:16:50  nas         grafana     /data/containers/grafana
+614f9123  2021-07-31 06:19:38  nas         traefik     /data/containers/traefik
+17698a8a  2021-08-14 06:05:34  nas         git         /data/containers/git
+b5674e76  2021-08-16 13:47:52  aptos       home        /home/fcuny
+d7c251f6  2021-08-31 06:16:07  nas         gitea       /data/containers/gitea
+ef20f101  2021-08-31 06:16:11  nas         unifi       /data/containers/unifi
+b7cd0d5c  2021-08-31 06:16:16  nas         grafana     /data/containers/grafana
+facffc9a  2021-08-31 06:16:19  nas         traefik     /data/containers/traefik
+b2d31938  2021-08-31 06:16:22  nas         syncthing   /data/containers/syncthing
+8ab3bee2  2021-09-27 10:35:27  aptos       home        /home/fcuny
+1559f48c  2021-09-30 04:11:21  nas         gitea       /data/containers/gitea
+353d202d  2021-09-30 04:11:25  nas         unifi       /data/containers/unifi
+b567fec1  2021-09-30 04:11:30  nas         grafana     /data/containers/grafana
+d7b239c1  2021-09-30 04:11:33  nas         traefik     /data/containers/traefik
+4890d748  2021-09-30 04:11:35  nas         syncthing   /data/containers/syncthing
+4d6b6646  2021-10-31 04:11:55  nas         gitea       /data/containers/gitea
+b2820465  2021-10-31 04:12:01  nas         unifi       /data/containers/unifi
+cd2230ff  2021-10-31 04:12:07  nas         grafana     /data/containers/grafana
+807f1bb3  2021-10-31 04:12:12  nas         traefik     /data/containers/traefik
+5d9c2314  2021-10-31 04:12:15  nas         syncthing   /data/containers/syncthing
+5f1a2de0  2021-10-31 12:38:40  carmel      home        /home/fcuny
+89f6bbec  2021-10-31 14:53:27  aptos       home        /home/fcuny
+5bb120c9  2021-11-05 15:54:28  aptos       home        /home/fcuny
+5fb31f63  2021-11-06 16:05:30  aptos       home        /home/fcuny
+9bfd32e2  2021-11-07 18:02:06  aptos       home        /home/fcuny
+d4dd252f  2021-11-17 13:40:16  aptos       home        /home/fcuny
+b072a3a1  2021-11-21 04:18:17  nas         gitea       /data/containers/gitea
+6ba6bff3  2021-11-21 04:18:32  nas         unifi       /data/containers/unifi
+bb697aae  2021-11-21 04:18:38  nas         grafana     /data/containers/grafana
+33ba0e83  2021-11-21 04:18:41  nas         traefik     /data/containers/traefik
+e2cae3b5  2021-11-21 04:18:43  nas         syncthing   /data/containers/syncthing
+1caaca88  2021-11-21 13:35:29  carmel      home        /home/fcuny
+97d034ce  2021-11-27 19:16:12  aptos       home        /home/fcuny
+5fa6b510  2021-11-28 04:11:27  nas         gitea       /data/containers/gitea
+6670d391  2021-11-28 04:11:32  nas         unifi       /data/containers/unifi
+77d11ce4  2021-11-28 04:11:38  nas         grafana     /data/containers/grafana
+04ee74c6  2021-11-28 04:11:40  nas         traefik     /data/containers/traefik
+1371d8d2  2021-11-28 04:11:43  nas         syncthing   /data/containers/syncthing
+3b2a45ee  2021-11-28 09:19:13  aptos       home        /home/fcuny
+b19902e6  2021-11-28 15:25:29  carmel      home        /home/fcuny
+02fb34d8  2021-11-30 04:05:15  nas         gitea       /data/containers/gitea
+1ac8f79f  2021-11-30 04:05:21  nas         unifi       /data/containers/unifi
+848505be  2021-11-30 04:05:26  nas         grafana     /data/containers/grafana
+2e48e232  2021-11-30 04:05:29  nas         traefik     /data/containers/traefik
+47732732  2021-11-30 04:05:34  nas         syncthing   /data/containers/syncthing
+dd141856  2021-11-30 12:06:56  carmel      home        /home/fcuny
+00e5429b  2021-12-03 18:31:51  aptos       home        /home/fcuny
+31b849ad  2021-12-05 04:06:10  nas         gitea       /data/containers/gitea
+8cc78932  2021-12-05 04:06:26  nas         unifi       /data/containers/unifi
+b7364a55  2021-12-05 04:06:38  nas         grafana     /data/containers/grafana
+043c4b36  2021-12-05 04:06:43  nas         traefik     /data/containers/traefik
+2e415963  2021-12-05 04:06:48  nas         syncthing   /data/containers/syncthing
+1ef944db  2021-12-05 11:14:51  carmel      home        /home/fcuny
+e58a2421  2021-12-06 04:02:44  nas         gitea       /data/containers/gitea
+907bb839  2021-12-06 04:02:50  nas         unifi       /data/containers/unifi
+050dcff3  2021-12-06 04:02:55  nas         grafana     /data/containers/grafana
+72092444  2021-12-06 04:03:00  nas         traefik     /data/containers/traefik
+d04b79bb  2021-12-06 04:03:03  nas         syncthing   /data/containers/syncthing
+2ef060ec  2021-12-06 11:36:51  carmel      home        /home/fcuny
+a3036320  2021-12-07 04:19:42  nas         gitea       /data/containers/gitea
+18af7ba5  2021-12-07 04:19:48  nas         unifi       /data/containers/unifi
+ba7adae4  2021-12-07 04:19:53  nas         grafana     /data/containers/grafana
+b71283de  2021-12-07 04:19:57  nas         traefik     /data/containers/traefik
+d1918837  2021-12-07 04:19:59  nas         syncthing   /data/containers/syncthing
+ec06c179  2021-12-07 17:24:07  carmel      home        /home/fcuny
+49722319  2021-12-08 04:11:10  nas         gitea       /data/containers/gitea
+b7cfa0d8  2021-12-08 04:11:18  nas         unifi       /data/containers/unifi
+64e98ec2  2021-12-08 04:11:25  nas         grafana     /data/containers/grafana
+d5f848fd  2021-12-08 04:11:30  nas         traefik     /data/containers/traefik
+ce58becc  2021-12-08 04:11:33  nas         syncthing   /data/containers/syncthing
+8342e5b7  2021-12-08 17:45:07  carmel      home        /home/fcuny
+93584f9e  2021-12-09 04:06:27  nas         gitea       /data/containers/gitea
+fb0e6073  2021-12-09 04:06:33  nas         unifi       /data/containers/unifi
+68d354c2  2021-12-09 04:06:39  nas         grafana     /data/containers/grafana
+73e199bd  2021-12-09 04:06:46  nas         traefik     /data/containers/traefik
+47e0e0a6  2021-12-09 04:06:49  nas         syncthing   /data/containers/syncthing
+9d7bcb97  2021-12-09 11:53:49  carmel      home        /home/fcuny
+c2130706  2021-12-10 04:00:56  nas         gitea       /data/containers/gitea
+29af7e4f  2021-12-10 04:01:03  nas         unifi       /data/containers/unifi
+393b006b  2021-12-10 04:01:08  nas         grafana     /data/containers/grafana
+433a00d1  2021-12-10 04:01:13  nas         traefik     /data/containers/traefik
+d4949919  2021-12-10 04:01:18  nas         syncthing   /data/containers/syncthing
+ce2a8a73  2021-12-10 12:10:49  carmel      home        /home/fcuny
+c8d56977  2021-12-11 04:11:20  nas         gitea       /data/containers/gitea
+40f3c6d8  2021-12-11 04:11:25  nas         unifi       /data/containers/unifi
+f24178f5  2021-12-11 04:11:30  nas         grafana     /data/containers/grafana
+3ca4553f  2021-12-11 04:11:33  nas         traefik     /data/containers/traefik
+ca41fe42  2021-12-11 04:11:35  nas         syncthing   /data/containers/syncthing
+b2643ef9  2021-12-11 12:40:49  carmel      home        /home/fcuny
+50cb9254  2021-12-12 04:10:34  nas         gitea       /data/containers/gitea
+85de9005  2021-12-12 04:10:40  nas         unifi       /data/containers/unifi
+0fd36196  2021-12-12 04:10:46  nas         grafana     /data/containers/grafana
+bd8f14dd  2021-12-12 04:10:50  nas         traefik     /data/containers/traefik
+ee0735e3  2021-12-12 04:10:53  nas         syncthing   /data/containers/syncthing
+---------------------------------------------------------------------------------
+148 snapshots
+#+end_example
+
+** How to configure a backup
+All daily backups are added to the [[file:~/workspace/infrastructure/puppet/site-modules/backup/files/etc/systemd/system/backups.service][unit file]]. Each backup needs a tag (to make it easier to filter/search).
+
+This will run once a day. The backups will be stored in =/data/backups= and then be exported to GCS.
+** How to restore the backup
+First, this is the [[https://restic.readthedocs.io/en/latest/050_restore.html][documentation]] to read. Here's an example:
+#+begin_src sh
+$ sudo restic -r /data/backups/ -p /etc/restic/password restore 8dbaaf98 --target /tmp/this-is-a-test
+repository a37cfab5 opened successfully, password is correct
+restoring <Snapshot 8dbaaf98 of [/data/containers/traefik] at 2021-08-14 06:05:49.547829076 -0700 PDT by restic@nas> to /tmp/this-is-a-test
+$ sudo ls -l /tmp/this-is-a-test/data/containers/traefik
+total 4
+drwxrwxr-x 2 root root 4096 Nov  6  2020 config
+#+end_src
+* rclone / GCP
+Backups are exported off-site to some GCS buckets, using [[https://rclone.org/][rclone]].
+
+=restic= snapshots are exported to this [[https://console.cloud.google.com/storage/browser/fcuny-restic;tab=objects?forceOnBucketsSortingFiltering=false&project=fcuny-backups][bucket]], while our music collection is stored in this [[https://console.cloud.google.com/storage/browser/fcuny-music;tab=objects?forceOnBucketsSortingFiltering=false&project=fcuny-backups&prefix=&forceOnObjectsSortingFiltering=false][one]].
+
+The timer for the backup can be found in [[file:~/workspace/infrastructure/puppet/site-modules/backup/manifests/service.pp][service.pp]]. All the configuration bits for =rclone= are parts of the unit file for the backups.