about summary refs log tree commit diff
path: root/docs/backups.org
blob: 0b0d25a75fb851370c0a986cf7d2aa82dfa3fc21 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
#+TITLE: Backups

There's a number of backups that are managed by the NAS.

In order for the backup to work, there's two files that need to be provisioned:
- =/etc/restic/password= this contains the password for restic. It's currently stored in 1password (named *backup on nas*).
- =/etc/restic/google.json= this contains the authn/authz information to store our data in various GCS. This is stored in 1password, with restic's password.

* restic
For backups I'm using [[https://restic.readthedocs.io/][restic]].

On the NAS itself, we backup the git repositories to =/data/backups=.

The password is stored in =/etc/restic/password= (this is not managed by puppet for now, and the password is stored within 1password).
** List the snapshots
To get a list of snapshots:
#+BEGIN_SRC sh :dir /ssh:nas: :results verbatim
sudo restic -r /data/backups/ -p /etc/restic/password snapshots
#+END_SRC

#+RESULTS:
#+begin_example
repository a37cfab5 opened successfully, password is correct
ID        Time                 Host        Tags        Paths
---------------------------------------------------------------------------------
e36e9100  2020-02-29 08:43:37  nas                     /home/git/repositories
603a46a7  2020-03-31 08:39:03  nas                     /home/git/repositories
e890453b  2020-04-30 08:22:37  nas                     /home/git/repositories
0affa4d9  2020-05-10 08:47:18  nas                     /home/git/repositories
a01d8be4  2020-07-31 08:41:25  nas                     /home/git/repositories
78afb27a  2020-08-31 08:23:52  nas                     /home/git/repositories
68a417b1  2020-09-30 08:44:49  nas                     /home/git/repositories
ac6701b4  2020-10-18 06:00:00  nas         git         /home/git/repositories
4f183431  2020-10-25 06:00:00  nas         git         /home/git/repositories
aec0b472  2020-10-25 07:24:10  aptos       home        /home/fcuny
3e98a872  2020-10-30 06:00:00  nas         git         /home/git/repositories
0268f733  2020-10-31 06:00:00  nas         git         /home/git/repositories
1b840de3  2020-11-01 06:00:00  nas         git         /home/git/repositories
2d224944  2020-11-02 06:00:00  nas         git         /home/git/repositories
fa0107dd  2020-11-03 06:00:00  nas         git         /home/git/repositories
1165032b  2020-11-04 06:00:00  nas         git         /home/git/repositories
612b66e3  2020-11-05 06:00:00  nas         git         /home/git/repositories
2de6fb79  2020-12-31 06:01:19  nas         gitea       /data/containers/gitea
ece08207  2020-12-31 06:01:41  nas         traefik     /data/containers/traefik
d59bd75a  2020-12-31 06:06:19  nas         grafana     /data/containers/grafana
168c0ddf  2020-12-31 06:07:24  nas         unifi       /data/containers/unifi
5882ffe4  2021-01-27 18:58:06  aptos       home        /home/fcuny
3565b23b  2021-01-31 06:05:18  nas         traefik     /data/containers/traefik
653d4411  2021-01-31 06:14:12  nas         gitea       /data/containers/gitea
38a3e50e  2021-01-31 06:15:13  nas         unifi       /data/containers/unifi
542e2c80  2021-01-31 06:15:13  nas         grafana     /data/containers/grafana
8c804805  2021-02-06 19:13:24  aptos       home        /home/fcuny
3f38d369  2021-02-28 06:03:28  nas         grafana     /data/containers/grafana
ef2042e2  2021-02-28 06:11:50  nas         unifi       /data/containers/unifi
b429ef99  2021-02-28 06:18:02  nas         gitea       /data/containers/gitea
b73f5128  2021-02-28 06:18:04  nas         traefik     /data/containers/traefik
7a7e3e06  2021-03-28 09:05:35  aptos       home        /home/fcuny
3a0c790f  2021-03-30 06:12:20  nas         grafana     /data/containers/grafana
58179a2f  2021-03-31 06:05:04  nas         gitea       /data/containers/gitea
fc4ede5d  2021-03-31 06:08:18  nas         unifi       /data/containers/unifi
5eaa5148  2021-03-31 06:17:13  nas         traefik     /data/containers/traefik
d7c95e53  2021-04-27 18:10:36  aptos       home        /home/fcuny
4c702501  2021-04-30 06:02:11  nas         gitea       /data/containers/gitea
8de29c3c  2021-04-30 06:04:42  nas         unifi       /data/containers/unifi
66664254  2021-04-30 06:08:25  nas         traefik     /data/containers/traefik
9a3ad896  2021-04-30 06:15:15  nas         grafana     /data/containers/grafana
344ef4c3  2021-05-15 14:22:05  aptos       home        /home/fcuny
6141b888  2021-05-30 06:14:37  nas         traefik     /data/containers/traefik
106c4819  2021-05-31 06:04:56  nas         grafana     /data/containers/grafana
8e0ba4c3  2021-05-31 06:12:37  nas         gitea       /data/containers/gitea
8cba7fbf  2021-05-31 06:17:26  nas         unifi       /data/containers/unifi
2cc04ad6  2021-06-28 17:08:25  aptos       home        /home/fcuny
8b04e195  2021-06-30 06:03:56  nas         grafana     /data/containers/grafana
d21a464f  2021-06-30 06:09:56  nas         unifi       /data/containers/unifi
f180e1a0  2021-06-30 06:10:20  nas         gitea       /data/containers/gitea
b9e0ce43  2021-06-30 06:11:50  nas         traefik     /data/containers/traefik
512e80fb  2021-07-23 17:25:45  aptos       home        /home/fcuny
28b32d1f  2021-07-31 06:03:50  nas         gitea       /data/containers/gitea
884574c8  2021-07-31 06:11:13  nas         unifi       /data/containers/unifi
a61cd90f  2021-07-31 06:16:50  nas         grafana     /data/containers/grafana
614f9123  2021-07-31 06:19:38  nas         traefik     /data/containers/traefik
17698a8a  2021-08-14 06:05:34  nas         git         /data/containers/git
b5674e76  2021-08-16 13:47:52  aptos       home        /home/fcuny
d7c251f6  2021-08-31 06:16:07  nas         gitea       /data/containers/gitea
ef20f101  2021-08-31 06:16:11  nas         unifi       /data/containers/unifi
b7cd0d5c  2021-08-31 06:16:16  nas         grafana     /data/containers/grafana
facffc9a  2021-08-31 06:16:19  nas         traefik     /data/containers/traefik
b2d31938  2021-08-31 06:16:22  nas         syncthing   /data/containers/syncthing
8ab3bee2  2021-09-27 10:35:27  aptos       home        /home/fcuny
1559f48c  2021-09-30 04:11:21  nas         gitea       /data/containers/gitea
353d202d  2021-09-30 04:11:25  nas         unifi       /data/containers/unifi
b567fec1  2021-09-30 04:11:30  nas         grafana     /data/containers/grafana
d7b239c1  2021-09-30 04:11:33  nas         traefik     /data/containers/traefik
4890d748  2021-09-30 04:11:35  nas         syncthing   /data/containers/syncthing
4d6b6646  2021-10-31 04:11:55  nas         gitea       /data/containers/gitea
b2820465  2021-10-31 04:12:01  nas         unifi       /data/containers/unifi
cd2230ff  2021-10-31 04:12:07  nas         grafana     /data/containers/grafana
807f1bb3  2021-10-31 04:12:12  nas         traefik     /data/containers/traefik
5d9c2314  2021-10-31 04:12:15  nas         syncthing   /data/containers/syncthing
5f1a2de0  2021-10-31 12:38:40  carmel      home        /home/fcuny
89f6bbec  2021-10-31 14:53:27  aptos       home        /home/fcuny
5bb120c9  2021-11-05 15:54:28  aptos       home        /home/fcuny
5fb31f63  2021-11-06 16:05:30  aptos       home        /home/fcuny
9bfd32e2  2021-11-07 18:02:06  aptos       home        /home/fcuny
d4dd252f  2021-11-17 13:40:16  aptos       home        /home/fcuny
b072a3a1  2021-11-21 04:18:17  nas         gitea       /data/containers/gitea
6ba6bff3  2021-11-21 04:18:32  nas         unifi       /data/containers/unifi
bb697aae  2021-11-21 04:18:38  nas         grafana     /data/containers/grafana
33ba0e83  2021-11-21 04:18:41  nas         traefik     /data/containers/traefik
e2cae3b5  2021-11-21 04:18:43  nas         syncthing   /data/containers/syncthing
1caaca88  2021-11-21 13:35:29  carmel      home        /home/fcuny
97d034ce  2021-11-27 19:16:12  aptos       home        /home/fcuny
5fa6b510  2021-11-28 04:11:27  nas         gitea       /data/containers/gitea
6670d391  2021-11-28 04:11:32  nas         unifi       /data/containers/unifi
77d11ce4  2021-11-28 04:11:38  nas         grafana     /data/containers/grafana
04ee74c6  2021-11-28 04:11:40  nas         traefik     /data/containers/traefik
1371d8d2  2021-11-28 04:11:43  nas         syncthing   /data/containers/syncthing
3b2a45ee  2021-11-28 09:19:13  aptos       home        /home/fcuny
b19902e6  2021-11-28 15:25:29  carmel      home        /home/fcuny
02fb34d8  2021-11-30 04:05:15  nas         gitea       /data/containers/gitea
1ac8f79f  2021-11-30 04:05:21  nas         unifi       /data/containers/unifi
848505be  2021-11-30 04:05:26  nas         grafana     /data/containers/grafana
2e48e232  2021-11-30 04:05:29  nas         traefik     /data/containers/traefik
47732732  2021-11-30 04:05:34  nas         syncthing   /data/containers/syncthing
dd141856  2021-11-30 12:06:56  carmel      home        /home/fcuny
00e5429b  2021-12-03 18:31:51  aptos       home        /home/fcuny
31b849ad  2021-12-05 04:06:10  nas         gitea       /data/containers/gitea
8cc78932  2021-12-05 04:06:26  nas         unifi       /data/containers/unifi
b7364a55  2021-12-05 04:06:38  nas         grafana     /data/containers/grafana
043c4b36  2021-12-05 04:06:43  nas         traefik     /data/containers/traefik
2e415963  2021-12-05 04:06:48  nas         syncthing   /data/containers/syncthing
1ef944db  2021-12-05 11:14:51  carmel      home        /home/fcuny
e58a2421  2021-12-06 04:02:44  nas         gitea       /data/containers/gitea
907bb839  2021-12-06 04:02:50  nas         unifi       /data/containers/unifi
050dcff3  2021-12-06 04:02:55  nas         grafana     /data/containers/grafana
72092444  2021-12-06 04:03:00  nas         traefik     /data/containers/traefik
d04b79bb  2021-12-06 04:03:03  nas         syncthing   /data/containers/syncthing
2ef060ec  2021-12-06 11:36:51  carmel      home        /home/fcuny
a3036320  2021-12-07 04:19:42  nas         gitea       /data/containers/gitea
18af7ba5  2021-12-07 04:19:48  nas         unifi       /data/containers/unifi
ba7adae4  2021-12-07 04:19:53  nas         grafana     /data/containers/grafana
b71283de  2021-12-07 04:19:57  nas         traefik     /data/containers/traefik
d1918837  2021-12-07 04:19:59  nas         syncthing   /data/containers/syncthing
ec06c179  2021-12-07 17:24:07  carmel      home        /home/fcuny
49722319  2021-12-08 04:11:10  nas         gitea       /data/containers/gitea
b7cfa0d8  2021-12-08 04:11:18  nas         unifi       /data/containers/unifi
64e98ec2  2021-12-08 04:11:25  nas         grafana     /data/containers/grafana
d5f848fd  2021-12-08 04:11:30  nas         traefik     /data/containers/traefik
ce58becc  2021-12-08 04:11:33  nas         syncthing   /data/containers/syncthing
8342e5b7  2021-12-08 17:45:07  carmel      home        /home/fcuny
93584f9e  2021-12-09 04:06:27  nas         gitea       /data/containers/gitea
fb0e6073  2021-12-09 04:06:33  nas         unifi       /data/containers/unifi
68d354c2  2021-12-09 04:06:39  nas         grafana     /data/containers/grafana
73e199bd  2021-12-09 04:06:46  nas         traefik     /data/containers/traefik
47e0e0a6  2021-12-09 04:06:49  nas         syncthing   /data/containers/syncthing
9d7bcb97  2021-12-09 11:53:49  carmel      home        /home/fcuny
c2130706  2021-12-10 04:00:56  nas         gitea       /data/containers/gitea
29af7e4f  2021-12-10 04:01:03  nas         unifi       /data/containers/unifi
393b006b  2021-12-10 04:01:08  nas         grafana     /data/containers/grafana
433a00d1  2021-12-10 04:01:13  nas         traefik     /data/containers/traefik
d4949919  2021-12-10 04:01:18  nas         syncthing   /data/containers/syncthing
ce2a8a73  2021-12-10 12:10:49  carmel      home        /home/fcuny
c8d56977  2021-12-11 04:11:20  nas         gitea       /data/containers/gitea
40f3c6d8  2021-12-11 04:11:25  nas         unifi       /data/containers/unifi
f24178f5  2021-12-11 04:11:30  nas         grafana     /data/containers/grafana
3ca4553f  2021-12-11 04:11:33  nas         traefik     /data/containers/traefik
ca41fe42  2021-12-11 04:11:35  nas         syncthing   /data/containers/syncthing
b2643ef9  2021-12-11 12:40:49  carmel      home        /home/fcuny
50cb9254  2021-12-12 04:10:34  nas         gitea       /data/containers/gitea
85de9005  2021-12-12 04:10:40  nas         unifi       /data/containers/unifi
0fd36196  2021-12-12 04:10:46  nas         grafana     /data/containers/grafana
bd8f14dd  2021-12-12 04:10:50  nas         traefik     /data/containers/traefik
ee0735e3  2021-12-12 04:10:53  nas         syncthing   /data/containers/syncthing
---------------------------------------------------------------------------------
148 snapshots
#+end_example

** How to configure a backup
All daily backups are added to the [[file:~/workspace/infrastructure/puppet/site-modules/backup/files/etc/systemd/system/backups.service][unit file]]. Each backup needs a tag (to make it easier to filter/search).

This will run once a day. The backups will be stored in =/data/backups= and then be exported to GCS.
** How to restore the backup
First, this is the [[https://restic.readthedocs.io/en/latest/050_restore.html][documentation]] to read. Here's an example:
#+begin_src sh
$ sudo restic -r /data/backups/ -p /etc/restic/password restore 8dbaaf98 --target /tmp/this-is-a-test
repository a37cfab5 opened successfully, password is correct
restoring <Snapshot 8dbaaf98 of [/data/containers/traefik] at 2021-08-14 06:05:49.547829076 -0700 PDT by restic@nas> to /tmp/this-is-a-test
$ sudo ls -l /tmp/this-is-a-test/data/containers/traefik
total 4
drwxrwxr-x 2 root root 4096 Nov  6  2020 config
#+end_src
* rclone / GCP
Backups are exported off-site to some GCS buckets, using [[https://rclone.org/][rclone]].

=restic= snapshots are exported to this [[https://console.cloud.google.com/storage/browser/fcuny-restic;tab=objects?forceOnBucketsSortingFiltering=false&project=fcuny-backups][bucket]], while our music collection is stored in this [[https://console.cloud.google.com/storage/browser/fcuny-music;tab=objects?forceOnBucketsSortingFiltering=false&project=fcuny-backups&prefix=&forceOnObjectsSortingFiltering=false][one]].

The timer for the backup can be found in [[file:~/workspace/infrastructure/puppet/site-modules/backup/manifests/service.pp][service.pp]]. All the configuration bits for =rclone= are parts of the unit file for the backups.